Test any new model on your own tasks (a personal eval set)
Keep ten to twelve of your own tasks with written pass rules, run them on any new model or plan, judge them yourself, and switch only by a rule you wrote in advance.
- What this is
- How to build a small test, called an eval set, from your own everyday tasks, to see whether a new AI model does them better.
- Why it matters
- You switch models or plans based on how they do on your own work, not on public rankings or a good first impression.
- The short version
- Save ten to twelve of your own tasks, each with a fixed input and a pass rule, such as all three dates match the source, written before any run.
- Write the switch rule first, then run every task on the new model in fresh sessions and judge each output yourself.
- Read the high-stakes tasks on their own, and switch only if the rule you wrote says so.
On this page
Use it when a new model, a new plan or a price change makes you wonder whether to switch, or whether a smaller and cheaper model would do the job.
Not when you do not yet have ten or so tasks you repeat. Note the tasks you hand to AI for a month, and come back with that list.
You need a plain text or spreadsheet file called evals.csv, one saved input for each task, and about 90 minutes the first time. Each later re-test takes about 30 minutes. Remove names and private details from saved inputs first: What not to paste. The steps, times and switch rule are our suggestion.
Steps
Build the set once:
| # | Step | Who | Time | Check |
|---|---|---|---|---|
| 1 | List 10 to 12 tasks: about four you do every week, four where AI has let you down before, and two to four where a mistake would matter. | Person | 20 min | The list has all three kinds. If it fails: start with what you have and add to it after a month. |
| 2 | For each task, save one fixed input and write its pass rule before any run, such as "all three dates match the source". Make sure two of the tasks are ones the current model fails. | Person | 55 min | Every task has an input file and a rule a stranger could apply. If it fails: make the rule more specific, or drop the task. |
| 3 | Run every task on the model you use now, in a fresh session each, and record pass or fail, minutes and cost. | Person and AI | 15 min | Every row of the baseline is filled. |
Re-test whenever something new appears:
| # | Step | Who | Time | Check |
|---|---|---|---|---|
| 4 | Name the challenger (a new model, a new plan or a smaller model) and write down the switch rule, with its numbers, before you run anything. | Person | 2 min | The rule is written. If it fails: copy the rule below and change the numbers now, not later. |
| 5 | Run every task on the challenger with the same saved inputs, in a fresh session each. | Person and AI | 12 min | Every task has an output. If it fails: mark a task n/a if the challenger lacks a feature it needs, and leave it out of both scores. |
| 6 | Judge each output against its pass rule yourself. Do not ask a model to grade its own work. | Person | 9 min | Every task has a pass or a fail. If it fails: where the rule was unclear, rewrite it and judge both models' outputs again. |
| 7 | Optional. For tasks with no clear pass rule, put the two outputs side by side with the labels hidden and the order swapped, and count wins. | Person | 4 min | The labels were hidden and the order was swapped. |
| 8 | Apply the switch rule and decide: switch, keep, or step down. | Person | 2 min | The decision follows the rule you wrote in step 4. |
| 9 | Record the scores, the cost and the date in evals.csv. | Person | 1 min | Every row has the challenger's name and the date. |
Keep the results in one file, one row per task and model. The columns are: task code, saved input, pass rule, stakes (L, M or H), model, date, pass (Y or N), whole minutes including your fixes, cost (0 on a flat plan) and a note.
Example (illustrative). Four of Lena's twelve tasks, run on her current model and on a challenger on the same flat plan:
task_id,file,pass_rule,stakes,model,date,pass,minutes,cost,note
T01,inputs/T01-invoice.txt,"all three dates match the source",H,current,2026-11-03,Y,3,0,
T01,inputs/T01-invoice.txt,"all three dates match the source",H,challenger,2026-11-03,N,4,0,"due date moved by one day"
T04,inputs/T04-notes.txt,"names every decision in the notes, under 150 words",L,current,2026-11-03,N,6,0,"missed the budget decision"
T04,inputs/T04-notes.txt,"names every decision in the notes, under 150 words",L,challenger,2026-11-03,Y,2,0,
T07,inputs/T07-sales.csv,"formula returns the totals in check.csv",M,current,2026-11-03,N,8,0,"summed the header row"
T07,inputs/T07-sales.csv,"formula returns the totals in check.csv",M,challenger,2026-11-03,Y,3,0,
T10,inputs/T10-complaint.txt,"keeps the refund amount and date, promises nothing new",M,current,2026-11-03,Y,2,0,
T10,inputs/T10-complaint.txt,"keeps the refund amount and date, promises nothing new",M,challenger,2026-11-03,Y,2,0,
Across all twelve tasks the current model passed 8 and the challenger 10. That is the gap of 2 the rule below asks for, at the same cost, but the challenger failed T01, a high-stakes task. The rule says keep the current model and re-test in three months. Without the rule written in advance, the faster summaries would have been easy to switch for.
Sample pass rules (our suggestion; make each one specific to the task):
- Summary: names every decision or action in the source, adds nothing that is not there, under a word limit.
- Pulling out facts (dates, amounts, names): every value matches the source and none is missing.
- Rewrite or change of tone: keeps every fact, amount and date, and promises nothing the original did not.
- Reply to a message: answers every question in the original.
- Formula or calculation: gives the same result as a total you worked out yourself and saved.
- Script or code: runs on the saved input and produces the expected output file.
The switch rule
With a set this small, one task more or less is noise. That is why the rule asks for a gap of 2 before a switch. With only five tasks, treat the result as a hint about whether a full re-test is worth the time, not as a basis for switching.
Try the rule on an example scorecard below. Change the results, the high-stakes boxes and the cost to see which part of the rule decides.
How to tell it worked
- You can name the tasks on which the two models differ, not only the totals.
- The decision matches the rule you wrote before the run.
- At the next re-test you have retired some tasks and added others, and
evals.csvshows the dates. - It has failed if you changed a pass rule after seeing the outputs. Change it for both models, and judge both again.
Why judge your own tasks yourself
- Model judges are biased. When answer order was swapped, GPT-4 gave a consistent verdict 65% of the time, GPT-3.5 46.2% and Claude-v1 23.8%, and a padded, repetitive answer fooled GPT-3.5 and Claude-v1 91.3% of the time (Zheng et al., 2023, benchmark, peer-reviewed). These were 2023-era judges. That is why you judge, and why step 7 hides the labels and swaps the order.
- A cheaper model can pass. A learned router kept about 95% of GPT-4's quality while cutting costs by over 85% on one benchmark (RouteLLM, benchmark, preprint). Your set tells you whether that holds for your tasks.
- Public rankings cannot tell you how a model does on your tasks. Why the task decides is in When not to use AI.
Sources
- Zheng et al., 2023. Study of AI judges on chat and question sets. Peer-reviewed benchmark. Link
- RouteLLM, 2024. Learned router between a strong and a cheaper model. Preprint benchmark. Link
Changes
- v1.010 Oct 2026first version
- Time to do it
- 30 minutes per re-test, and about 90 to build the set the first time (our estimate)
- Sources
- 2
- Updated
- 10 Oct 2026
- Last checked
- 4 Oct 2026