Run one prompt over hundreds of rows, then measure how often it's wrong

Use it when you have a list of hundreds or thousands of similar items, such as survey comments, support emails, product names or invoice lines, and you want each one labelled, sorted or turned into fields by an assistant.

Not when a single wrong row would cause real harm, such as a payment, a medical record or a decision about one person. Have a person handle every row of those, and use the model at most to flag rows for them.

You need the list in a spreadsheet with one item per row, a short written definition of each label, an assistant you are allowed to paste this data into, and about two hours for 1,000 rows. Remove names and contact details first: What not to paste. The steps, sizes and verdicts are our suggestion.

Why you measure instead of spot-checking

A model can label text well. In a 2023 study of 2,382 tweets, ChatGPT's zero-shot labels beat crowd workers on four of five tasks, at under $0.003 a label (Gilardi et al., 2023, benchmark, peer-reviewed).
The same model is not consistent. In another 2023 test, small rewordings of the prompt and repeats of the same input gave different labels, and the author advised against using it unchecked (Reiss, 2023, preprint).
Formatting alone matters. Trivial format changes moved one older model's accuracy by up to 76 points (Sclar et al., 2024, benchmark, peer-reviewed).

So how well it does depends on your task, your labels and your exact prompt. Reading the first twenty rows tells you little. A random sample with a count tells you how often it is wrong.

Steps

Set up once:

#	Step	Who	Time	Check
1	Write the labels or fields, one line each, with an example. Add an `unsure` label. Write the pass rule for a row, such as "the label matches the definition a colleague would pick".	Person	15 min	A colleague could apply your definitions without asking you.
2	Set the target: the highest error rate you can live with, such as 5%. Write it down now, before any run.	Person	2 min	The target is written.
3	Label 30 rows yourself as a test set. Pick some easy ones and every hard kind you can think of. Keep this file apart from the rest.	Person	25 min	30 rows, each with your label.
4	Write the prompt from the template below and run it on the test set in a fresh chat. Compare row by row.	Person and AI	10 min	Your labels and the model's agree on at least 28 of 30. If it fails: fix the definitions or the prompt where they disagreed and run the test set again. After three tries, the task may need a person.
5	Freeze the prompt. Save it with a version number and the model's name.	Person	2 min	The prompt file has a version and a date.

Run the batch:

#	Step	Who	Time	Check
6	Give every row an ID. Send the rows in numbered chunks of 50 to 100, in a fresh chat each, with the frozen prompt.	Person and AI	20 min	Each chunk has its own output.
7	Check mechanically: rows in equals rows out, every ID appears once, every label is on your list, nothing is blank.	Person	10 min	All four hold. If it fails: re-run that chunk. Never fill gaps by hand guesses.
8	Read every row labelled `unsure` yourself, and every row where a mistake would matter most. Leave these out of the sample.	Person	15 min	Each one has your label.
9	Draw a random sample of 100 of the other rows. Do not use the test set. Label them yourself without looking at the model's label, then compare.	Person	30 min	100 rows, each marked right or wrong.
10	Put the counts in the tool below and follow the verdict.	Person	2 min	You accepted, checked more, or fixed and re-ran.
11	Log the prompt version, model, date, sample size, errors and range.	Person	2 min	The log row is complete.

To draw the sample in a spreadsheet, add a column with =RAND(), copy it and paste it back as values so it stops changing, sort by it, and take the first 100 rows.

Batch prompt · v1.0

You will label each row of the table below.

Labels (use exactly one per row):
<LABEL_1>: <DEFINITION_1>. Example: <EXAMPLE_1>
<LABEL_2>: <DEFINITION_2>. Example: <EXAMPLE_2>
unsure: the row fits no label, or fits two equally.

Rules:
- Return one line per input row, in the same order, as: id,label
- Return every id exactly once. Do not skip, merge or add rows.
- Do not explain. Do not add a header.
- If you are not sure, use unsure. Do not guess.

Rows (id,text):
<ROWS>

Replace the following:

<LABEL_1>, <DEFINITION_1>, <EXAMPLE_1>: a short label, one sentence on what belongs in it, and one real example. Repeat for each label.
<ROWS>: one chunk of 50 to 100 rows as id,text.

batch-log.csv · v1.0

date,prompt_version,model,rows,checked,errors,low,high,target,decision
<RUN_DAY>,<PROMPT_VERSION>,<MODEL>,<ROWS_TOTAL>,<CHECKED>,<ERRORS>,<LOW>,<HIGH>,<TARGET>,<DECISION>

How often is it wrong?

Enter the rows you checked in step 9 and the errors you found. The tool gives the error rate with a 95% range and a verdict against your target.

Worked example (illustrative). 1,200 survey comments sorted into six themes. The random sample of 100 had 2 errors. The likely error rate is 2%, but the 95% range runs from about 0.6% to 7%, so against a 5% target the answer is "not sure yet". If the next 42 rows checked have no errors, the upper limit falls under 5%. Even so, expect about 22 wrong labels in the 1,100 rows nobody checked.

The range uses the Wilson interval, which a standard review recommends for small samples over the textbook formula (Brown, Cai and DasGupta, 2001, peer-reviewed). Two numbers worth knowing: with no errors in 100 checked rows, the upper limit is still about 3.7%, and with no errors in 30 it is about 11%. A small clean sample proves less than it feels like.

How to tell it worked

Rows in equals rows out, and every label is on your list.
You can state the error rate as a range, not only as a single number.
The decision matches the rule you wrote before the run.
The log names the prompt version, so a later re-run can be compared.
It has failed if you changed the target after seeing the sample, or if the sample was not random, such as the first 100 rows.

Sources

Gilardi, Alizadeh and Kubli, 2023. ChatGPT against crowd workers on 2,382 tweets. Peer-reviewed (PNAS). Link
Reiss, 2023. Reliability of ChatGPT for text classification. Preprint. Link
Sclar et al., 2024. Sensitivity of models to prompt formatting. Peer-reviewed benchmark (ICLR). Link
Brown, Cai and DasGupta, 2001. Interval estimation for a binomial proportion. Peer-reviewed (Statistical Science). Link

Changes

v1.010 Oct 2026first version