# Get data you can trust from an API: schemas, validation and retries

Valid JSON, JSON that matches your schema and JSON with the right values are three different things. Let the vendor guarantee the shape, check the values in your own code, retry once with the errors, then hand the item to a person.

Guides · v1.0 · draft · https://hayzlabs.com/guides/structured-output-you-can-trust/

In short:
- Let the vendor's schema feature fix the shape, then check values in your code: schema, business rules and quotes.
- Make every field required, use null for missing values, enums with OTHER, and a quote field for each value that matters.
- Retry once with the exact errors, then send the item to a person, and log every failure to fix the prompt or schema.

**Use it when** a script asks a model to pull fields out of an email, an invoice or a form, and you need to use those fields without reading every reply.

**Not when** you have only a handful of documents (read them yourself), or a wrong field would move money on its own (keep a person on every item). Without code, the same idea still works: check each value in a spreadsheet formula or by eye, and read every number you will rely on against the original, as in [Photos in, data out](https://hayzlabs.com/workflows/photos-in-data-out/).

**You need** a working API call with a spending cap ([Your first API call](https://hayzlabs.com/guides/first-api-call-with-a-spending-cap/)), Python 3, and about an hour to adapt the example. Remove personal data you do not need first: [What not to paste](https://hayzlabs.com/start/what-not-to-paste/). The checks, the retry rule and the order of the steps are our suggestion.

## Short answer

Let the vendor's schema feature guarantee the shape. Check the values in your own code: schema, business rules, and a check that each quoted piece of evidence is really in the document. On a failure, retry once with the errors, then send the item to a person. Log every failure.

## Three different failures

| Level | What it means | Example (illustrative) | What catches it |
|---|---|---|---|
| Valid JSON | It parses | `{"total": 152.5` cut off, so it does not parse | The vendor's schema feature, plus `json.loads` |
| Matches the schema | Right fields, types and allowed values | `"total": "125.50"` as text, or a missing field | The vendor's schema feature, plus your own validator |
| Correct values | The values are true to the document | `"total": 152.5` when the invoice says 125.50 | Only your own checks, and a person |

OpenAI separates the first two: JSON mode and Structured Outputs both "ensure valid JSON is produced", but "only Structured Outputs ensure schema adherence". No vendor feature reaches the third. Google: "While output is syntactically correct JSON, always validate values in your application".

Try the three levels on one invoice. Pick a reply, or edit it, and watch which check catches it:

<!-- WIDGET replyChecker -->

## What the vendor features guarantee

**Dated box, read 2026-10-10.** Check the vendor pages before you build.

| | Anthropic | OpenAI | Google Gemini |
|---|---|---|---|
| Name and switch | Structured outputs. Put the schema in `output_config.format` with `"type": "json_schema"`. For tools, set `strict: true` on the tool, next to `input_schema`. No beta header is needed now. | Structured Outputs. Responses API: `text.format` with `"type": "json_schema"` and `"strict": true`. Chat Completions: `response_format` with `json_schema` and `"strict": true`. | Structured output. The current guide (Interactions API) sets `response_format` with `"mime_type": "application/json"` and a `schema`. The `generateContent` reference shows `response_mime_type` and `response_schema` in `generationConfig`. |
| Models | Listed on the page, including Haiku 4.5 and 5.5, Sonnet 4.5 to 5.5 and Opus 4.5 to 5.5 | "starting with GPT-4o" | Not listed on the page |
| When the shape can still break | `stop_reason` of `"refusal"` or `"max_tokens"`. Enum and const capitalisation is not guaranteed. | Refusals are "programmatically detectable"; the examples check a `refusal` field | "Very large or deeply nested schemas may be rejected" |
| Schema limits | No recursion, no `minimum` or `maximum`, no `minLength` or `maxLength`, `additionalProperties` only `false`, and some regex features. Per request: 20 strict tools, 24 optional parameters, 16 parameters with union types. | The supported-schemas section did not load for us. OpenAI's own examples set `additionalProperties: false`, list every field in `required`, and use `["string", "null"]` where a value may be missing. | A subset of JSON Schema: `enum`, `format`, `minimum`, `maximum`, `minItems`, `maxItems` and a few more |

If you use Anthropic tool use with `input_schema` instead, set `strict: true`. Without it, Anthropic says, an input may come back as `"2"` instead of `2`, or miss a required field. Also, Anthropic puts required properties before optional ones, so do not depend on key order, and the first request with a new schema is slower while it compiles.

## Design the schema

> [!OUR RULE]
> Every field is required. A value that is not in the document is `null`, never a guess. Categories are enums with an `OTHER` value. Every value that matters has a `_quote` field with the exact words from the document. No free text where a number, a date or an enum will do.

- **Required plus `null`** keeps "not in the document" apart from "the model left it out".
- **Enums** stop near-duplicates such as `GBP`, `£` and `pounds`. `OTHER` means the model is never forced into a wrong one. Compare enum values case-insensitively, as Anthropic advises.
- **Rules the API ignores**, such as `minimum` on Anthropic, go in a local copy of the schema that your code checks.
- **An `unsure` flag** helps when the input is hard to read, such as a photo of a receipt: a boolean that is true if any field was guessed. An item with it set goes to a person, and nothing is entered. This is our suggestion, and the example code below does not include it.
- **Keep it small and flat.** Large or deep schemas can be rejected. Split a long form into two calls.

## Check it in your own code

On every reply, in order:

1. **Parse and validate** against your full local schema (with `jsonschema`, or a Pydantic model).
2. **Business rules.** Line items add up to the total. Dates are real and in range. IDs exist in your system.
3. **Quote check.** Each quote appears word for word in the input, after collapsing spaces and line breaks. The value appears in its own quote.
4. **Route.** Pass, retry once with the errors, or hand to a person.

In the offline test, a reply giving a total of 152.50 for an invoice that says 125.50 passed the schema check and failed both the business rule and the quote check.

```python title="extract_invoice.py · v1.1"
import datetime, json, re
import anthropic
from jsonschema import Draft202012Validator

MODEL = "claude-haiku-5-5"
NULLABLE_STR = {"type": ["string", "null"]}
SCHEMA = {
    "type": "object",
    "properties": {
        "invoice_number": NULLABLE_STR,
        "invoice_number_quote": NULLABLE_STR,
        "invoice_date": {"type": ["string", "null"], "description": "YYYY-MM-DD"},
        "invoice_date_quote": NULLABLE_STR,
        "currency": {"type": "string", "enum": ["USD", "EUR", "GBP", "OTHER"]},
        "total": {"type": ["number", "null"]},
        "total_quote": NULLABLE_STR,
        "line_items": {"type": "array", "items": {
            "type": "object",
            "properties": {"description": {"type": "string"}, "amount": {"type": "number"}},
            "required": ["description", "amount"],
            "additionalProperties": False}},
    },
    "required": ["invoice_number", "invoice_number_quote", "invoice_date",
                 "invoice_date_quote", "currency", "total", "total_quote", "line_items"],
    "additionalProperties": False,
}
# Rules we check here ourselves (minimum is on the API's unsupported list; we check the date pattern locally too).
LOCAL_SCHEMA = json.loads(json.dumps(SCHEMA))
LOCAL_SCHEMA["properties"]["total"]["minimum"] = 0
LOCAL_SCHEMA["properties"]["invoice_date"]["pattern"] = r"^\d{4}-\d{2}-\d{2}$"
VALIDATOR = Draft202012Validator(LOCAL_SCHEMA)

PROMPT = ("Extract the invoice fields from the document below. Use null when a value "
          "is not in the document; never guess. For each *_quote field, copy the exact "
          "text from the document that shows the value.\n\n<document>\n{doc}\n</document>")


def squash(text):
    return re.sub(r"\s+", " ", text).strip()


def check(document, raw):
    """Return (data, errors). An empty error list means every check passed."""
    try:
        data = json.loads(raw)
    except json.JSONDecodeError as e:
        return None, [f"not valid JSON: {e}"]
    if isinstance(data, dict) and isinstance(data.get("currency"), str):
        data["currency"] = data["currency"].upper()  # enum case is not guaranteed
    errors = [f"{'/'.join(map(str, e.path)) or 'root'}: {e.message}"
              for e in VALIDATOR.iter_errors(data)]
    if errors:
        return data, errors
    doc = squash(document)
    for field in ("invoice_number", "invoice_date", "total"):
        value, quote = data[field], data[field + "_quote"]
        if value is not None and not quote:
            errors.append(f"{field}: value given without a quote")
        if quote and squash(quote) not in doc:
            errors.append(f"{field}_quote: not found word for word in the document")
    if data["invoice_number"] and data["invoice_number_quote"] \
            and data["invoice_number"] not in data["invoice_number_quote"]:
        errors.append("invoice_number: not in its own quote")
    if data["total"] is not None and data["total_quote"]:
        numbers = [float(n.replace(",", "")) for n in re.findall(r"\d[\d,]*\.?\d*", data["total_quote"])]
        if data["total"] not in numbers:
            errors.append("total: not in its own quote")
    if data["total"] is not None and data["line_items"]:
        items = round(sum(i["amount"] for i in data["line_items"]), 2)
        if abs(items - data["total"]) > 0.01:
            errors.append(f"total: {data['total']} but line items add up to {items}")
    if data["invoice_date"]:
        try:
            day = datetime.date.fromisoformat(data["invoice_date"])
            if not datetime.date(2020, 1, 1) <= day <= datetime.date.today():
                errors.append(f"invoice_date: {day} is outside the expected range")
        except ValueError:
            errors.append(f"invoice_date: {data['invoice_date']} is not a real date")
    return data, errors


def ask(client, document, feedback=None):
    messages = [{"role": "user", "content": PROMPT.format(doc=document)}]
    if feedback:
        messages += [{"role": "assistant", "content": feedback["raw"]},
                     {"role": "user", "content": "Your answer failed these checks:\n- "
                      + "\n- ".join(feedback["errors"]) + "\nReturn a corrected answer."}]
    response = client.messages.create(
        model=MODEL, max_tokens=1024, messages=messages,
        output_config={"format": {"type": "json_schema", "schema": SCHEMA}})
    if response.stop_reason in ("refusal", "max_tokens"):
        return None, [f"stop_reason: {response.stop_reason}"]
    return next(b.text for b in response.content if b.type == "text"), []


def extract(document, doc_id, client=None, log_path="extract_failures.jsonl"):
    client = client or anthropic.Anthropic()  # reads ANTHROPIC_API_KEY
    feedback = None
    for attempt in (1, 2):  # one try, one retry, never a loop
        raw, errors = ask(client, document, feedback)
        data, errors = check(document, raw) if raw else (None, errors)
        if not errors:
            return {"status": "ok", "data": data, "attempts": attempt}
        with open(log_path, "a") as log:
            log.write(json.dumps({"doc_id": doc_id, "attempt": attempt, "model": MODEL,
                                  "at": datetime.datetime.now().isoformat(),
                                  "errors": errors, "raw": raw}) + "\n")
        if raw is None:
            break  # refusal or cut off: feedback will not fix it (for a cut-off, raise max_tokens)
        feedback = {"raw": raw, "errors": errors}
    return {"status": "needs_person", "errors": errors}
```

Run it with `pip install anthropic jsonschema` and your key in `ANTHROPIC_API_KEY`, then `extract(text, doc_id="<DOC_ID>")`. `check()` was tested on 16 sample replies and `extract()` with a stubbed API (anthropic 1.13.0, jsonschema 4.26.0, Python 3.13). It has not been run against the live API.

Replace the following:
- `<DOC_ID>`: your own ID for the document, such as a file name, so a failure can be traced back.
- `MODEL`: any model the structured outputs page lists. `claude-haiku-5-5` is Anthropic's cheapest.
- `SCHEMA` and the field names: your own fields, each important one with a `_quote` partner.
- The date range and the rules in `check()`: your own business rules, such as "the supplier ID is in our supplier list".

## Retry once, then a person

> [!OUR RULE]
> One retry, with the exact errors fed back. If it fails again, the item goes to a person. Never loop. A refusal or a cut-off reply is not retried with feedback: raise `max_tokens` for a cut-off, and send a refusal to a person.

A loop can run up your bill on one bad document, so keep the cap from [Your first API call](https://hayzlabs.com/guides/first-api-call-with-a-spending-cap/) as well.

## Log every failure

The example writes one line per failed attempt to `extract_failures.jsonl`: document ID, attempt, model, time, errors and raw reply. Count the errors by type each week. The same error on many documents points at the prompt or the schema. Raw replies can hold personal data, so keep the log private and delete it on a schedule ([What not to paste](https://hayzlabs.com/start/what-not-to-paste/)).

## Steps

| # | Step | Who | Time | Check |
|---|---|---|---|---|
| 1 | List the fields, which ones matter, and the business rules that tie them together. | Person | 15 min | Each field has a type, and each rule could be checked by code. |
| 2 | Write the schema: all fields required, `null` for missing, enums with `OTHER`, a `_quote` per important value. | Person | 15 min | It uses nothing on the vendor's unsupported list. **If it fails:** move that rule to the local schema. |
| 3 | Adapt `check()` and test it on hand-made replies: one good, and one for each kind of failure. | Person | 20 min | Every bad reply fails, and the good one passes. |
| 4 | Run it on 20 real documents and read every result. | Person and AI | 20 min | No reply that passed `check()` has a wrong value. **If it fails:** add a rule that would have caught it. |
| 5 | Run the batch. Read every `needs_person` item and the failure log. | Person and AI | varies | Every item is `ok` or handled by a person. |
| 6 | Hand-check a random sample of the `ok` items to measure how often they are still wrong. | Person | 30 min | You have an error rate with a range, as in [One prompt, many rows](https://hayzlabs.com/workflows/one-prompt-many-rows/). |

## How to tell it worked

- Every document ends as `ok` or `needs_person`, and nothing crashes.
- Every hand-made bad reply fails `check()` with an error that names the field.
- The share of `needs_person` falls as you fix the prompt from the log.
- A random sample of `ok` items meets your error target ([One prompt, many rows](https://hayzlabs.com/workflows/one-prompt-many-rows/)). Re-run it when you change model ([A personal eval set](https://hayzlabs.com/workflows/personal-eval-set/)).

> [!LIMIT]
> A schema guarantees shape, not truth. A quote check proves the words are in the document, not that they are the right words: a quote of the wrong line passes unless a business rule catches it. These checks suit clear, quotable values, and do little for judgments such as "is this email urgent". Vendor features change often. The code was tested offline only.

## Sources

1. Anthropic. Structured outputs. Vendor documentation, read 2026-10-10. [Link](https://platform.claude.com/docs/en/build-with-claude/structured-outputs)
2. Anthropic. Strict tool use. Vendor documentation, read 2026-10-10. [Link](https://platform.claude.com/docs/en/agents-and-tools/tool-use/strict-tool-use)
3. Anthropic. Define tools. Vendor documentation, read 2026-10-10. [Link](https://platform.claude.com/docs/en/agents-and-tools/tool-use/implement-tool-use)
4. Anthropic. Models overview. Vendor documentation, read 2026-10-10. [Link](https://platform.claude.com/docs/en/about-claude/models/overview)
5. OpenAI. Structured model outputs. Vendor documentation, read 2026-10-10. [Link](https://developers.openai.com/api/docs/guides/structured-outputs)
6. Google. Structured output, Gemini API. Vendor documentation, read 2026-10-10. [Link](https://ai.google.dev/gemini-api/docs/structured-output)