Evaluating Structured Output Reliability: JSON Mode Failure Rates

The 2 AM Page That Started This

A pipeline that extracts invoice line items into JSON, running clean for six weeks, started throwing downstream type errors. The on-call engineer pulled the raw model output and found this: "quantity": "twelve". Not 12. Not "12". The literal word "twelve," sitting in a field that a lenient JSON parser had happily passed through, because nobody had put a strict validator in front of it.

That's the trap with structured output. Teams ship a feature that calls itself "JSON mode" or "structured outputs," watch it work cleanly in a demo, and quietly assume the reliability problem is solved. It isn't. JSON mode guarantees you get parseable JSON. Depending on which mode you're actually using, it may or may not guarantee the output matches your schema — and even when it does, it never guarantees the JSON says anything true. Structured output reliability is not a boolean you flip on. It's a metric you have to measure, the same way you'd measure latency or cost. This article is about how to actually do that measurement, why "it validated" is a much weaker claim than people think, and where the failures hide once you go looking for them.

What "Structured Output" Actually Means, Mechanically

Before you can evaluate failure rates, you need to know which guarantee you're actually holding the model to, because "structured output" gets used to describe at least three different mechanisms with very different failure profiles.

Prompted JSON mode. This is the oldest and weakest form: you ask the model, in the system prompt, to "respond only with valid JSON matching this schema," and you hope. Nothing at the decoding level enforces this. The model can wrap the JSON in markdown fences, add a preamble ("Sure, here's the JSON you asked for:"), or drift off-schema on a long response. Still common, because it's the only option for models or endpoints without native structured output support.

Syntax-constrained JSON mode. OpenAI's response_format: {"type": "json_object"} and Gemini's response_mime_type: "application/json" (used alone, without a schema) fall here. The API guarantees the output is *syntactically valid JSON* — it will parse — but it does not guarantee the JSON matches any particular schema. You can get back {"result": "the answer is 42"} when you wanted {"answer": 42}. Valid JSON, wrong shape.

Grammar-constrained (schema-enforced) decoding. This is the real upgrade, and it's what OpenAI's response_format: {"type": "json_schema", "strict": true} (Structured Outputs), Anthropic's strict: true tool definitions and messages.parse() helper, and Gemini's response_schema are all trying to deliver. Under the hood, the JSON Schema is compiled into a constrained grammar, and the token sampler is restricted at every decoding step to only tokens that keep the output on a legal path through that grammar. If the schema says a field must be one of three enum values, the model is mechanically incapable of emitting a fourth string there — the sampler masks out every other token before sampling. Open-source equivalents (Outlines, guidance, llama.cpp's GBNF grammars, jsonformer) do the same thing at the logit level for self-hosted models.

This distinction matters for evaluation because it changes what kind of failure is even possible. With grammar-constrained decoding, schema-shape failures — wrong types, invalid enums, missing required fields — should approach zero for schemas the provider fully supports. What survives is semantic failure: the model can still confidently put the wrong number in the right slot. With prompted or syntax-only JSON mode, you're exposed to both structural and semantic failures stacked on top of each other. Your eval harness needs to check for both, and it needs to know which layer of failure is even possible given the mode you're using.

It's also worth knowing that "strict" schema modes have real constraints of their own. OpenAI's strict mode requires additionalProperties: false and effectively treats every property as required (you simulate optional fields with nullable types). Anthropic's structured-output tool schemas don't support recursive schemas or certain string/number constraints like minLength or maximum. These aren't bugs — they're the tradeoff for a hard guarantee — but they mean a schema that works fine in prompted mode can get rejected outright, or silently simplified, when you switch to strict mode. Test the actual provider and mode you ship with, not a generic "JSON schema" assumption.

The Failure Taxonomy: Where Structured Output Actually Breaks

When people say "JSON mode failure rate," they usually mean one number, but there are at least five distinct failure classes stacked on top of each other, and conflating them hides where the real risk is.

1. Parse failure. The output isn't valid JSON at all — an unescaped quote inside a string, a trailing comma, a truncated object because max_tokens was hit mid-generation. This is rare with grammar-constrained modes and common with prompted-only modes, especially on long outputs where truncation clips the closing braces before the object finishes. Constrained decoding controls *which* tokens can come next, but it can't stop the provider from cutting generation off when the token budget runs out — a long array of nested objects under a tight max_tokens is a classic way to get a well-formed-until-truncated response.

2. Schema violation. The JSON parses, but it doesn't match your schema: a required field is missing, a field has the wrong type, an enum value isn't one of the allowed options, or an array contains objects with the wrong shape. Grammar-constrained decoding should reduce this close to zero for well-formed, provider-supported schemas, but deeply nested schemas, oneOf/anyOf unions, and recursive structures are where constrained-decoding implementations are weakest — compiling a valid grammar for complex unions is genuinely hard, and providers sometimes reject the schema at request time rather than silently degrading enforcement.

3. Type coercion drift. The schema says quantity: integer, the model returns "12" as a string under a mode that doesn't strictly enforce types, and your code silently coerces it because most JSON libraries and lenient validators are permissive by default. This failure is dangerous precisely because it doesn't look like a failure — code that does int(data["quantity"]) downstream just works, until you hit a value that can't coerce cleanly (the "twelve" case above breaks even a forgiving parser).

4. Semantic failure with valid schema. The JSON is perfectly well-formed and passes every validator, but the content is wrong: a hallucinated field value, a date computed incorrectly, a summary field that misrepresents the source text, a classification label that's schema-legal but factually wrong. This is invisible to any schema validator by construction — you need ground truth or a judge to catch it, which is the whole reason evaluating structured output can't stop at "did it validate."

5. Constrained refusal collapse. When a model wants to refuse or hedge but is locked into a schema with no field for refusal, it sometimes stuffs a refusal message into a field that expects something else (a summary field containing "I cannot provide this information"), or worse, satisfies the schema with a plausible-looking non-answer rather than admitting uncertainty. If your schema has no explicit way to represent "I don't know" or "not applicable," you're quietly incentivizing the model to fabricate rather than abstain — this is one of the underappreciated costs of forcing structure onto every response.

The "Format Tax": When Structure Fights Reasoning

There's a real tension worth naming explicitly: grammar-constrained decoding restricts the token sampler at every step, which means the model can't "think out loud" inside the fields you've constrained. If you force the model straight into a strict JSON object with no room for intermediate reasoning, you can measurably hurt task accuracy on anything that benefits from working through the problem first, because the model has to commit to an answer token-by-token with no scratch space.

The common mitigation is to separate reasoning from structuring: let the model reason freely first, either in prose before the JSON or in a reasoning field placed before the answer fields in the schema, and only apply the hard constraint to the final answer fields. Ordering fields so a free-form reasoning field precedes constrained output fields is a cheap, effective fix, and it's worth testing explicitly — run the same eval both with and without a leading reasoning field and compare accuracy, not just schema-conformance rate. A schema that's perfectly conformant but consistently wrong because you starved the model of reasoning room is a worse outcome than a schema that occasionally needs a repair pass but gets the substance right.

Building an Eval Harness: Beyond "Did It Parse"

A serious structured-output eval needs to report at least three separate numbers per run, not one pass/fail:

• Parse rate — percentage of outputs that are valid JSON at all.
• Schema conformance rate — percentage of parseable outputs that validate against your schema (types, required fields, enums, nesting).
• Semantic accuracy — percentage of schema-conformant outputs where the actual field values are correct, measured against ground truth or an LLM judge.

Here's a minimal harness using Pydantic for schema validation, which is the standard tool in the Python ecosystem for exactly this job:

from pydantic import BaseModel, ValidationError, Field
from typing import Literal
import json

class InvoiceLineItem(BaseModel):
    description: str
    quantity: int = Field(gt=0)
    unit_price: float = Field(ge=0)
    category: Literal["goods", "service", "tax", "shipping"]

def evaluate_batch(raw_outputs: list[str], ground_truth: list[dict]) -> dict:
    parsed_ok = 0
    schema_ok = 0
    semantic_ok = 0
    total = len(raw_outputs)

    for raw, truth in zip(raw_outputs, ground_truth):
        try:
            data = json.loads(raw)
            parsed_ok += 1
        except json.JSONDecodeError:
            continue

        try:
            item = InvoiceLineItem.model_validate(data)
            schema_ok += 1
        except ValidationError:
            continue

        # Semantic check: does the validated data match ground truth?
        if (
            item.quantity == truth["quantity"]
            and abs(item.unit_price - truth["unit_price"]) < 0.01
            and item.category == truth["category"]
        ):
            semantic_ok += 1

    return {
        "parse_rate": parsed_ok / total,
        "schema_conformance_rate": schema_ok / total,
        "semantic_accuracy": semantic_ok / total,
    }

Run this across a batch of N samples per prompt, not a single spot check — model output is stochastic, and one clean run tells you almost nothing about reliability at scale. If parse rate is 100% but semantic accuracy is 80%, your problem isn't the JSON mode, it's the model's understanding of the task, and no amount of schema tightening will fix that on its own.

For TypeScript stacks, Zod does the equivalent job:

import { z } from "zod";

const LineItemSchema = z.object({
  description: z.string(),
  quantity: z.number().int().positive(),
  unitPrice: z.number().nonnegative(),
  category: z.enum(["goods", "service", "tax", "shipping"]),
});

function validateOutput(raw: string) {
  let parsed: unknown;
  try {
    parsed = JSON.parse(raw);
  } catch {
    return { parsed: false, valid: false };
  }
  const result = LineItemSchema.safeParse(parsed);
  return { parsed: true, valid: result.success, errors: result.success ? null : result.error.issues };
}

Retry-and-Repair Loops: Measuring Recovery, Not Just First-Pass Rate

The number that matters most in production is rarely first-pass success — it's success after a bounded retry budget, because almost every serious structured-output pipeline includes a repair loop. The pattern, popularized by libraries like instructor in Python (which wraps OpenAI and Anthropic clients with automatic Pydantic validation and retries), is:

1. Call the model, get raw output.
2. Attempt to parse and validate.
3. If validation fails, send the validation error message back to the model as a follow-up turn ("Your last response failed validation: quantity must be an integer, got 'twelve'. Please correct and resend the full JSON object.").
4. Retry up to a fixed budget — two or three attempts is typical, since retries have diminishing returns and cost real latency and tokens.

import instructor
from openai import OpenAI

client = instructor.from_openai(OpenAI())

def extract_line_item(text: str) -> InvoiceLineItem:
    return client.chat.completions.create(
        model="gpt-4o-mini",
        response_model=InvoiceLineItem,
        max_retries=2,
        messages=[{"role": "user", "content": f"Extract the line item: {text}"}],
    )

When you evaluate a pipeline like this, report first-pass conformance rate and final conformance rate after retries as separate numbers, plus the average retry count. A pipeline that's 70% correct on the first try and 98% correct after two retries is a very different system, with a very different cost and latency profile, than one that's 95% correct on the first try with no retries needed — even though the final reliability numbers might look similar on a dashboard. If you only report the post-retry number, you're hiding the actual per-call cost of the feature from whoever reads the eval report.

Where Failure Rate Actually Climbs: Complexity, Not Just Model Choice

Two variables move structured-output failure rate more than anything else, and both are within your control as the schema author, not just the model provider's problem to solve.

Nesting depth and union types. A flat schema with five string and number fields is close to a solved problem for any current frontier model under grammar-constrained decoding. A schema with three levels of nested objects, optional fields, arrays of arrays, or unions between structurally different shapes is where both schema-shape failures and semantic failures climb, because the model has to track more state across a longer generation, and constrained-decoding grammar compilers themselves have documented weak spots on complex unions and recursive definitions. If you can flatten a schema — pull nested objects up a level, replace a union of three shapes with three separate optional fields — you'll usually see a real reliability improvement for free, before touching prompting at all.

Field count and required-field pressure. Schemas with a large number of required fields increase the chance that at least one gets a low-confidence or hallucinated value, since the model can't leave anything blank. Marking genuinely optional data as optional in the schema, using nullable types instead of forcing a placeholder empty string, reduces the pressure to fabricate a value just to satisfy the shape.

It's also worth being honest in your eval writeups about model tier: smaller and cheaper models generally show higher failure rates on both schema conformance, in prompted or syntax-only modes, and semantic accuracy than frontier models, and the gap tends to widen as schema complexity increases. This is exactly the kind of claim that should be backed by your own eval numbers on your own schemas rather than a borrowed benchmark figure — the failure curve for a twelve-field nested invoice schema on your production data can look nothing like a public benchmark's simple single-field extraction task.

A related, easy-to-miss variable is prompt-schema mismatch: schemas that were written by one team and prompts written by another often drift apart over time, especially after a "quick fix" to either side. If your prompt says "return the customer's full name" but your schema field is named customer_name with no description, you're relying entirely on the model's guess about what that key means. Adding a description to every field in the JSON Schema, not just the top-level object, is a cheap habit that measurably reduces semantic failures, because it gives the model an explicit definition to anchor to instead of inferring one from the field name alone. Treat the schema itself as part of the prompt, not as a separate contract that exists only for the validator's benefit.

A Practical Eval Checklist Before You Ship

Before calling a structured-output pipeline production-ready, run through this:

• Test with a batch of samples per representative prompt, not a single spot-check — stochastic decoding means one clean run tells you almost nothing.
• Report parse rate, schema conformance rate, and semantic accuracy as three separate numbers, never collapsed into one "success rate."
• Test your actual schema at its real nesting depth. Don't validate a flattened toy version and assume the production schema behaves the same.
• Include at least one adversarial or edge-case input per field (missing data, ambiguous category, boundary numeric values) so you can see how the model handles "I don't actually know" rather than only the happy path.
• Measure first-pass and post-retry conformance separately if you use a repair loop, along with average retries and added latency.
• Check for the format tax by running the same task with and without a reasoning field preceding the constrained fields, and compare accuracy, not just conformance.
• Re-run the full eval suite whenever you change model version, provider, or schema. Structured-output reliability is a property of the whole model-schema-decoding-mode combination, not of your prompt alone, and changing any one leg invalidates prior numbers.

Closing: Schema Validity Is the Floor, Not the Finish Line

The uncomfortable truth about structured output is that the industry's tooling has gotten very good at solving the easy part of the problem — making sure the braces close and the types match — while the hard part, whether the content inside those correctly typed fields is actually right, still requires the same judgment a human reviewer would apply. A schema validator will happily pass a JSON object where every field is confidently, precisely wrong.

That's why any serious structured-output eval eventually needs a semantic layer that a schema check can't provide, and increasingly the tool for that layer is LLM-as-a-Judge: a separate model call that looks at the source input, the extracted structured output, and scores whether the fields are semantically faithful to the source, not just syntactically legal. A judge prompt for this doesn't need to be elaborate — pass it the original text, the candidate JSON, and a short rubric per field ("does quantity match a number explicitly stated or clearly computable from the source, yes or no"), and log its verdicts alongside your schema-conformance numbers rather than instead of them.

Treat schema conformance as the floor your pipeline should never fall below, and treat LLM-as-a-Judge scoring on semantic accuracy as the metric that actually tells you whether the feature is trustworthy in front of a real user. The two numbers answer different questions — one tells you the model followed instructions, the other tells you it told the truth — and a mature eval report keeps them separate instead of collapsing everything into a single green checkmark.