Prompt Engineeringstructured output jsonJSON schematool useLLM API

Getting Reliable JSON and Structured Output from LLMs

Pramod Dutta · Jun 19, 2026 · 16 min read

If you've ever shipped a feature that parses JSON out of an LLM response, you've hit the same wall: the model wraps it in a markdown fence, adds a sentence before it, or drops a trailing comma under load, and your JSON.parse() throws in production. Structured output json is the fix: instead of asking nicely and hoping, you constrain the model's response format at the API level so it's either valid against your schema or the request fails loudly. This article covers the three real mechanisms (schema-constrained output, tool calling, and forced tool choice), how they differ across providers, and the validation layer you still need even when the API "guarantees" the shape.

Every major provider now has a version of this. The terms differ (JSON mode, structured outputs, function calling, tool use) but they solve the same problem: get a response your code can json.loads() without a try/except that logs and prays. What they don't solve is semantic correctness. A schema-valid response can still have the wrong values, a hallucinated field, or a plausible-looking null where you needed a real answer. This article treats both halves: getting valid JSON, and getting JSON you can trust.

Why plain prompting doesn't reliably give you JSON

Before structured output existed as an API feature, the pattern was: tell the model "respond only in JSON" and parse the output. This fails in predictable ways:

The model adds a preamble ("Here's the JSON you requested:") before the actual object.
It wraps the object in a markdown code fence, which your parser doesn't strip.
Under longer generations, it truncates mid-object if it hits a token limit.
It invents field names close to but not exactly what you specified (user_name instead of username).
On edge-case inputs, it apologizes in prose instead of returning the requested shape at all.

None of this is a model quality problem in isolation. It's that free-text generation has no structural constraint. The model is predicting the next token based on everything that came before, including its own drift. A prompt instruction is a strong prior, not a hard rule. If you're validating this with try: json.loads(response) except: retry, you're paying for a retry loop that structured output APIs make mostly unnecessary.

Structured output json via schema-constrained generation

The strongest guarantee available today is schema-constrained decoding: the provider restricts what tokens the model can emit at each step so the output is guaranteed to conform to your JSON Schema. This isn't prompt engineering, it's decoding-time enforcement, and it's the mechanism to reach for whenever the provider supports it.

On the Claude API this is output_config.format. The recommended entry point is client.messages.parse(), which sends the schema and validates the response for you:

from anthropic import Anthropic
from pydantic import BaseModel

class Invoice(BaseModel):
    vendor: str
    amount_cents: int
    currency: str
    line_items: list[str]

client = Anthropic()

response = client.messages.parse(
    model="claude-opus-4-8",
    max_tokens=1024,
    messages=[{"role": "user", "content": "Extract the invoice fields from: ..."}],
    output_config={"format": {"type": "json_schema", "schema": Invoice.model_json_schema()}},
)

invoice = response.parsed_output  # an Invoice instance, or None if parsing failed
if invoice is None:
    # stop_reason tells you why, see the error-handling section below
    raise ValueError(f"structured output failed: {response.stop_reason}")

If you call messages.create() directly instead of .parse(), you get the raw JSON string back in the response content and validate it yourself against the same schema. Either way, the schema lives in one place: output_config.format, not scattered across the prompt as English instructions.

OpenAI's equivalent is response_format with type: "json_schema" and strict: true, which behaves the same way conceptually: the schema is enforced during decoding, not just requested in the prompt. Gemini's equivalent is responseSchema on the generation config. If you're building against multiple providers, write your schema once (Pydantic, Zod, whatever your language's canonical schema library is) and translate it to each provider's parameter shape at the call site, rather than hand-writing three copies that drift out of sync.

JSON Schema limitations you'll actually hit

Schema-constrained output isn't full JSON Schema. Every provider that implements this trims the spec down to what's efficiently enforceable during decoding. On the Claude API, supported constructs include basic types, enum, const, anyOf, allOf, $ref/$def, and a handful of string formats (date-time, email, uuid, and similar). Not supported: recursive schemas, numeric constraints like minimum/maximum, string length constraints, and additionalProperties set to anything other than false.

That last one matters more than it looks. Every object in your schema needs additionalProperties: false explicitly set, or the constraint isn't enforced the way you'd expect. If you're generating schemas from Pydantic or Zod, check what your library emits by default; some don't set this without an explicit flag.

Practical implication: don't lean on the schema to do validation the schema can't express. If you need amount_cents > 0, or a string under 200 characters, that's a job for your validation layer after the response comes back, not a constraint you can bake into output_config.format. The SDK's Python and TypeScript clients will silently strip unsupported constraints and validate them client-side when you use their schema helpers, but if you're constructing the schema by hand, verify what actually made it into the request.

Tool use as a structured output pattern

Before dedicated structured-output parameters existed, tool calling was already a de facto structured output mechanism, and it's still the right tool when you want the model to choose between multiple possible actions rather than fill in one fixed shape. Define a tool with an input_schema, and when the model decides to call it, the arguments arrive as a JSON object matching that schema.

tools = [{
    "name": "book_flight",
    "description": "Book a flight for a passenger",
    "strict": True,
    "input_schema": {
        "type": "object",
        "properties": {
            "destination": {"type": "string"},
            "date": {"type": "string", "format": "date"},
            "passengers": {"type": "integer", "enum": [1, 2, 3, 4]},
        },
        "required": ["destination", "date", "passengers"],
        "additionalProperties": False,
    },
}]

response = client.messages.create(
    model="claude-opus-4-8",
    max_tokens=1024,
    tools=tools,
    messages=[{"role": "user", "content": "Book a flight to Tokyo for 2 on March 15"}],
)

The strict: true field on the tool definition (not on tool_choice) is the key line here. It guarantees the tool_use.input block validates exactly against the schema, the same decoding-time enforcement as structured outputs, just scoped to one tool's arguments instead of the whole response. Without strict: true, tool inputs are usually close to schema-valid but not guaranteed, especially on smaller or older models.

Tool calling and dedicated structured-output parameters solve overlapping but distinct problems. Use structured output when you always want the same shape back: an extraction schema, a classification result, a form. Use tool calling when the model needs to decide *which* action to take, or when you're building an agent loop where the "output" is really a function call your code executes.

Forcing tool choice for guaranteed structured output

By default, tool-enabled models decide for themselves whether to call a tool or just respond in text. If you're using tools purely as a structured-output mechanism, that ambiguity is a bug: you don't want the model choosing to explain itself in prose when you specifically need the extract_fields tool called every time. tool_choice removes that ambiguity. The options:

{"type": "auto"}: the model decides whether to call a tool. This is the default and the wrong setting if you always need structured output.
{"type": "any"}: the model must call some tool, but picks which one if you've defined several.
{"type": "tool", "name": "extract_fields"}: the model must call this specific tool. This is the setting you want for pure extraction and classification tasks.
{"type": "none"}: tools are disabled for this turn even if defined.

Forcing a specific tool with {"type": "tool", "name": "..."} is the closest thing to a hard contract you can put on a model call: it must produce a tool_use block for that tool, and if strict: true is also set, that block's arguments must validate against your schema. This combination (single defined tool, forced choice, strict schema) is functionally equivalent to dedicated structured output for cases where a tool-call shape fits your use case, and it's the pattern most teams reach for when their provider doesn't yet have a first-class output_config.format-style parameter.

One caveat worth knowing if you're mixing forced tool choice with extended thinking or chain-of-thought reasoning modes: some providers require you to disable reasoning when forcing a specific tool, because forced-tool responses skip the reasoning step entirely. Check your provider's docs for that interaction before shipping it, since it's a common source of a confusing 400 error that has nothing to do with your schema.

Designing schemas that actually work

A schema that's technically valid JSON Schema can still produce bad extraction results. A few practices that consistently help:

Keep field names literal and unambiguous. amount_cents beats amount when currency handling is ambiguous, because it removes a decision the model would otherwise have to infer.
Use enum wherever the set of valid values is fixed. This is both a correctness win (the model can't invent "maybe" for a boolean-shaped field) and, on schema-constrained decoding, an efficiency win, since the model's choices are literally restricted to the enum values.
Avoid deeply nested optional structures. If a field is genuinely optional, prefer a flat schema with null allowed over conditionally-present keys; conditional presence is one of the more common sources of subtly-wrong extractions.
Write field descriptions like you're briefing a new teammate, not documenting an API. "The customer's stated reason for cancellation, verbatim" gets better results than "reason: string".
Don't try to encode business logic in the schema. additionalProperties: false and required are structural constraints; "the discount must not exceed the subtotal" is a validation rule that belongs in your code after the response comes back.

If you're extracting the same shape across many documents (invoices, resumes, support tickets), version your schema like you'd version an API contract. A schema change that adds a required field breaks every prompt template and few-shot example that assumed the old shape, and that failure mode is invisible until a downstream consumer chokes on a null.

Validating what comes back, even with structured output

"Guaranteed to be valid JSON matching the schema" is not the same as "guaranteed to be correct." Schema-constrained decoding guarantees structure, not semantics. The model can still return {"amount_cents": 0} when it couldn't find an amount, rather than surfacing that as an error, because your schema said amount_cents was required and an integer. This is the most common way structured output silently produces bad data: the model satisfies the letter of the schema by picking a plausible-looking default instead of telling you it doesn't know.

Two mitigations that work in practice:

Make "unknown" a valid schema value instead of forcing a field to always resolve. If a field can legitimately be absent from the source document, make it nullable in the schema and instruct the model explicitly to return null rather than guess. A schema that has no escape hatch for "not present" all but guarantees the model will hallucinate a placeholder to satisfy required.

Run business-rule validation as a separate pass after the schema check. Use Pydantic validators, Zod refinements, or plain assertions to catch things the schema can't express: totals that don't sum, dates in the future for historical records, enum values that are valid JSON Schema but nonsensical in context. Treat schema validation and business validation as two different layers with two different failure modes: one is a hard API-level guarantee, the other is your domain knowledge checking the model's work.

from pydantic import BaseModel, field_validator

class Invoice(BaseModel):
    vendor: str
    amount_cents: int
    currency: str

    @field_validator("amount_cents")
    @classmethod
    def amount_must_be_positive(cls, v):
        if v <= 0:
            raise ValueError("amount_cents must be positive")
        return v

try:
    invoice = Invoice.model_validate_json(raw_json)
except ValidationError as e:
    # retry with the validator's error message fed back to the model,
    # or route to a human review queue
    ...

Handling refusals and partial output

Structured output requests can still fail to produce content at all. A model can refuse (safety-related decline), hit max_tokens mid-object, or run into a context-window limit. Code that assumes response.content[0] always has your JSON will throw an index error on any of these, so check stop_reason before you touch the content:

if response.stop_reason == "refusal":
    # empty or partial content, handle as a content outcome, not a parse failure
    ...
elif response.stop_reason == "max_tokens":
    # output was truncated: the "valid JSON" guarantee does not survive truncation
    # retry with a higher max_tokens, or stream and reassemble
    ...
elif response.parsed_output is not None:
    use(response.parsed_output)

Two things worth internalizing here. First, stop_reason == "max_tokens" means your object was cut off mid-generation regardless of what schema you specified; the enforcement guarantees each *token* is schema-consistent as it's generated, not that the object was ever allowed to finish. Give structured-output calls a generous max_tokens, especially for schemas with arrays or long free-text fields nested inside them. Second, citations and structured outputs are commonly mutually exclusive on APIs that support both; if you need cited sources and a fixed schema in the same call, check whether your provider requires picking one.

Streaming structured output

If you're extracting a large object (a long document summary with many fields, or an array of many items) and want to render progress instead of waiting for the whole response, you can stream structured output the same way you stream free text, then reassemble and validate once the stream completes. Don't attempt to JSON.parse() partial streamed content; it will fail on every chunk except the last one by definition. Instead, either:

Buffer the full stream and parse once at the end (simplest, and the schema guarantee only holds on the complete object anyway), or
Use an incremental JSON parser built for streaming (most languages have one) if you need a progressively-rendering UI, and treat any partial state as provisional until the stream's final event confirms completion.

For agentic loops where the model calls a tool, your code executes it, and the loop continues, most SDKs ship a tool-runner helper that manages this streaming-plus-looping logic for you rather than requiring you to hand-roll message-array bookkeeping. Reach for that before writing your own loop; the manual version is easy to get subtly wrong, especially around when to append the full response content (including tool_use blocks) back into the conversation.

Common mistakes

Retrying without telling the model what was wrong. If a response fails your business-rule validation, don't just re-send the identical prompt. Feed the validation error back into the retry as context ("the previous response had amount_cents: 0, which failed the must-be-positive check; if the amount isn't stated in the document, return null instead"). A blind retry on an identical prompt will often produce the identical wrong answer.

Encoding format instructions in the prompt when the API has a schema parameter. If your provider supports output_config.format or response_format, use it instead of "Respond only in valid JSON matching this schema: {...}" as prose. The API-level parameter is decoding-time enforcement; the prose instruction is a strong suggestion the model can still drift from, especially over a long response.

Skipping `additionalProperties: false`. Without it, a model that's slightly unsure what field name you wanted can add a plausible extra key alongside the real one, and code that does data["field"] still works while quietly ignoring the model telling you it wasn't confident.

Forcing tool choice with more than one tool defined. {"type": "tool", "name": "X"} forces tool X regardless of how many other tools exist in the tools array; unused tool definitions in that request just add tokens for no benefit. If you truly need pure extraction, define exactly one tool for that call.

Assuming the schema guarantee survives model or provider switches. A schema that validates cleanly against one model's structured-output implementation isn't automatically portable; JSON Schema support differs across models and versions (recursive schemas, numeric constraints, and format keywords are common points of divergence). Re-validate your schema against the target model any time you migrate.

FAQ

What's the difference between JSON mode and structured outputs? JSON mode (an older, looser feature on some APIs) guarantees the output is syntactically valid JSON, but not that it matches any particular shape. Structured outputs (schema-constrained generation) guarantees the output validates against a specific JSON Schema you provide. If your API offers both, prefer structured outputs whenever you know the shape in advance; JSON mode is only useful when you genuinely don't have a fixed schema.

Should I use tool calling or dedicated structured output for extraction tasks? Use dedicated structured output (output_config.format or your provider's equivalent) when there's exactly one shape you always want back. Use a forced single tool call when your workflow is already built around tool use, or when you want the extraction to look identical in your code to every other tool invocation in an agentic loop. Functionally they converge to the same guarantee when you force a single tool with strict: true.

Does structured output cost more? Schema-constrained generation typically has a small first-request cost for the API to compile a new schema, cached for reuse afterward, and otherwise bills the same as a normal request based on tokens generated. It's not a separate pricing tier: you're paying for the tokens either way. What it actually saves you is retry volume from malformed-JSON failures, which is usually the bigger cost in practice.

Can I combine structured output with extended thinking or reasoning modes? Often yes, but check your provider's compatibility notes: some combinations (forced tool choice plus a reasoning mode, or citations plus a fixed schema) are explicitly incompatible and return a 400 rather than silently degrading. Test the specific combination you need before building around it.

What happens if the model can't extract a field the schema marks as required? This is the single most common way structured output produces silently bad data: a required field with no legitimate value gets filled with a plausible placeholder instead of an error. The fix is schema design, not validation: make fields nullable where absence is a legitimate outcome, and instruct the model explicitly to prefer null over guessing.

Is structured output the same as function calling? They're related but not identical. Function calling (tool use) is about the model deciding to invoke one of several defined actions, with arguments matching that action's schema. Structured output is about constraining the model's entire response to one fixed shape, regardless of "action." You can use tool calling to achieve structured output by forcing a single tool, but tool calling's real purpose is letting the model choose among several possible actions, which is a different use case.

Do I still need to validate the response if the API guarantees schema conformance? Yes. Schema conformance is a structural guarantee: correct types, correct required fields, no extra properties. It says nothing about whether the values are correct. Always run a second, business-rule validation pass (range checks, cross-field consistency, "is this plausible") on top of schema validation.