Structured Outputs Explained: Getting Reliable JSON from LLMs

Pramod Dutta · Jul 1, 2026 · 15 min read

Why Your LLM Keeps Breaking Your JSON Parser

You asked the model for JSON. You even said "return only valid JSON, no markdown." And yet, three calls later, your parser throws because the model wrapped everything in a json markdown fence, added a cheerful "Here is your data:" preamble, or trailing-commaed an array into oblivion. If you have shipped anything on top of an LLM, you know this pain. The gap between "the model usually returns JSON" and "the model always returns parseable, schema-valid JSON" is where production systems go to die.

Structured outputs are the fix. They are a feature offered by every major model provider that guarantees the model's response conforms to a schema you define. Not "usually conforms." Guarantees. The model is constrained at the token level so it is literally incapable of emitting a token that would violate your schema. That single shift, from hoping to guaranteeing, is what makes LLMs usable as reliable components inside larger software instead of chatbots you babysit.

In this guide we will unpack what structured outputs actually are under the hood, how they differ from the old prompt-and-pray approach, how constrained decoding works, and how to wire them into real applications with concrete code. By the end you will know when to reach for them, what their limits are, and how to design schemas models can fill reliably. This is the same foundation we teach engineers who want to build serious AI products, not demos.

What Structured Outputs Actually Are

Structured outputs are a mode where you hand the model a schema, usually a JSON Schema, and the provider guarantees the returned string parses cleanly and matches that schema. Every required field is present. Every field has the declared type. Enums only ever contain allowed values. No prose, no fences, no apologies.

There are three distinct things people lump together under "structured outputs," and keeping them separate saves a lot of confusion:

JSON mode. The model is told to emit syntactically valid JSON. You get parseable JSON, but nothing enforces your specific shape. It might return {"answer": 42} when you wanted {"result": {"value": 42}}. Valid JSON, wrong structure.
Schema-constrained structured outputs. You supply a full JSON Schema and the provider constrains generation so the output matches it exactly. This is the strong guarantee. Types, required keys, enums, nesting, all enforced.
Tool or function calling. You define a function with a typed parameter schema, and the model returns matching arguments. Under the hood this uses the same machinery, just framed as "call this tool" instead of "fill this object."

The mental model to hold onto: a schema is a contract. In normal software, a function signature tells callers what to pass and what they get back. Structured outputs give an LLM the same kind of signature. Instead of parsing freeform text and praying, you get a typed object you can pass straight into the next function.

Here is the before-and-after. Without structured outputs, you write defensive parsing code that assumes the worst:

import json, re

raw = call_llm("Extract the name and age as JSON: 'Priya is 29'")
# the model often wraps JSON in a fenced code block, so strip the fence first
cleaned = strip_markdown_fences(raw).strip()
try:
    data = json.loads(cleaned)
    name = data.get("name")
    age = data.get("age")
except json.JSONDecodeError:
    # now what? retry? give up? log and move on?
    data = None

With structured outputs, the contract is enforced upstream and the defensive layer mostly evaporates. You describe the shape once and trust the result.

The Old Way: Prompt Engineering and Prayer

Before providers shipped native schema enforcement, everyone did the same dance. You wrote a prompt that begged the model to behave: "Respond ONLY with valid JSON. Do not include any explanation. Do not wrap in markdown. Use exactly these keys." You pasted an example output, set the temperature low, and added a small mountain of post-processing to clean up whatever came back anyway.

This approach fails in predictable, maddening ways:

Chatty preambles. "Sure! Here's the JSON you requested:" followed by the actual payload. Your parser chokes on the first character.
Markdown fences. The model wraps the object in a json code block because that is what it saw a million times in training data.
Trailing commas and single quotes. Syntactically invalid JSON that looks fine to a human but explodes json.loads.
Hallucinated or renamed keys. You asked for email and got email_address, or a flat object came back nested one level deeper.
Type drift. The age comes back as the string "29" instead of the number 29, or a boolean shows up as "true".
Silent schema violations. The worst case. The JSON parses fine, so your code proceeds, but a required field is missing and you get a None that corrupts data three functions downstream.

Teams patched these with regex extraction, retry loops, and validation libraries. A common pattern was "generate, validate, and if it fails, feed the error back and ask again," burning tokens and latency on every miss. It worked, sort of, at low volume. At scale it becomes a reliability tax you pay on every request, and the failure rate never quite reaches zero.

The fundamental problem is that prompting influences the model's probabilities but never removes the possibility of a bad token. As long as "H" (the start of "Here is...") has nonzero probability, the model can and eventually will pick it. Structured outputs attack the problem at that exact level.

How Constrained Decoding Works Under the Hood

To understand why structured outputs are a guarantee and not a suggestion, you need a quick picture of how an LLM generates text. At each step the model produces a probability distribution over its entire vocabulary, tens of thousands of possible next tokens. Normally it samples from that distribution, weighted by temperature, and appends the chosen token. Then it repeats.

Constrained decoding inserts a gate between "here are the probabilities" and "pick a token." The gate knows your schema. At every step it computes which tokens are legal given what has been generated so far, and zeroes out the probability of everything else before sampling. This is called masking. The model can only ever choose from tokens that keep the output valid.

Walk through generating {"age": 29} against a schema that says age is an integer:

The only legal first character is {. Every other token is masked to zero. The model emits {.
Now a key must follow. With one required property, age, the decoder can force the exact token sequence for "age":.
After the colon, the schema says integer, so digit tokens are allowed but a quote, a letter, or { are masked out. The model literally cannot start a string here.
Once the digits form a complete value, the closing } becomes legal and the decoder can require it.

The engine that enforces this is usually a compiled grammar or a finite state machine derived from your JSON Schema. Many implementations build a context-free grammar and track which productions are reachable at each position. Because the mask is applied to the raw logits before sampling, no invalid token can slip through. That is the source of the guarantee. It is not the model being well-behaved, it is the model being physically unable to misbehave.

A rough sketch of the idea in pseudocode makes it concrete:

def constrained_generate(model, grammar, prompt):
    tokens = []
    state = grammar.start_state()
    while not state.is_complete():
        logits = model.next_token_logits(prompt, tokens)
        # zero out any token that the grammar forbids right now
        allowed = grammar.allowed_tokens(state)
        for token_id in range(len(logits)):
            if token_id not in allowed:
                logits[token_id] = float("-inf")
        next_token = sample(logits)      # softmax over surviving tokens
        tokens.append(next_token)
        state = grammar.advance(state, next_token)
    return decode(tokens)

Two things fall out of this design. First, structured outputs cost almost nothing in extra latency because the masking is cheap relative to the forward pass. Second, the guarantee is only as good as the schema you provide. If your schema allows a field to be a string when you meant an email, the decoder happily lets through "not-an-email". Constrained decoding enforces shape, not semantics. That distinction matters and we will come back to it.

Defining Schemas That Models Can Actually Fill

The schema is where your leverage is. A well-designed schema makes the model's job easy and your downstream code safe. A sloppy one either over-constrains the model into nonsense or under-constrains it into uselessness. Here are the patterns worth internalizing.

Start with JSON Schema as your lingua franca. Most providers accept it directly, and it maps cleanly to types in every language. A basic object schema:

{
  "type": "object",
  "properties": {
    "title":    { "type": "string" },
    "priority": { "type": "string", "enum": ["low", "medium", "high"] },
    "estimate": { "type": "integer", "minimum": 1 },
    "tags":     { "type": "array", "items": { "type": "string" } },
    "assignee": { "type": ["string", "null"] }
  },
  "required": ["title", "priority", "estimate", "tags", "assignee"],
  "additionalProperties": false
}

Several deliberate choices are packed in there:

`enum` for closed sets. Priority can only be one of three values. The decoder will never emit "urgent" or "HIGH". Enums are the single highest-leverage constraint you can add. Any time a field has a fixed set of valid values, use one.
`additionalProperties: false`. This forbids the model from inventing extra keys. Without it, some models pad objects with helpful-looking junk. Many providers actually require this flag to be false for their strict structured-output mode to engage.
Everything in `required`. For strict modes, several providers mandate that every property be listed as required. To express "this field is optional," you make it required but allow null as a type, exactly like assignee above. That way the field always appears, and absence is modeled explicitly as null rather than a missing key. Your parsing code loves this because the shape is always identical.
Constraints like `minimum`. These add semantic guardrails on top of type. An estimate below 1 is impossible.

Beyond mechanics, a few design principles separate schemas that work from schemas that fight you:

Prefer flat over deeply nested. Every level of nesting is another place for the model to lose track of context. If you can express something as a flat object with a few well-named keys, do that instead of a five-deep tree.
Name fields the way a human would. Models were trained on human text. A field called customer_email is filled more reliably than ce or field_3. Descriptive names double as instructions.
Add `description` fields to guide content. Providers feed a property's description to the model, so {"type": "string", "description": "ISO 8601 date, e.g. 2026-07-06"} steers the value without any extra prompting.
Model uncertainty explicitly. If the model might not find an answer, give it a legal way to say so, such as a nullable field or an enum value like "unknown". Otherwise it will hallucinate a plausible value to satisfy the required field, which is worse than an honest null.

That last point is the one people learn the hard way. Constrained decoding forces the model to produce something that fits the schema. If the only schema-valid thing is a fabricated answer, you get a fabricated answer, confidently formatted. The schema is a shape contract, so design the shape to permit honesty.

Wiring Structured Outputs Into a Real App

Concepts are nice, but let us build something. Say you are ingesting free-text support tickets and want each one turned into a structured record: a category, a severity, a short summary, and whether it needs a human. The schema encodes exactly that, and the model fills it.

The provider-agnostic shape of the code looks like this. You define the schema, pass it in the request as the required response format, and get back a string you can parse with confidence:

import json

ticket_schema = {
    "type": "object",
    "properties": {
        "category": {
            "type": "string",
            "enum": ["billing", "bug", "feature_request", "account", "other"]
        },
        "severity": {
            "type": "string",
            "enum": ["low", "medium", "high", "critical"]
        },
        "summary": {
            "type": "string",
            "description": "One sentence, under 120 characters."
        },
        "needs_human": { "type": "boolean" }
    },
    "required": ["category", "severity", "summary", "needs_human"],
    "additionalProperties": False
}

def classify_ticket(client, ticket_text):
    response = client.generate(
        model="your-model-of-choice",
        messages=[
            {"role": "system", "content": "You triage support tickets."},
            {"role": "user", "content": ticket_text}
        ],
        response_format={
            "type": "json_schema",
            "json_schema": {"name": "ticket", "schema": ticket_schema, "strict": True}
        }
    )
    # The content is guaranteed to parse and match ticket_schema.
    return json.loads(response.output_text)

record = classify_ticket(client, "I was charged twice for my subscription this month!")
# record == {"category": "billing", "severity": "high",
#            "summary": "Customer double-charged for monthly subscription.",
#            "needs_human": True}

Notice what is absent. No regex. No fence stripping. No retry loop guarding against malformed JSON. The strict: True flag is what flips the provider into constrained decoding for that call. The parsed record drops straight into a database insert or a routing function.

For teams that live in typed languages, the ergonomics get even better. Many SDKs let you define the schema as a native class and receive a fully typed object back. A Pydantic-based flow in Python collapses the schema definition and the parsing into one step:

from pydantic import BaseModel
from typing import Literal

class Ticket(BaseModel):
    category: Literal["billing", "bug", "feature_request", "account", "other"]
    severity: Literal["low", "medium", "high", "critical"]
    summary: str
    needs_human: bool

def classify_ticket(client, ticket_text):
    response = client.parse(
        model="your-model-of-choice",
        messages=[{"role": "user", "content": ticket_text}],
        response_format=Ticket,   # SDK converts this to JSON Schema for you
    )
    return response.parsed   # already a Ticket instance, fully typed

ticket = classify_ticket(client, "The export button does nothing when I click it.")
print(ticket.category)     # "bug", with editor autocomplete and type checks

The class serves triple duty: it generates the JSON Schema sent to the model, validates the response, and gives your IDE full autocomplete on the result. This is the pattern that makes LLM calls feel like ordinary function calls. The model becomes a component with a signature, not a text oracle you interrogate.

Structured Outputs Versus Tool Calling

People often ask whether they should use structured outputs or function calling, as if they were rivals. They are close cousins sharing the same constrained-decoding engine, and the right choice comes down to intent.

Use plain structured outputs when the model's whole job is to return data. Extraction, classification, summarization into fields, converting messy text into clean records. There is one output, it is the answer, and you want it shaped. The response format is the deliverable.

Use tool or function calling when the model needs to decide whether and how to invoke external capabilities. It might call a function, might not, might call several in sequence, might ask a clarifying question first. The typed arguments to each tool are structured outputs under the hood, but the framing adds an orchestration layer: the model is choosing actions, not just filling one object.

A practical way to decide:

If you always want structured data back on every call, reach for structured outputs directly. It is simpler and more predictable.
If the model should choose among several typed operations, or decide to take no action, use tool calling and let each tool carry its own schema.
If you are building an agent that reasons over multiple steps and calls real functions, tool calling is the natural fit and its argument schemas give you the same reliability guarantee.

The important takeaway is that the underlying safety, guaranteed-valid, schema-conformant arguments, is present in both. You are choosing an ergonomic framing, not trading away reliability. Tool calling is structured outputs wearing a workflow hat.

Limits, Failure Modes, and How to Handle Them

Structured outputs are powerful, but treating them as magic will bite you. Know the edges.

Shape is guaranteed, correctness is not. The single most important caveat. Constrained decoding ensures the output matches the schema. It says nothing about whether the values are true. Ask for a person's birth year as an integer and the model will return a valid integer, which might be completely wrong. The schema stops malformed data, not hallucinated data. You still need evals and, for high-stakes fields, validation against a source of truth.

Over-constraining degrades quality. If your schema is so rigid that the correct answer does not fit, the model is forced to distort reality to comply. A classic example is forcing a single enum value when the real answer is "two of these apply." The fix is to loosen the schema to match reality: allow an array, add an "other" escape hatch, or make a field nullable. Design the shape around the truth, not the other way around.

Complex, deeply nested schemas strain the machinery. Very large schemas, deep recursion, or exotic JSON Schema features can slow grammar compilation or hit provider limits on schema size and nesting depth. Some advanced keywords are only partially supported. Keep schemas as simple as the task allows, and test the exact schema you plan to ship rather than assuming full spec coverage.

Semantic validation still belongs to you. Constraints like "this string is a valid email" or "this date is in the future" are mostly beyond what the schema enforces at decode time. Layer your own validation after parsing. The model gives you a well-shaped object; your code confirms the object makes sense.

A resilient production setup combines the guarantee with a thin safety net:

from pydantic import BaseModel, field_validator

class Invoice(BaseModel):
    amount: float
    currency: str
    due_date: str

    @field_validator("amount")
    @classmethod
    def positive_amount(cls, v):
        if v <= 0:
            raise ValueError("amount must be positive")
        return v

def extract_invoice(client, text, max_retries=2):
    for attempt in range(max_retries + 1):
        response = client.parse(
            model="your-model-of-choice",
            messages=[{"role": "user", "content": text}],
            response_format=Invoice,
        )
        try:
            # shape is guaranteed by the model; this catches semantic issues
            return Invoice.model_validate(response.parsed.model_dump())
        except ValueError:
            if attempt == max_retries:
                raise
            # optionally feed the error back into the next attempt
    return None

The structured output handles the shape so you never write JSON-repair code again. Your validators handle meaning. Together they give you data you can actually trust in a pipeline.

Putting It All Together

Structured outputs turn a fundamentally unreliable interface, freeform text from a probabilistic model, into a dependable one. By constraining generation at the token level against a schema you define, providers guarantee that what comes back parses cleanly and matches the shape you asked for. That moves LLMs out of the demo drawer and into production, where they behave as typed components with real signatures instead of chatbots you supervise.

The playbook is short. Define a JSON Schema, or a typed class that compiles to one. Use enums for closed sets, keep it flat, name fields like a human, and give the model a legal way to express uncertainty. Flip on strict mode so constrained decoding engages. Then add a thin layer of semantic validation for the correctness the schema cannot enforce. Do that and the whole category of "the model broke my parser" bugs simply disappears.

The deeper skill is knowing which tool fits which job: JSON mode for loose parseability, schema-constrained outputs for guaranteed shape, tool calling for orchestrating typed actions in an agent. Get that mental map right and you can build extraction pipelines, classifiers, and multi-step agents that behave predictably at scale.

If you want to go from understanding these ideas to shipping systems that use them well, that is the ground we cover in depth in the AI Engineering Roadmap course. It walks you through structured outputs, tool calling, evals, retrieval, and the full stack of skills for building LLM applications that hold up in the real world, with hands-on projects instead of toy snippets. Structured outputs are one of the first things that separate people who prototype with LLMs from people who ship with them. Learn them well, and everything you build on top gets sturdier.