Controlling LLM Output Formatting with Prompts

Prompt output formatting is the discipline of making a language model return text in a shape your code can parse reliably, every single time, not just most of the time. If you have ever piped a model's response into json.loads() and watched it crash on a stray sentence like "Sure, here's the JSON you asked for", you already understand the problem this article solves. The fix is not one trick, it is a small stack of techniques: explicit schemas, delimiters, structured output modes, few-shot examples, and a validation layer that catches the rare miss. Stack all four and you get output that behaves like an API response instead of a chat transcript.

This matters more in 2026 than it did a couple of years ago because agents now chain model calls together. One malformed field breaks the next step, and a pipeline of five tool calls has five chances to fail on formatting alone. Getting this right early saves you from debugging "why did the agent stop" tickets later.

Why models drift away from your requested format

Language models are trained to be helpful conversationalists first. Left alone, a model wants to explain itself, add caveats, and wrap answers in friendly prose. When you ask for "just the JSON", you are fighting that training pull. Three things make drift worse:

• Long system prompts bury the format instruction. If your format rule is buried in paragraph four of a ten-paragraph system prompt, the model weighs it against everything else it read.
• Ambiguous examples. If your few-shot examples show slightly different formatting between them, the model averages the inconsistency into its own output.
• No penalty for getting it wrong. Unless you validate and retry, a model has no way to know its output broke your parser. It only "knows" if you tell it, either in the prompt or through a correction loop.

The rest of this article is about removing that ambiguity at every layer: the instruction, the example, the generation mechanism, and the safety net.

Start with the simplest lever: explicit format instructions

Before reaching for structured output APIs, try being ruthlessly explicit in the prompt itself. Vague requests produce vague compliance.

Weak:

Summarize this article and list the key points.

Strong:

Summarize the article in exactly 2 sentences under a "Summary" heading.
Then list 3 to 5 key points as a markdown bulleted list under a "Key Points" heading.
Do not include any text before "Summary" or after the last bullet.

The strong version removes every decision the model would otherwise have to make: how many sentences, how many bullets, what comes first, what comes last, whether to add a closing remark. Every decision you make for the model is a decision it cannot get wrong.

A useful pattern is to state the format as a numbered contract at the end of the prompt, right before generation starts:

Output format:
1. First line: the category, one of [bug, feature, question]
2. Second line: a one-sentence summary
3. Remaining lines: a markdown bulleted list of affected files
No other text.

Putting format rules last, right next to where generation begins, keeps them fresh in the model's attention window.

Use delimiters to make sections unambiguous

When you need the model to separate reasoning from the final answer, or separate multiple outputs in one response, delimiters remove any doubt about where one section ends and another begins.

Analyze the customer message below. Put your reasoning between
<reasoning></reasoning> tags and your final classification between
<answer></answer> tags. Only the content inside <answer> will be parsed,
so it must contain nothing but the single word: refund, complaint, or praise.

Customer message: "This is the third time my order has arrived broken."

XML-style tags work well because they are visually distinct from markdown and from natural prose, so the model rarely emits stray angle brackets by accident. On the parsing side, a simple regex or an XML-aware string extractor pulls the answer out cleanly:

import re

def extract_tag(text: str, tag: str) -> str:
    match = re.search(rf"<{tag}>(.*?)</{tag}>", text, re.DOTALL)
    if not match:
        raise ValueError(f"Missing <{tag}> block in model output")
    return match.group(1).strip()

answer = extract_tag(response_text, "answer")

Triple backtick fences work the same way for code or JSON blocks, and most model providers are well trained to keep code fences clean since so much of their training data is documentation and Stack Overflow answers.

Reach for structured output modes when you need JSON

Explicit instructions get you most of the way, but for anything feeding a downstream program, use the provider's native structured output feature instead of hoping the model formats JSON correctly on its own. Both major API families now support this directly.

With OpenAI-style APIs, you pass a JSON Schema and the API guarantees the response conforms to it:

from openai import OpenAI
from pydantic import BaseModel

client = OpenAI()

class TicketClassification(BaseModel):
    category: str
    priority: str
    summary: str

response = client.chat.completions.parse(
    model="gpt-4.1",
    messages=[
        {"role": "system", "content": "Classify the support ticket."},
        {"role": "user", "content": "My login keeps failing after the password reset."},
    ],
    response_format=TicketClassification,
)

ticket = response.choices[0].message.parsed
print(ticket.category, ticket.priority)

With Anthropic's Claude models, the equivalent pattern uses tool use (function calling) to force a structured response, since Claude treats a "tool" definition as a schema it must satisfy:

import anthropic

client = anthropic.Anthropic()

tools = [
    {
        "name": "record_ticket",
        "description": "Record the classified support ticket",
        "input_schema": {
            "type": "object",
            "properties": {
                "category": {"type": "string", "enum": ["billing", "bug", "account"]},
                "priority": {"type": "string", "enum": ["low", "medium", "high"]},
                "summary": {"type": "string"},
            },
            "required": ["category", "priority", "summary"],
        },
    }
]

response = client.messages.create(
    model="claude-sonnet-4-5",
    max_tokens=1024,
    tools=tools,
    tool_choice={"type": "tool", "name": "record_ticket"},
    messages=[
        {"role": "user", "content": "My login keeps failing after the password reset."}
    ],
)

tool_call = next(block for block in response.content if block.type == "tool_use")
ticket = tool_call.input
print(ticket["category"], ticket["priority"])

Forcing tool_choice to a specific tool is the key move here. It stops the model from responding in prose and routes generation directly through the schema, which is enforced at decode time rather than hoped for through instructions.

The advantage of native structured output over prompt-only formatting is that the constraint is applied during token generation, not just requested in the prompt. That means enum fields cannot contain values outside the allowed list, required fields cannot be omitted, and types cannot be swapped. Prompt instructions alone cannot guarantee any of that.

Few-shot examples still matter, even with structured output

Schemas control shape, they do not control judgment. If you want consistent tone, consistent level of detail, or consistent handling of edge cases, show examples. Two or three examples, each demonstrating a slightly different input, teach the model the pattern faster than any amount of explanation.

Example 1
Input: "App crashes on startup after update"
Output: {"category": "bug", "priority": "high", "summary": "App crashes on startup post-update"}

Example 2
Input: "Can I get an invoice for last month?"
Output: {"category": "billing", "priority": "low", "summary": "Customer requests prior month invoice"}

Now classify:
Input: "I was charged twice for the same subscription"

Keep examples short and keep the field order identical across every example. Inconsistent field order in your few-shot examples is a surprisingly common cause of inconsistent field order in the model's own output.

Control randomness so formatting does not wobble between calls

Temperature affects word choice more than structure, but at higher temperatures a model is more likely to take a creative detour, like adding an extra explanatory sentence or renaming a field. For formatting-sensitive tasks, set temperature low, often 0 to 0.2, and let your prompt do the creative work elsewhere in the pipeline.

If your provider exposes a stop parameter, use it to hard-cut generation right after the structured content ends, which prevents trailing commentary even if the model wants to add some:

response = client.chat.completions.create(
    model="gpt-4.1",
    messages=messages,
    temperature=0,
    stop=["\n\n"],
)

This is a blunt tool, so test it against your actual outputs. If your JSON legitimately contains double newlines inside a string field, a stop sequence like this will truncate mid-response.

Prefill the response to skip the preamble entirely

Some APIs let you seed the start of the assistant's turn, which is one of the most reliable formatting tricks available. If the assistant's response already begins with {, the model has no room to write "Here is the JSON you requested" because that text would come after the opening brace, which breaks the pattern it was trained to continue naturally.

response = client.messages.create(
    model="claude-sonnet-4-5",
    max_tokens=1024,
    messages=[
        {"role": "user", "content": "Extract name and age as JSON from: John is 34 years old."},
        {"role": "assistant", "content": "{"},
    ],
)
full_json = "{" + response.content[0].text

Prefilling works because the model treats everything in the conversation history, including the partial assistant turn you supply, as already-committed output it must continue coherently. This trick is especially useful for models or SDKs that do not offer a native structured output mode.

Validate and retry instead of trusting blindly

Even with schemas, delimiters, and prefilling, treat model output the way you would treat any external API response: validate it before using it. A short validate-then-retry loop catches the rare malformed response without a human ever noticing.

import json

def get_structured_response(prompt, schema_model, max_retries=2):
    messages = [{"role": "user", "content": prompt}]
    for attempt in range(max_retries + 1):
        raw = call_model(messages)
        try:
            data = json.loads(raw)
            return schema_model(**data)
        except (json.JSONDecodeError, TypeError, ValueError) as e:
            if attempt == max_retries:
                raise
            messages.append({"role": "assistant", "content": raw})
            messages.append({
                "role": "user",
                "content": f"That output failed validation: {e}. Return only valid JSON matching the schema, nothing else.",
            })

Feeding the validation error back to the model, rather than just retrying the same prompt, meaningfully improves the second attempt's success rate. The model can see exactly what it did wrong instead of guessing again in the dark.

Libraries like instructor (Python) wrap this retry loop around Pydantic models automatically, and zod combined with a thin retry wrapper does the equivalent job in TypeScript projects. Either approach saves you from hand-rolling the same try-except loop across every call site in a codebase.

Formatting patterns for common output shapes

A few shapes come up constantly enough to be worth having on hand.

CSV-style rows for tabular data, when you specifically do not want markdown tables (since markdown-lite renderers often reject them):

Output one row per line, comma-separated, no header row:
name,category,price

Numbered steps for procedural output, useful for anything that gets rendered as instructions:

Output only a numbered list. Each item must be a single imperative sentence
under 15 words. Do not add sub-bullets.

Key-value blocks for quick extraction without full JSON overhead:

Output exactly these three lines, nothing else:
NAME: <value>
DATE: <value>
STATUS: <value>

Key-value blocks are worth remembering because they parse trivially with a simple line split, they are easy for a model to produce correctly even at higher temperatures, and they avoid the bracket-matching failures that plain JSON without a structured output mode sometimes produces.

Testing your format prompts before shipping

Treat formatting prompts like code: write a small test harness that runs the prompt against ten or twenty representative inputs and checks that every single response parses. A prompt that works on your first three manual tests can still fail one time in twenty once real user input starts hitting it.

def test_format_consistency(prompt_template, test_inputs, parser_fn):
    failures = []
    for input_text in test_inputs:
        raw = call_model(prompt_template.format(input=input_text))
        try:
            parser_fn(raw)
        except Exception as e:
            failures.append((input_text, raw, str(e)))
    return failures

Run this whenever you change the prompt, the model version, or the schema. Formatting reliability is not a one-time property of a prompt, it shifts when any of those three things change underneath it.

FAQ

Do I still need prompt instructions if I use structured output mode? Yes. Structured output modes constrain the shape (field names, types, enums) but they do not control tone, level of detail, or how the model handles ambiguous input. Keep clear instructions in the prompt and use structured output for the hard shape guarantee.

Why does my model add a sentence before the JSON even though I told it not to? This usually happens with prompt-only formatting on models that were not forced through a schema or tool call. Try prefilling the response with { or switching to a native structured output feature, both of which prevent the model from generating any text before the structured content starts.

Is markdown a reliable output format for LLMs? Markdown headings and bullet lists are reliable because they mirror common training data patterns. Markdown tables are less reliable, since column alignment and cell counts drift more easily, especially with longer tables. Prefer key-value blocks or CSV-style rows over tables when precision matters.

Should I set temperature to 0 for all formatting-sensitive tasks? Not always. Temperature 0 maximizes consistency but can make some models loop or repeat on certain inputs. A low but nonzero value like 0.1 to 0.2 is often a safer default, combined with a schema or delimiter strategy that does the real formatting enforcement.

What is the fastest way to debug a formatting failure in production? Log the raw, unparsed model response alongside the parse error, not just the error itself. Most formatting bugs are visible at a glance once you see the actual text, whether it is a missing closing brace, an extra explanatory sentence, or a field name that does not match your schema's casing.

Can few-shot examples alone replace structured output modes? For low-stakes internal tools, sometimes. For anything feeding a parser in production, no. Few-shot examples improve the model's judgment about content, but only a schema-enforced structured output mode or a strict validation-and-retry loop gives you a hard guarantee the output will parse.