Prompt Templates for AI Agents

Agent prompt templates are the reusable, parameterized instructions that turn a raw language model into a consistent, testable agent. Instead of hand-typing a fresh prompt every time you spin up an agent, you write one template with slots for the task, the tools, the memory, and the constraints, then fill those slots at runtime. Get the template right and your agent behaves the same way on Monday as it does three months later after the model provider ships an update. Get it wrong and you end up debugging "vibes" instead of code. This guide walks through the anatomy of a good agent prompt template, shows working code for the most common agent shapes, and covers how to version and test templates like any other production asset.

What Are Agent Prompt Templates, Really

A prompt template is not just a string with {variable} placeholders. For an agent, it is a small contract between three things: the system's role definition, the tools the model is allowed to call, and the state the agent carries between steps. A plain chatbot prompt only needs to answer a question. An agent prompt template needs to also tell the model how to decide when it is done, what format to respond in when it wants to call a tool, and what to do when a tool call fails.

Treat the template as configuration, not as prose you improvise. That means:

• It lives in a file (.txt, .md, .yaml, or a Python string constant), not inline in a chat window.
• It has named variables with defaults, so a missing variable fails loudly instead of silently degrading output quality.
• It is versioned in git, so you can diff behavior changes against prompt changes.
• It is testable, so a regression in reasoning quality shows up in CI, not in a support ticket.

Core Anatomy of an Agent Prompt Template

Every agent prompt template, regardless of framework, tends to have the same six sections. Skipping any of them is usually where agents start hallucinating tool names or looping forever.

1. Identity and scope , who the agent is and, just as important, what it must refuse to do.
2. Tool manifest , the exact names, parameters, and return shapes of every tool available this turn.
3. Operating loop , the reasoning pattern (plan, act, observe, repeat) and the stop condition.
4. Context injection , retrieved documents, conversation history, or user profile data.
5. Output contract , the exact format the model must return, usually JSON or a structured tool call.
6. Guardrails , what to do on ambiguous input, missing data, or a failed tool call.

Here is a minimal template that captures all six sections using Python's built-in string.Template, which avoids the curly-brace collisions you get with f-strings when your output contract itself contains JSON:

from string import Template

AGENT_TEMPLATE = Template("""
You are $agent_name, an agent that helps with $domain tasks.
You must never fabricate data you cannot retrieve from a tool.

## Available tools
$tool_manifest

## Operating loop
1. Read the user request.
2. Decide if you need a tool. If yes, emit exactly one tool call.
3. After receiving a tool result, decide if you have enough information to answer.
4. If not enough information, call another tool. Do not call the same tool
   with the same arguments twice in a row.
5. Stop and answer once you have what you need, or after $max_steps steps,
   whichever comes first.

## Context
$context

## Output contract
Respond with a single JSON object matching this shape:
{"action": "tool_call" | "final_answer", "content": ...}

## Guardrails
- If a tool returns an error, retry once with corrected arguments, then
  report the failure to the user instead of guessing.
- If the user request is ambiguous, ask one clarifying question instead
  of assuming.
""")

Rendering it is a one-liner:

prompt = AGENT_TEMPLATE.substitute(
    agent_name="Ledger",
    domain="expense reconciliation",
    tool_manifest="- lookup_transaction(id: str) -> Transaction\n- flag_duplicate(id: str) -> bool",
    max_steps=6,
    context="User is reviewing March expenses for the marketing team.",
)
print(prompt)

string.Template raises a KeyError on .substitute() if you forget a variable, which is exactly the loud failure you want. If you prefer templates that tolerate missing keys with defaults, swap in .safe_substitute() during development and switch back to strict .substitute() for production builds.

Templates for Tool-Calling Agents

Most production agents today use native tool calling rather than free-text tool syntax, because the model provider handles the parsing and you get structured arguments back instead of regex-extracting them from prose. When you use native tool calling, your prompt template gets shorter, the tool manifest moves out of the prompt text and into a separate tools parameter, and the template's job shrinks to identity, operating loop, context, and guardrails.

Here is a working example using the Anthropic Python SDK's tool-use interface:

import anthropic

client = anthropic.Anthropic()

SYSTEM_TEMPLATE = Template("""
You are $agent_name. Use the provided tools to answer questions about
$domain. Only call a tool when you lack the information to answer
directly. Cite the tool result when you use one.
""")

tools = [
    {
        "name": "lookup_transaction",
        "description": "Fetch a single transaction by its ID.",
        "input_schema": {
            "type": "object",
            "properties": {"id": {"type": "string"}},
            "required": ["id"],
        },
    },
    {
        "name": "flag_duplicate",
        "description": "Check whether a transaction ID is a duplicate.",
        "input_schema": {
            "type": "object",
            "properties": {"id": {"type": "string"}},
            "required": ["id"],
        },
    },
]

def run_agent(user_message: str, agent_name="Ledger", domain="expense reconciliation"):
    system_prompt = SYSTEM_TEMPLATE.substitute(agent_name=agent_name, domain=domain)
    messages = [{"role": "user", "content": user_message}]

    while True:
        response = client.messages.create(
            model="claude-opus-4-6",
            max_tokens=1024,
            system=system_prompt,
            tools=tools,
            messages=messages,
        )

        if response.stop_reason != "tool_use":
            return response.content[0].text

        messages.append({"role": "assistant", "content": response.content})
        tool_results = []
        for block in response.content:
            if block.type == "tool_use":
                result = dispatch_tool(block.name, block.input)
                tool_results.append({
                    "type": "tool_result",
                    "tool_use_id": block.id,
                    "content": str(result),
                })
        messages.append({"role": "user", "content": tool_results})


def dispatch_tool(name, args):
    if name == "lookup_transaction":
        return {"id": args["id"], "amount": 42.50, "vendor": "AWS"}
    if name == "flag_duplicate":
        return False
    raise ValueError(f"Unknown tool: {name}")

Notice the template only carries the identity and behavioral rules. The tool schema itself is structured data, not a string you interpolate into the prompt. This separation matters: when a tool's parameters change, you edit the tools list, and your prompt template does not need a diff at all.

Templates for ReAct-Style Reasoning Agents

If you are not using a provider's native tool-calling feature, or you want the model to show its reasoning trace explicitly, a ReAct-style template (Reason, Act, Observe) is the standard pattern. This is common when building agents on top of open-weight models without structured tool support, or when you want a human-readable transcript for debugging.

REACT_TEMPLATE = Template("""
Answer the following question using the Thought/Action/Observation loop.

Available actions:
$actions

Format strictly as:
Thought: <your reasoning>
Action: <action_name>[<action_input>]
Observation: <result, filled in by the system>
... (repeat Thought/Action/Observation as needed)
Thought: I now know the final answer.
Final Answer: <answer>

Question: $question
""")

actions_block = "\n".join([
    "search[query] - search the knowledge base for a query",
    "calculate[expression] - evaluate a math expression",
])

prompt = REACT_TEMPLATE.substitute(actions=actions_block, question="What was Q1 revenue growth compared to Q4?")

The parsing side of a ReAct template is where most bugs live, so write it once as a shared utility instead of re-implementing it per agent:

import re

def parse_react_step(text: str):
    action_match = re.search(r"Action:\s*(\w+)\[(.*?)\]", text)
    final_match = re.search(r"Final Answer:\s*(.*)", text, re.DOTALL)
    if final_match:
        return {"type": "final", "content": final_match.group(1).strip()}
    if action_match:
        return {
            "type": "action",
            "name": action_match.group(1),
            "input": action_match.group(2),
        }
    raise ValueError(f"Could not parse ReAct step: {text!r}")

Keep the regex tolerant of trailing whitespace and multi-line final answers, and always log the raw model output when parsing fails so you can see whether the template's formatting instructions were unclear rather than guessing blind.

Templates for RAG Agents

Retrieval-augmented agents need a template section dedicated to injected context, and the template has to make it unambiguous which text came from retrieval versus which came from the user. Blurring that line is the single most common cause of an agent citing a source it never actually saw.

RAG_AGENT_TEMPLATE = Template("""
You are a support agent for $product. Answer only using the retrieved
documents below. If the documents do not contain the answer, say you
don't know and offer to escalate to a human.

## Retrieved documents
$retrieved_docs

## Conversation so far
$history

## User question
$question

Cite the document title in parentheses after any claim you make from it.
""")

def format_docs(docs):
    return "\n\n".join(f"[{d['title']}]\n{d['excerpt']}" for d in docs)

prompt = RAG_AGENT_TEMPLATE.substitute(
    product="TeachYou billing",
    retrieved_docs=format_docs([
        {"title": "Refund policy", "excerpt": "Refunds are processed within 5 business days."},
    ]),
    history="User: How long do refunds take?",
    question="Can I get a refund after 30 days?",
)

Keep the retrieved-documents block bounded. Truncate each excerpt to a fixed character length inside format_docs, and cap the number of documents you inject, so the template's token budget stays predictable even when your retriever returns more hits than expected.

Variable Injection with Jinja2 for Complex Templates

string.Template is fine for flat variable substitution, but once your template needs conditionals (show the tool manifest only if tools are enabled) or loops (render a variable number of few-shot examples), reach for Jinja2. It is the templating engine most prompt-management libraries use under the hood.

from jinja2 import Environment, BaseLoader

env = Environment(loader=BaseLoader())

TEMPLATE_SRC = """
You are {{ agent_name }}, an agent for {{ domain }}.

{% if examples %}
## Examples
{% for ex in examples %}
Input: {{ ex.input }}
Output: {{ ex.output }}
{% endfor %}
{% endif %}

{% if tools %}
## Tools
{% for tool in tools %}
- {{ tool.name }}: {{ tool.description }}
{% endfor %}
{% endif %}

Now respond to: {{ question }}
"""

template = env.from_string(TEMPLATE_SRC)

rendered = template.render(
    agent_name="Ledger",
    domain="expense reconciliation",
    examples=[{"input": "Flag TXN-882", "output": "TXN-882 is a duplicate of TXN-870."}],
    tools=[{"name": "flag_duplicate", "description": "Checks for duplicate transactions."}],
    question="Is TXN-991 a duplicate?",
)

Jinja2's {% if %} blocks let one template file serve multiple agent configurations (with tools, without tools, with few-shot examples, zero-shot) instead of maintaining four near-identical template files that drift apart over time.

Versioning and Testing Prompt Templates

Once a template ships to production, changing it is a code change, not a copy-paste edit in a chat window. Two practices keep this sane.

Store templates with a version identifier. A simple convention is a filename suffix or a version field in a YAML front matter block at the top of the template file:

version: 3
changelog: "Added guardrail against calling the same tool twice in a row"

Log the template version alongside every agent run so that when behavior regresses, you can bisect which template version introduced it, the same way you would bisect a code commit.

Write eval cases, not just unit tests. A prompt template test does not check for exact string output, it checks for properties: did the agent call the right tool, did it stop within the step budget, did it avoid a banned phrase. Here is a lightweight eval harness pattern:

import json

EVAL_CASES = [
    {
        "question": "Flag transaction TXN-100 as duplicate if it is one.",
        "expect_tool": "flag_duplicate",
        "expect_tool_input_contains": "TXN-100",
    },
    {
        "question": "What is the capital of France?",
        "expect_no_tool_call": True,
    },
]

def run_eval(agent_fn, cases):
    results = []
    for case in cases:
        trace = agent_fn(case["question"])
        passed = True
        if "expect_tool" in case:
            passed = any(step.get("name") == case["expect_tool"] for step in trace)
        if case.get("expect_no_tool_call"):
            passed = not any(step.get("type") == "tool_call" for step in trace)
        results.append({"question": case["question"], "passed": passed})
    return results

Run this eval set against every template version before rollout, and store the results next to the template's changelog entry. This turns "the agent feels worse since last week" into a diffable, reproducible signal.

Common Mistakes When Templating Agent Prompts

• Baking secrets or user PII directly into the template file. Templates get committed to git. Inject sensitive values at render time from environment variables or a secrets manager, never as a literal default in the template string.
• Letting the tool manifest drift from the actual tool implementations. If you hand-write the tool list inside a prompt string instead of generating it from your tool registry, it will eventually go stale. Generate the manifest text programmatically from the same schema you pass to the API.
• No stop condition. Every operating loop section needs an explicit max-steps or max-tool-calls limit. Without one, a confused agent will loop until it hits the provider's token limit and burns your budget doing it.
• Mixing instructions and data in the same block. If retrieved documents or user input sit in the same paragraph as your instructions, a crafted document can override your guardrails. Keep instructions in the system prompt and untrusted content in clearly delimited, separately labeled sections.
• Testing only the happy path. Add eval cases for tool failures, empty retrieval results, and ambiguous questions, not just the cases where everything works.

FAQ

What is the difference between a prompt template and a system prompt? A system prompt is one rendered instance of a prompt template, the fixed instruction block sent with every request. The template is the reusable source file with variable slots; the system prompt is what you get after filling those slots for a specific run.

Should tool schemas live inside the prompt text or in a separate parameter? Prefer a separate structured parameter (like the tools argument in the Anthropic SDK) whenever the model provider supports native tool calling. It is more reliable to parse and keeps your prompt template shorter. Only inline tool descriptions as text when you are working with a model that lacks structured tool-calling support.

How do I handle templates that need to support multiple LLM providers? Split the template into provider-agnostic sections (identity, operating loop, guardrails) and a provider-specific adapter layer that maps your internal tool schema into whatever shape each provider's API expects. Keep the prose identical across providers so behavior differences are easier to attribute to the model rather than to wording drift.

How long should an agent prompt template be? As short as it can be while still covering the six core sections. Longer is not safer. Every extra paragraph is something the model has to weigh against your actual instructions, and bloated templates make eval failures harder to diagnose because you cannot tell which sentence caused the regression.

Can I generate prompt templates automatically from a spec? Yes, and it is worth doing once you have more than a handful of agents. Define your agent's tools, guardrails, and stop conditions as structured data (a YAML or JSON spec), then render the template from that spec using Jinja2. This keeps every agent's template consistent in structure and makes bulk updates, like adding a new guardrail to every agent, a one-line change instead of an editing pass across a dozen files.

What is the best way to store prompt templates in a codebase? Put them in a dedicated prompts/ directory as plain files (.md or .j2), one file per agent or per operating loop variant, with a version field at the top. Import them the same way you import any other configuration, and keep them out of inline Python strings scattered across your codebase so a reviewer can see every prompt change in one diff.