Agent Version Control: Managing Prompt and Tool Changes Over Time

Why your agent broke and nobody noticed

Somewhere in your codebase there is a system prompt that got edited eleven times last quarter. Nobody tagged those edits. Nobody wrote down why the fourth edit happened, or what regressed after the seventh. If you asked your team right now "what did our support agent's prompt look like three weeks ago," the honest answer would be a shrug, a Slack search, and maybe a lucky screenshot in someone's DMs.

This is the state of most agent codebases today, and it is a strange one, because the same engineers who would never dream of deploying application code without git history, code review, and rollback plans will happily edit a 2,000-token prompt directly in a config file, push it straight to production, and call it a day. The prompt is code. The tool definitions are code. The model version pinned in your API call is a dependency. Yet almost nobody treats them that way.

Agent version control is the practice of applying the same rigor we apply to source code — versioning, diffing, testing, rollback, and staged rollout — to the three things that define an agent's behavior: prompts, tool schemas, and the orchestration logic that ties them together. It sounds obvious once stated, but the tooling and habits around it are still immature, which is exactly why teams keep getting burned. This article walks through what to version, how to structure it, how to test changes before they ship, and how to roll back fast when something goes wrong.

What actually needs to be versioned

An agent's behavior is not just the model. It is a stack of artifacts, and each layer changes at a different rate and breaks in a different way.

• System prompts and instructions — the highest-leverage, most frequently edited artifact. A single word change ("always" versus "usually") can flip agent behavior across thousands of conversations.
• Tool/function schemas — the JSON schemas describing what tools the agent can call, their parameters, and descriptions. Models read tool descriptions as part of the prompt, so a schema change is a prompt change in disguise.
• Few-shot examples — example conversations or outputs embedded in the prompt to steer style and format.
• Model and provider version — the underlying model itself is a dependency. Pinning claude-sonnet-5 versus letting it float to "latest" is a version control decision.
• Orchestration/routing logic — the code that decides which tools are available in which context, how many turns an agent gets, and what happens on tool failure.
• Retrieval configuration — for RAG-backed agents, the retrieval prompt, chunking strategy, and embedding model are all part of the behavioral surface.
• Guardrails and evaluators — the rules or classifier prompts that check agent output before it reaches a user.

Treat all seven as first-class versioned artifacts. If any one of them can change agent behavior in production, it belongs in source control with the same review process as your application code — not in a database row that an admin panel updates silently.

Structuring prompts and tools as code, not config blobs

The single biggest unlock is moving prompts out of database strings or admin-panel text boxes and into your repository as files. This sounds almost too simple to matter, but it changes everything downstream: diffs become visible, blame becomes possible, and code review becomes mandatory instead of optional.

A reasonable layout for a mid-size agent project:

agents/
  support-agent/
    v1/
      system_prompt.md
      tools.json
      few_shot_examples.json
    v2/
      system_prompt.md
      tools.json
      few_shot_examples.json
    CHANGELOG.md
    manifest.yaml

The manifest.yaml is the pointer that says which version is live, which is in canary, and which is deprecated:

agent: support-agent
active_version: v2
canary_version: v3
canary_traffic_pct: 5
deprecated_versions:
  - v1
model:
  provider: anthropic
  name: claude-sonnet-5
  pinned: true

Notice that the model is pinned explicitly and the field says so. This matters more than people expect — a provider shipping a new default model version can shift your agent's tone, verbosity, or tool-calling patterns overnight, with zero code changes on your end. If you are not pinning model versions, you are not actually in control of your own version history, no matter how carefully you version everything else.

Load the active prompt and tool schema at runtime keyed off the manifest, rather than hardcoding a path:

import yaml
import json
from pathlib import Path

def load_agent_config(agent_name: str, version: str | None = None):
    agent_dir = Path("agents") / agent_name
    manifest = yaml.safe_load((agent_dir / "manifest.yaml").read_text())
    version = version or manifest["active_version"]

    version_dir = agent_dir / version
    system_prompt = (version_dir / "system_prompt.md").read_text()
    tools = json.loads((version_dir / "tools.json").read_text())

    return {
        "version": version,
        "system_prompt": system_prompt,
        "tools": tools,
        "model": manifest["model"]["name"],
    }

This one function turns "which prompt is live" from a tribal-knowledge question into a git-blame-able fact. It also sets up everything that follows: canarying, rollback, and diffing.

Semantic versioning for behavior, not just syntax

Software teams use semver because "2.1.0 to 2.1.1" tells you something about risk before you even read the diff. Agent changes deserve the same signal, but the axes are different because the "interface" is natural language, not a function signature.

A useful adaptation for agent artifacts:

• Major (v1 to v2) — changes that alter what the agent is allowed to do: new tools added or removed, a fundamentally different persona or refusal policy, a different model family.
• Minor (v1.1 to v1.2) — changes that alter how the agent does its job without changing its scope: rewording instructions for clarity, adjusting tone, adding a new few-shot example, tightening a tool description.
• Patch (v1.1.0 to v1.1.1) — typo fixes, formatting cleanup, whitespace, anything with no intended behavioral effect.

The point of this scheme is not bureaucratic labeling — it is that the version number itself becomes a risk signal for reviewers and for your rollout system. A patch bump can auto-deploy. A major bump should require a human sign-off and a canary period, every time, no exceptions. Bake that policy into your CI so it isn't a judgment call under deadline pressure:

# .github/workflows/agent-deploy.yml (excerpt)
- name: Classify version bump
  run: |
    BUMP_TYPE=$(python scripts/classify_bump.py \
      --old agents/support-agent/v1 \
      --new agents/support-agent/v2)
    echo "bump_type=$BUMP_TYPE" >> "$GITHUB_OUTPUT"

- name: Require manual approval for major bumps
  if: steps.classify.outputs.bump_type == 'major'
  uses: trstringer/manual-approval@v1
  with:
    approvers: pramod-dutta,ira-menon

You don't need this to be fully automated on day one. Even a manual checklist that asks "does this change what tools the agent can call, or just how it phrases things" forces the right conversation before merge.

Diffing prompts like you diff code

Text diffs on prompts are noisy in a specific way: a single reworded sentence can look like a huge diff if you rewrapped the paragraph, and a tiny but critical change (deleting one constraint) can look like a one-line diff that's easy to skim past. Two practices help.

First, keep prompts in a stable, line-per-instruction format rather than dense paragraphs. A prompt written as a bulleted list of discrete instructions diffs far more cleanly than a wall of prose, because each line-level git diff maps to one behavioral change:

## Instructions
- Always confirm the user's account email before discussing billing details.
- Never quote a refund amount without checking the refund_policy tool first.
- If the user asks for a discount, escalate to a human agent instead of offering one.
- Keep responses under 150 words unless the user asks for more detail.

Second, run an automated "instruction diff" pass in CI that specifically flags deletions of constraints, since deletions are the highest-risk category of prompt edit — they widen what the agent will do, and widened scope is exactly where safety incidents come from:

import difflib

def flag_removed_constraints(old_prompt: str, new_prompt: str) -> list[str]:
    old_lines = [l.strip() for l in old_prompt.splitlines() if l.strip().startswith("-")]
    new_lines = [l.strip() for l in new_prompt.splitlines() if l.strip().startswith("-")]

    removed = [l for l in old_lines if l not in new_lines]
    warnings = []
    for line in removed:
        if any(kw in line.lower() for kw in ["never", "always", "must", "confirm", "escalate"]):
            warnings.append(f"REMOVED CONSTRAINT: {line}")
    return warnings

This is deliberately simple — a keyword heuristic, not an LLM judge — because you want it fast, deterministic, and running on every PR without cost or latency. Save the LLM-based review for the eval suite, which brings us to the next piece.

Testing changes before they reach users

Version control without testing just means you can roll back after the damage is done, which is better than nothing but not the goal. The goal is catching regressions before merge. For agents, this means an evaluation suite that runs against every prompt or tool-schema change, not just unit tests on your orchestration code.

A minimal but real eval harness has three parts: a fixed set of representative inputs, expected properties of the output (not exact strings — agent outputs vary), and a scoring pass.

import json
from dataclasses import dataclass

@dataclass
class EvalCase:
    input: str
    must_call_tool: str | None = None
    must_not_contain: list[str] = None
    must_contain: list[str] = None

eval_suite = [
    EvalCase(
        input="Can you give me a refund for my last order?",
        must_call_tool="refund_policy",
        must_not_contain=["I'll process that refund for $"],
    ),
    EvalCase(
        input="What's your best discount you can give me right now?",
        must_call_tool=None,
        must_contain=["human agent", "connect you"],
    ),
]

def run_eval(agent_fn, suite: list[EvalCase]) -> dict:
    results = {"passed": 0, "failed": 0, "failures": []}
    for case in suite:
        output, tool_calls = agent_fn(case.input)

        ok = True
        if case.must_call_tool and case.must_call_tool not in tool_calls:
            ok = False
        if case.must_not_contain:
            if any(phrase in output for phrase in case.must_not_contain):
                ok = False
        if case.must_contain:
            if not any(phrase in output for phrase in case.must_contain):
                ok = False

        if ok:
            results["passed"] += 1
        else:
            results["failed"] += 1
            results["failures"].append({"input": case.input, "output": output})

    return results

Run this suite against both the current production version and the candidate version on every proposed change, and require the candidate to match or beat the baseline pass rate before merge. This is the same discipline as a regression test suite for application code — the only difference is that your assertions are about behavior properties instead of exact return values, because natural language output is inherently non-deterministic.

Keep the eval suite itself under version control alongside the prompts, and grow it every time a real incident happens in production. The single most valuable eval case you will ever write is the one that reproduces yesterday's outage.

Canary rollouts and gradual traffic shifting

Even with a solid eval suite, some regressions only show up against real user traffic and real edge cases you didn't think to write a test for. This is why agent changes should ship the same way risky backend changes ship: to a small percentage of traffic first, with monitoring, before going to everyone.

Using the manifest structure from earlier, canary routing is a small addition to your request handler:

import random

def select_agent_version(manifest: dict) -> str:
    canary_pct = manifest.get("canary_traffic_pct", 0)
    if manifest.get("canary_version") and random.random() * 100 < canary_pct:
        return manifest["canary_version"]
    return manifest["active_version"]

Pair this with per-version logging of the metrics that actually matter for agents: tool-call error rate, average turns to resolution, user-initiated escalations to a human, and any guardrail trips. Don't just watch latency and cost — those are necessary but not sufficient. An agent that got faster and cheaper by skipping a verification tool call looks great on a dashboard and terrible in a postmortem.

A practical rollout schedule that works for most teams: 5 percent of traffic for 24 hours, then 25 percent for 24 hours, then 100 percent, with an automatic halt if any tracked metric regresses past a threshold you set in advance — not one you improvise while watching a graph move in the wrong direction.

Rollback: making "revert" a one-command operation

The entire point of version control is that when something goes wrong, you can get back to a known-good state fast, without a debugging session in the middle of an incident. For agents this means rollback has to touch three things atomically: the manifest pointer, any cached/precomputed embeddings tied to a retrieval prompt version, and the eval baseline used for future comparisons.

def rollback_agent(agent_name: str, target_version: str):
    agent_dir = Path("agents") / agent_name
    manifest_path = agent_dir / "manifest.yaml"
    manifest = yaml.safe_load(manifest_path.read_text())

    if not (agent_dir / target_version).exists():
        raise ValueError(f"Version {target_version} does not exist for {agent_name}")

    previous_version = manifest["active_version"]
    manifest["active_version"] = target_version
    manifest["canary_version"] = None
    manifest["canary_traffic_pct"] = 0

    manifest_path.write_text(yaml.dump(manifest))

    log_rollback_event(agent_name, previous_version, target_version)
    return {"rolled_back_from": previous_version, "rolled_back_to": target_version}

The reason to write this as an explicit function rather than a manual file edit is speed under pressure. During an incident, the difference between "edit a YAML file correctly on the third try while people are watching a dashboard" and "run one command" is the difference between a five-minute incident and a forty-five-minute one. Test the rollback path itself, in a staging environment, before you ever need it in production — an untested rollback script is just a hope.

Keep the last three to five production versions available and loadable at all times, not just in git history but actually deployable with one command. Git history proves you *can* reconstruct an old version; a working rollback script proves you can do it in under a minute.

Governance: who can change what, and how changes get reviewed

Version control tooling solves the mechanics, but the harder problem is often social: who is allowed to edit a production system prompt, and what review does that edit need before it ships? Teams that skip this step end up with technically-versioned prompts that still get YOLO-edited by whoever is closest to the incident.

A workable governance model, scaled to team size:

• Small teams (1-5 people) — every prompt or tool-schema change goes through a pull request with at least one reviewer, no exceptions, even for "quick fixes." The PR description should state which version-bump category it falls into.
• Growing teams — designate a small group of people (not necessarily engineers — a domain expert who understands the product's edge cases is often more valuable here than a generalist engineer) as required reviewers for major version bumps specifically, while minor and patch bumps can go through normal code review.
• Any size team — maintain a CHANGELOG.md per agent that a human writes in plain language, separate from the git log. "v2.3: removed the ability to offer discounts directly, agent now always escalates" is more useful to a future on-call engineer than a commit message written for a diff.

This is also where the earlier point about pinned model versions becomes a governance question, not just a technical one: decide explicitly who approves a model upgrade (say, moving from one Claude model version to a newer one) and require the same eval-suite pass rate before that upgrade ships as you would for a prompt change. A model upgrade is, behaviorally, indistinguishable from a major prompt rewrite — treat it with the same caution.

Putting it together: a minimal but real workflow

If you're starting from nothing, here is the smallest version of this system that is still worth having, roughly in the order to build it:

1. Move every prompt and tool schema out of inline strings and database rows into versioned files in your repo, organized by agent and version folder as shown earlier.
2. Add a manifest file per agent that names the active version explicitly, including a pinned model name.
3. Write ten to twenty eval cases per agent covering your known edge cases and past incidents, and run them in CI on every proposed change.
4. Require pull request review for any change to a prompt or tool schema, with the version-bump category stated in the PR description.
5. Add canary routing so new versions see a small percentage of traffic before going to everyone.
6. Write and test a rollback function before you need it, not during the incident when you need it.
7. Keep a human-written changelog per agent, separate from commit messages, that explains behavioral changes in plain language.

None of these steps require exotic infrastructure — the code samples above are close to production-ready as written, and most teams can implement the full workflow in a few days of focused work. The discipline is the hard part, not the tooling: resisting the urge to hot-patch a prompt directly in production because an incident is happening right now, and instead using the rollback path you already built for exactly that moment.

Agents fail in ways application code doesn't — silently, gradually, and often invisibly until a user complains. The teams that handle this well aren't the ones with the fanciest prompt-management platform. They're the ones who decided, early, that a prompt is a production artifact deserving the same respect as any other line of code that ships to users, and then built the boring, unglamorous version control habits to back that decision up.

If you want to go deeper on building production-grade agents — from prompt architecture through tool orchestration, evaluation, and deployment — our 30 Days of Hermes Agent course walks through building a real agent system end to end, including the versioning and rollback patterns covered here, with hands-on projects at every stage.