Building a Coding Review Agent for Your CI Pipeline

The 2 AM Merge Nobody Reviewed Properly

You know this story. It's late, the PR has three approvals from teammates who skimmed the diff between meetings, CI is green because the tests pass, and the merge button gets clicked. Two days later a null pointer crashes checkout in production, and the postmortem reveals the bug was sitting right there in the diff the whole time — a missing null check on a field that used to always be populated, back when the schema was simpler.

Human code review is excellent at judgment calls: does this abstraction make sense, is this the right long-term direction, should we even be building this. It is mediocre at the mechanical stuff: does this new code path handle the null case, does this SQL query have an N+1 problem, does this function silently swallow an exception, did someone paste an API key into a config file. That mechanical layer is exactly what a code review AI agent is good at, and it's exactly the layer that burns out reviewers when they have to do it fifteen times a day.

This article walks through building an actual code review agent — not a wrapper around a single "review this diff" prompt, but a pipeline-integrated system with tool access, structured output, and a feedback loop that gets better over time. We'll cover the architecture, the prompt design, the CI wiring, the failure modes you will hit, and how to keep it from becoming the thing everyone silences with a // eslint-disable-style shrug.

What a Code Review Agent Actually Needs to Do

Before writing any code, it helps to separate what people mean by "AI code review" into three distinct jobs, because conflating them produces a mediocre agent that tries to do all three badly.

Job one: linting with judgment. Catching things a linter can't — a race condition in async code, a resource leak, an off-by-one in pagination logic, a misuse of a library API. This requires understanding intent, not just syntax.

Job two: pattern and convention enforcement. Does this PR follow the team's actual conventions (not the ones in the wiki that nobody updated) — naming, error handling shape, logging format, test coverage expectations for the module being touched.

Job three: risk flagging. Security-sensitive changes (auth, payment, PII handling), changes to hot-path code, changes that touch files with a history of regressions. This job isn't about correctness — it's about routing attention.

A good agent does all three, but they need different inputs. Job one needs the diff and enough surrounding code to understand types and call sites. Job two needs access to the existing codebase to compare against, not just the diff in isolation. Job three needs git history and possibly incident data. If you build an agent that only ever sees the diff text with no repo context, you'll get plausible-sounding comments that are frequently wrong about how a function is actually used elsewhere — which is the single fastest way to lose a team's trust in the tool.

Architecture: Agent, Not Script

A single prompt-and-response call to an LLM is not an agent — it's a classifier with better vocabulary. A review agent needs to behave like a careful engineer opening a PR: read the diff, then go look at other things before forming an opinion.

The core loop looks like this:

1. Receive PR event (opened / synchronize) from CI webhook
2. Fetch diff + changed file list + PR metadata
3. Agent loop:
   a. Read diff hunks
   b. Tool: read_file(path, line_range) for surrounding context
   c. Tool: search_codebase(query) to find other call sites / similar patterns
   d. Tool: read_test_file(path) to check if new code is covered
   e. Tool: git_blame(path, line) to see who/when touched risky lines
   f. Decide: comment, request-changes, approve, or escalate
4. Post structured comments back to the PR
5. Emit a machine-readable verdict for the CI gate

The tool-use loop matters more than the prompt wording. An agent that can call search_codebase before flagging "this function is never used elsewhere so it's dead code" will catch its own mistake when it finds three call sites it initially missed. An agent restricted to the diff text alone will confidently assert wrong things.

Here's a minimal skeleton for the agent loop using a tool-calling LLM API. This example uses a generic structure — swap in your provider's SDK:

import json

def run_review_agent(pr_diff: str, changed_files: list[str], tools: dict):
    system_prompt = """You are a senior code reviewer. You have tools to
    read files, search the codebase, and check git history. Use them
    before making claims about how code is used elsewhere. Only flag
    issues you are confident about. For each issue, output:
    - file, line_start, line_end
    - severity: blocker | warning | nit
    - category: bug | security | performance | style | test-gap
    - message: a specific, actionable explanation
    - suggested_fix: a code snippet if applicable
    Do not comment on formatting already enforced by linters."""

    messages = [
        {"role": "system", "content": system_prompt},
        {"role": "user", "content": f"Review this diff:\n\n{pr_diff}\n\n"
                                     f"Changed files: {changed_files}"}
    ]

    max_turns = 12
    findings = []

    for turn in range(max_turns):
        response = call_llm(messages, tools=list(tools.keys()))

        if response.tool_calls:
            for call in response.tool_calls:
                fn = tools[call.name]
                result = fn(**call.arguments)
                messages.append({
                    "role": "tool",
                    "tool_call_id": call.id,
                    "content": json.dumps(result)
                })
            continue

        # Final structured answer
        findings = json.loads(response.content)
        break

    return findings

The max_turns cap is not optional. Without it, an agent chasing down "let me also check how this is used in the admin module" can spiral into reading half the repo on every PR, burning tokens and latency for marginal benefit. Cap it, and if the agent hits the cap, have it emit whatever findings it has plus a note that review was partial — never fail silently.

Designing the Tools: Read, Search, Blame, Test

The tools you give the agent define the ceiling of what it can catch. Four tools cover most of the value:

• `read_file(path, start_line, end_line)` — pulls surrounding context so the agent isn't reasoning about a function signature it can't see
• `search_codebase(query, file_glob=None)` — grep-like semantic or literal search to find other call sites, similar patterns, or existing utilities the PR reinvents
• `get_test_coverage(path)` — checks whether the changed lines have corresponding test file changes, ideally by mapping source files to their test files via convention or a coverage report
• `git_blame(path, line)` — surfaces who last touched a line and in what commit, useful for flagging changes to code with a history of hotfixes

A simple implementation of the search tool, using ripgrep under the hood since it's fast enough to run inline in a request-response loop:

import subprocess
import shlex

def search_codebase(query: str, file_glob: str = None) -> list[dict]:
    cmd = ["rg", "--json", "--max-count", "20", query]
    if file_glob:
        cmd += ["--glob", file_glob]

    result = subprocess.run(cmd, capture_output=True, text=True, timeout=10)

    matches = []
    for line in result.stdout.splitlines():
        event = json.loads(line)
        if event.get("type") == "match":
            data = event["data"]
            matches.append({
                "path": data["path"]["text"],
                "line_number": data["line_number"],
                "text": data["lines"]["text"].strip()
            })
    return matches

Keep every tool read-only. A review agent should never have write access to the repository, never run arbitrary shell commands beyond a fixed allowlist like rg and git blame, and never fetch external URLs. The blast radius of a misbehaving reviewer should be "posted a wrong comment," never "modified a file" or "leaked repo contents to an external endpoint." This constraint also makes the agent auditable — you can log every tool call and know exactly what it looked at before forming each opinion.

The Prompt: Specificity Beats Cleverness

The instinct is to write a long, impressive system prompt covering every possible code smell. Resist it. A review agent prompt that tries to catch everything ends up catching nothing well, because the model spreads attention across too many dimensions and produces vague, hedge-everything comments.

Better approach: scope the agent to a short list of high-value categories, and make each one concrete with examples of what counts and what doesn't.

Review categories, in priority order:

1. BUG: logic errors, unhandled edge cases (null/empty/zero),
   off-by-one errors, incorrect conditionals, race conditions
   in concurrent code. Only flag if you can point to a specific
   input that breaks it.

2. SECURITY: injection risk, missing auth checks on new endpoints,
   secrets in code, unsafe deserialization, missing input validation
   on user-facing fields.

3. RESOURCE: unclosed file handles/connections, missing timeouts
   on network calls, unbounded loops or recursion, N+1 query
   patterns in loops over DB results.

4. TEST-GAP: new branches or error paths with no corresponding
   test. Do not flag missing tests for trivial getters/setters
   or generated code.

Explicitly excluded from your review: formatting, import ordering,
naming style already covered by the linter config in .eslintrc,
subjective architecture opinions ("I would have structured this
differently"). If you're not at least 80% confident an issue is
real, do not report it — false positives cost more than missed
issues in this pipeline.

That last line — the confidence threshold — is the single highest-leverage sentence in the whole prompt. Teams abandon review agents almost universally because of noise, not because of missed bugs. A tool that's right 60% of the time and wrong 40% of the time gets its comments ignored wholesale within two weeks, at which point it's also missing the 60% that would have helped. Tune hard toward precision over recall, especially in the first few months of rollout.

Structured Output and the CI Gate

The agent's output needs to be machine-parseable so your CI pipeline can act on it, not just human-readable prose dumped into a comment. Enforce a JSON schema on the final response:

{
  "summary": "Reviewed 4 files, 2 blockers, 1 warning",
  "verdict": "request_changes",
  "findings": [
    {
      "file": "src/checkout/payment.py",
      "line_start": 142,
      "line_end": 142,
      "severity": "blocker",
      "category": "bug",
      "message": "amount is read from request.json without checking it's present, will raise KeyError on malformed requests instead of returning a 400",
      "suggested_fix": "amount = request.json.get('amount')\nif amount is None:\n    return error_response(400, 'amount is required')"
    }
  ]
}

Wire the verdict field into your CI gate logic. A common pattern:

# .github/workflows/ai-review.yml
name: AI Code Review
on:
  pull_request:
    types: [opened, synchronize]

jobs:
  review:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 0

      - name: Run review agent
        id: review
        run: python scripts/run_review_agent.py --pr ${{ github.event.number }}
        env:
          LLM_API_KEY: ${{ secrets.LLM_API_KEY }}

      - name: Post review comments
        run: python scripts/post_pr_comments.py --results review_output.json

      - name: Enforce gate
        run: |
          VERDICT=$(jq -r '.verdict' review_output.json)
          if [ "$VERDICT" = "request_changes" ]; then
            BLOCKERS=$(jq '[.findings[] | select(.severity=="blocker")] | length' review_output.json)
            if [ "$BLOCKERS" -gt 0 ]; then
              echo "Blocking merge: $BLOCKERS blocker(s) found"
              exit 1
            fi
          fi

Note the design choice: only blocker severity fails the build. Warnings and nits get posted as comments but don't block merge. This is deliberate — a gate that blocks on every finding trains engineers to add noisy suppression comments or route around the tool entirely. Reserve hard blocks for things you'd genuinely revert a merge over.

Handling the Diff-Only Blindness Problem

The most common failure mode in naive implementations is reviewing the diff as if it exists in a vacuum. A diff shows you lines added and removed, but not:

• The full function the changed lines live inside
• Other callers of a modified function signature
• Whether a "new" utility function already exists elsewhere under a different name
• Whether the change breaks an implicit contract documented only in a test file three directories away

Fix this by always expanding diff hunks to full-function context before the agent reasons about them, and by making the search tool cheap enough that the agent actually uses it rather than guessing.

def expand_hunks_to_functions(diff_hunks: list[dict], repo_path: str) -> list[dict]:
    """Given diff hunks, expand each to the enclosing function/class
    using a language-aware parser instead of raw line ranges."""
    expanded = []
    for hunk in diff_hunks:
        tree = parse_file_ast(repo_path, hunk["file"])
        enclosing = find_enclosing_scope(tree, hunk["start_line"], hunk["end_line"])
        expanded.append({
            "file": hunk["file"],
            "changed_lines": (hunk["start_line"], hunk["end_line"]),
            "full_context": enclosing.source_text,
            "context_start_line": enclosing.start_line
        })
    return expanded

Using a real AST parser (tree-sitter works well across languages) rather than naive line-counting for this step pays for itself immediately — it's the difference between the agent seeing "line 142 changed" and seeing the whole function that line lives in, including its docstring, its error handling, and its return type.

Rollout: Start as Advisory, Earn the Gate

Do not ship this agent as a hard merge gate on day one. Every team that has done this well ran it in shadow mode first:

1. Weeks 1-2, silent mode. Agent runs on every PR, logs findings to a dashboard, posts nothing to the PR. You review its output against what human reviewers actually caught and missed.
2. Weeks 3-4, comment-only mode. Agent posts comments on PRs but has zero effect on merge-ability. Track how often engineers react positively (thumbs up, "good catch," fixing the flagged line) versus dismissing it.
3. Month 2+, soft gate. Blockers require an explicit override comment (like /override-ai-review reason: <text>) rather than silently blocking, and overrides get logged for a monthly review of false-positive rate.
4. Only after sustained low false-positive rate, hard gate on blocker-severity findings for specific high-risk paths (auth, payments, data deletion) — not the whole repo.

Track two numbers religiously: precision (of flagged blockers, what fraction were real issues a human agreed with) and catch rate on a held-out set of past incidents (feed the agent diffs from PRs that later caused production bugs, see if it would have caught them). If precision drops below roughly 70%, pause and retune the prompt and confidence threshold before expanding scope — trust, once lost with a noisy bot, is expensive to rebuild.

Cost, Latency, and Model Choice

Running an agentic loop with multiple tool calls per PR adds up faster than a single completion call. A few practical levers:

• Only review changed hunks plus expanded context, not entire files, especially for large files where 95% of the content is untouched
• Cache repo-search results within a single PR review — if the agent searches for the same symbol twice across turns, don't re-run ripgrep
• Use a cheaper/faster model for the triage pass (deciding which files even need deep review) and reserve your strongest model for the actual finding generation on flagged files
• Batch PRs from the same author/timeframe if your CI queue allows it, to amortize any shared setup cost like cloning and indexing

For latency, most teams find 2-5 minutes of review time acceptable if it runs in parallel with the existing test suite rather than blocking it sequentially. Wire the review job as a parallel CI job, not a step after tests pass — there's no dependency between "do the tests pass" and "did the agent review the diff," so don't force them to run in series.

Common Pitfalls to Avoid

A short list of mistakes that show up repeatedly in early implementations:

• Re-litigating style the linter already owns. If Prettier or Black already enforces formatting, don't let the agent comment on it — it's a fast way to look redundant and untrustworthy on day one.
• No handling for very large PRs. A 3,000-line diff will blow context budgets or produce shallow, generic findings. Set a size threshold above which the agent posts "PR too large for full automated review, consider splitting" instead of a weak attempt.
• Ignoring PR description and linked ticket. A change that looks risky in isolation (deleting a validation check) might be exactly the intended fix described in the linked bug ticket. Feed the PR title, description, and linked issue into the prompt context.
• Treating every repo the same. A test-gap warning makes sense for application code, not for a one-off data migration script. Let per-directory or per-file config suppress categories that don't apply.
• No feedback loop. If engineers can't react to a finding (thumbs down, "not applicable"), you have no signal to improve the prompt. Wire reactions back into a small dataset you periodically review.

Where This Fits in the Bigger Agent Picture

A CI-embedded review agent is a good first production agent to build precisely because the stakes of a wrong answer are bounded — worst case, it posts a bad comment that a human ignores, rather than, say, executing a bad database migration. It also forces you to practice the core skills that carry over to every other agent you'll build: giving a model scoped, read-only tools; enforcing structured output; running an agent loop with a turn cap; and building the trust and rollout discipline that determines whether anyone actually uses what you shipped.

If you want to go deeper into building agents like this one — tool design, agent loops, evaluation harnesses, and the operational discipline of shipping agents that people actually trust in production — that's exactly the ground covered in the 30 Days of Hermes Agent course on teachyou.ai. It walks through building agentic systems from first principles up through the same kind of CI-integrated, tool-using architecture described here, with hands-on projects rather than just theory.

Start small, keep it read-only, measure precision obsessively, and let the agent earn its place in the merge gate one accurate finding at a time.