AI Agents

Building a Legal Research Agent: Accuracy Over Speed

Pramod Dutta · May 29, 2026 · 8 min read

The One Bug That Gets Lawyers Sanctioned

Somewhere out there, a lawyer filed a brief citing a case that never existed. An AI model made it up, complete with a plausible docket number and a confident summary of the holding. The lawyer didn't check. A judge did. This has now happened often enough across multiple jurisdictions that it's become the canonical cautionary tale in every conversation about AI and law.

If you're building a legal research agent, that story is not a footnote — it's the entire design brief. A legal research agent that answers in two seconds but invents a precedent is worse than useless. It's a liability generator wearing a helpful UI. The engineering challenge here is unusual compared to most agent projects: you are not optimizing for latency, fluency, or even breadth of coverage. You are optimizing for the ability to say "I don't know" convincingly, and for making every claim traceable back to a real, retrievable source.

This article walks through how to actually build one — architecture, retrieval strategy, citation verification, and the guardrails that matter — with the assumption that "confidently wrong" is the failure mode you're designing against, not "slow."

Why Legal Research Breaks Naive RAG

Most people's first instinct is: "This is just RAG. Chunk the case law, embed it, retrieve, generate." That gets you a demo. It does not get you a legal research tool, for a few structural reasons specific to this domain.

Citations are load-bearing, not decorative. In most RAG apps, a citation is a nice-to-have that builds user trust. In legal research, the citation *is* the deliverable. A lawyer doesn't want your summary of what the *Daubert* standard says — they want the exact case, the exact page, and ideally the exact paragraph, because they're going to cite it in a filing that another human will scrutinize.

Recency and jurisdiction change correctness. A case that's good law in the Ninth Circuit may have been explicitly overruled in the Second. A statute amended last session changes the answer even if the underlying question is unchanged. Naive semantic search has no concept of "this document is currently valid law in this jurisdiction" — it just finds text that's semantically similar, valid or not.

Negative treatment matters more than positive matches. The most dangerous failure isn't missing a supporting case — it's citing a case that's been overruled, vacated, or distinguished into irrelevance. A generic RAG pipeline that retrieves top-k similar chunks has no mechanism to flag "this precedent was reversed on appeal."

Silence is a valid, even preferred, answer. In most consumer AI products, refusing to answer is a UX failure. In legal research, "I found no controlling authority on this point in your jurisdiction" is frequently the *correct* and most valuable output. Your agent has to be built to prefer honest gaps over synthetic confidence.

Once you internalize these four points, the architecture stops looking like "chatbot with a vector database" and starts looking like a verification pipeline with a language model bolted onto the front and back.

Architecture: Retrieval, Verification, Synthesis as Separate Stages

The core design decision is to split the agent into three distinct stages that don't share trust. Each stage's output is treated as unverified input by the next stage.

Stage 1 — Structured Retrieval. Given a legal question, decompose it into sub-queries (issue, jurisdiction, date range, document type) and hit multiple sources: a case law database, a statute database, and possibly internal firm memos. This stage returns *candidates*, not answers.

Stage 2 — Citation Verification. Every candidate document is checked against an authoritative source before it's allowed to enter the context window used for generation. This is the stage most agent builders skip, and it's the one that actually prevents hallucination from reaching the user.

Stage 3 — Grounded Synthesis. Only verified documents are handed to the model for summarization, with a strict system prompt that forbids adding any claim not traceable to a passed-in source, and requires inline citation markers for every substantive sentence.

Here's a skeleton of that pipeline in Python, deliberately kept vendor-agnostic so you can swap in whatever legal database API or LLM provider you're using:

from dataclasses import dataclass, field
from typing import Optional

@dataclass
class LegalSource:
    citation: str
    jurisdiction: str
    doc_type: str  # "case", "statute", "regulation"
    text: str
    retrieved_url: str
    negative_treatment: Optional[str] = None  # e.g. "overruled", "distinguished"
    verified: bool = False

@dataclass
class ResearchQuery:
    question: str
    jurisdiction: str
    as_of_date: Optional[str] = None
    sub_queries: list[str] = field(default_factory=list)


class LegalResearchPipeline:
    def __init__(self, retriever, citation_checker, synthesizer):
        self.retriever = retriever
        self.citation_checker = citation_checker
        self.synthesizer = synthesizer

    def decompose(self, query: ResearchQuery) -> ResearchQuery:
        # Break a broad question into targeted sub-queries.
        # e.g. "Is a non-compete enforceable in CA?" ->
        #   ["California Business and Professions Code 16600",
        #    "non-compete enforceability exceptions California",
        #    "recent CA appellate rulings on non-compete 2024-2026"]
        query.sub_queries = self.retriever.plan_subqueries(query)
        return query

    def retrieve(self, query: ResearchQuery) -> list[LegalSource]:
        candidates: list[LegalSource] = []
        for sub_q in query.sub_queries:
            hits = self.retriever.search(
                sub_q,
                jurisdiction=query.jurisdiction,
                as_of_date=query.as_of_date,
            )
            candidates.extend(hits)
        return candidates

    def verify(self, candidates: list[LegalSource]) -> list[LegalSource]:
        verified = []
        for source in candidates:
            result = self.citation_checker.check(source.citation)
            if not result.exists:
                continue  # drop hallucinated or unresolvable citations
            source.negative_treatment = result.negative_treatment
            source.verified = True
            verified.append(source)
        return verified

    def run(self, question: str, jurisdiction: str, as_of_date: str = None) -> dict:
        query = ResearchQuery(question=question, jurisdiction=jurisdiction, as_of_date=as_of_date)
        query = self.decompose(query)
        candidates = self.retrieve(query)
        verified_sources = self.verify(candidates)

        if not verified_sources:
            return {
                "answer": "No verifiable authority found for this question in the specified jurisdiction.",
                "sources": [],
                "confidence": "none",
            }

        return self.synthesizer.answer(query, verified_sources)

Notice what's absent: there's no step where the model is asked to "just answer the question" from its own training data. Every path to an answer runs through retrieval and verification first. If verified_sources is empty, the pipeline returns an explicit "I don't know" rather than letting the synthesizer improvise.

The Citation Verification Layer: Where the Real Engineering Happens

This is the part most tutorials skip because it's unglamorous — it's not a clever prompt, it's a lookup against a source of truth. But it's the single highest-leverage piece of the whole system.

The pattern is simple: after retrieval, and again after synthesis, every citation the model produces gets checked against a real database, not against the model's own memory. Two checks matter:

Existence check — does this case or statute citation actually resolve to a real document? Formats like 410 U.S. 113 or 29 U.S.C. § 201 should map deterministically to a lookup.
Currency check — is this still good law? Has it been overruled, reversed, superseded by amendment, or limited by a later ruling?

import re
from dataclasses import dataclass

CITATION_PATTERN = re.compile(
    r"\b(\d+)\s+([A-Za-z.]+(?:\s[A-Za-z.]+)?)\s+(\d+)\b"
)

@dataclass
class VerificationResult:
    exists: bool
    negative_treatment: str | None
    canonical_url: str | None


class CitationChecker:
    def __init__(self, legal_db_client):
        self.db = legal_db_client

    def extract_citations(self, text: str) -> list[str]:
        return [m.group(0) for m in CITATION_PATTERN.finditer(text)]

    def check(self, citation: str) -> VerificationResult:
        record = self.db.lookup(citation)
        if record is None:
            return VerificationResult(exists=False, negative_treatment=None, canonical_url=None)

        treatment = self.db.get_treatment_history(record.id)
        flag = None
        if treatment.overruled:
            flag = f"overruled by {treatment.overruled_by}"
        elif treatment.reversed:
            flag = f"reversed on appeal ({treatment.reversal_citation})"
        elif treatment.superseded:
            flag = f"superseded by statute ({treatment.superseding_statute})"

        return VerificationResult(
            exists=True,
            negative_treatment=flag,
            canonical_url=record.url,
        )

    def audit_output(self, generated_text: str) -> list[dict]:
        """Post-generation pass: catch anything the model cited
        that wasn't in the verified source set it was given."""
        findings = []
        for citation in self.extract_citations(generated_text):
            result = self.check(citation)
            findings.append({
                "citation": citation,
                "verified": result.exists,
                "flag": result.negative_treatment,
            })
        return findings

The audit_output step is your last line of defense. Even with grounded synthesis, models occasionally paraphrase a citation slightly wrong, or blend two real cases into a citation that doesn't exist. Running a regex-and-lookup pass over the *final* generated text — not just the retrieved documents — catches this class of error before it reaches a human.

If a citation in the output fails this final audit, the correct behavior is to strip it and flag the sentence for human review, not to silently let it through because "it's probably fine."

Prompting for Refusal, Not Just Retrieval

A subtlety that trips up teams: even with a solid retrieval and verification pipeline, the synthesis prompt can still undo all of it if it's written like a generic assistant prompt. The system prompt for the synthesis stage needs explicit, almost adversarial instructions against filling gaps.

Every substantive sentence must carry a citation marker tied to a source explicitly passed into context.
If the provided sources don't cover part of the question, the model must say so rather than reasoning from general legal knowledge.
The model should be told to flag jurisdiction mismatches explicitly ("this authority is from the Fifth Circuit; you asked about New York").
Any source with negative treatment must be surfaced in the answer, not just quietly excluded.

A rough shape of that instruction:

You are a legal research assistant. You will be given a question and a
set of VERIFIED source documents. Rules, no exceptions:

1. Use ONLY the provided sources. Do not draw on outside knowledge of
   case law, even if you are confident it is correct.
2. Every claim of law must end with a citation marker like [1], [2]
   referencing the exact source it came from.
3. If the sources do not fully answer the question, say explicitly
   which part is unanswered. Do not fill the gap with inference.
4. If a source has negative treatment (overruled, reversed, superseded),
   you must mention this in the same sentence you cite it.
5. If no sources are relevant, respond only with:
   "No verifiable authority found in the provided sources."