Building an Internal Knowledge Agent for Engineering Teams

The 2 AM Slack Message Every Team Knows

"Hey, does anyone remember why we set that timeout to 30 seconds in the payments service?" It's 2 AM somewhere, a new engineer is debugging a production incident, and the answer to their question is buried in a Notion doc from eighteen months ago, a Slack thread that got archived, or the memory of someone who left the company last quarter. This is not a tooling problem you fix with "better documentation." Documentation always rots. What doesn't rot, if you build it right, is an agent that can search across all your fragmented knowledge sources in real time and reason about what it finds.

An internal knowledge agent is not a chatbot bolted onto a wiki. It is a retrieval system wired to an LLM with tool access, permission awareness, and enough engineering discipline around evaluation that engineers trust its answers instead of double-checking every single one. This article walks through what it actually takes to build one for an engineering team: the architecture, the retrieval strategy, the tool design, the permission model, and the failure modes that will bite you if you skip them.

Why Generic Search and Generic Chatbots Both Fail

Before writing any code, it's worth being precise about the problem. Engineering knowledge lives in at least six places: your codebase (comments, commit messages, README files), your issue tracker (Jira, Linear), your docs tool (Notion, Confluence), your chat history (Slack, Discord), your incident postmortems, and the tribal knowledge in people's heads that never got written down at all.

Generic full-text search (think Confluence's built-in search or Slack search) fails because it matches keywords, not intent. If an engineer asks "why does the retry logic in the checkout service back off exponentially instead of linearly," keyword search will surface documents containing the words "retry" and "checkout," ranked by recency or click history, none of which necessarily explain the *why*.

A generic chatbot wired to a single LLM without retrieval fails differently: it hallucinates confidently. Ask GPT-5 or Claude cold "what's our deployment rollback procedure" and it will produce a plausible-sounding, entirely fabricated answer, because it has no access to your actual runbooks. The model is not lying on purpose — it's doing what language models do when there's no grounding: pattern-matching to what a rollback procedure *usually* looks like.

The fix for both problems is the same architecture: retrieval-augmented generation (RAG) plus tool use, where the agent doesn't answer from parametric memory — it answers from documents it just retrieved, and it can call tools to go get more information when the first retrieval pass isn't enough. That's the core design principle behind everything in this article.

Core Architecture: Retrieval, Reasoning, Tools

At a high level, an internal knowledge agent has four layers:

• Ingestion layer — connectors that pull content from your sources (Git, Confluence, Slack, Jira, PagerDuty postmortems) and normalize it into a common document format
• Indexing layer — chunking, embedding, and storing documents in a vector store (and usually a keyword index alongside it for hybrid search)
• Retrieval and reasoning layer — the agent loop that takes a question, decides what to search for, executes searches, evaluates whether it has enough context, and either answers or searches again
• Tool layer — structured actions beyond plain search: querying a database, looking up a Jira ticket by ID, checking who owns a service, or pinging an on-call schedule

Here's a simplified skeleton of the reasoning loop using Claude's tool-use API. This is the part that actually decides "do I have enough information to answer, or do I need to search more":

import anthropic

client = anthropic.Anthropic()

tools = [
    {
        "name": "search_knowledge_base",
        "description": "Search internal docs, code comments, and postmortems for relevant passages.",
        "input_schema": {
            "type": "object",
            "properties": {
                "query": {"type": "string", "description": "Search query"},
                "source_filter": {
                    "type": "string",
                    "enum": ["all", "code", "docs", "incidents", "tickets"],
                    "description": "Restrict search to one source type"
                }
            },
            "required": ["query"]
        }
    },
    {
        "name": "get_ticket",
        "description": "Fetch a Jira/Linear ticket by ID including comments.",
        "input_schema": {
            "type": "object",
            "properties": {"ticket_id": {"type": "string"}},
            "required": ["ticket_id"]
        }
    }
]

def run_agent(question: str, history: list):
    messages = history + [{"role": "user", "content": question}]

    while True:
        response = client.messages.create(
            model="claude-sonnet-4-5",
            max_tokens=1500,
            system=KNOWLEDGE_AGENT_SYSTEM_PROMPT,
            tools=tools,
            messages=messages,
        )

        if response.stop_reason != "tool_use":
            return response.content

        messages.append({"role": "assistant", "content": response.content})
        tool_results = []

        for block in response.content:
            if block.type == "tool_use":
                result = dispatch_tool(block.name, block.input)
                tool_results.append({
                    "type": "tool_result",
                    "tool_use_id": block.id,
                    "content": result,
                })

        messages.append({"role": "user", "content": tool_results})

Notice what this loop is *not* doing: it's not a single retrieve-then-generate call. The model decides whether to call search_knowledge_base, inspects what comes back, and can choose to call it again with a narrower query, or call get_ticket to pull the full context of a referenced issue. This agentic retrieval loop consistently beats naive single-shot RAG on multi-hop questions — the kind where the answer requires connecting a Slack thread to a postmortem to a code comment.

Chunking and Indexing: The Unglamorous Part That Determines Quality

Most knowledge agent failures trace back to bad chunking, not bad models. If you chunk a 40-page architecture doc into fixed 500-token blocks without respecting section boundaries, you'll retrieve a chunk that says "as shown in the diagram above, we chose option B because" with no diagram and no idea what option B was.

A few rules that matter in practice:

• Chunk by semantic boundary, not fixed token count. Split on headings, function boundaries, or paragraph breaks first; only fall back to fixed-size splitting when a section is too large on its own.
• Keep code chunks whole per function or class where possible. A half-a-function chunk is nearly useless for an engineering agent.
• Attach metadata to every chunk: source system, author, last-modified date, and — critically — a permission scope. You will need this later.
• Overlap chunks slightly (10-15%) so that context spanning a boundary isn't lost entirely.
• Re-embed on change, not on a schedule. If your ingestion only refreshes nightly, an engineer who just merged a fix explaining a workaround won't show up in results until the next day, right when someone else hits the same issue.

Here's a minimal chunking function that respects markdown structure instead of blindly slicing text:

import re

def chunk_markdown(text: str, source_id: str, max_tokens: int = 400):
    sections = re.split(r"\n(?=#{1,3}\s)", text)
    chunks = []

    for section in sections:
        if estimate_tokens(section) <= max_tokens:
            chunks.append(make_chunk(section, source_id))
            continue

        paragraphs = section.split("\n\n")
        buffer = ""
        for para in paragraphs:
            if estimate_tokens(buffer + para) > max_tokens and buffer:
                chunks.append(make_chunk(buffer, source_id))
                buffer = para
            else:
                buffer += "\n\n" + para
        if buffer.strip():
            chunks.append(make_chunk(buffer, source_id))

    return chunks

def estimate_tokens(text: str) -> int:
    return len(text) // 4

def make_chunk(text: str, source_id: str) -> dict:
    return {
        "text": text.strip(),
        "source_id": source_id,
        "token_estimate": estimate_tokens(text),
    }

For code, use language-aware splitting (tree-sitter is the standard choice) rather than treating source files as plain text — you want the chunk boundary to fall on a function or class definition, never mid-body.

One more detail that's easy to underweight: chunk provenance display. When the agent answers, show the engineer exactly which chunk backed which sentence, with a link back to the original document or commit. This does two things at once. It lets a skeptical senior engineer verify the answer in ten seconds instead of trusting it blindly, and it gives you a feedback signal for free — if engineers keep clicking through to the same three stale documents and correcting the agent in the same way, that's a strong hint those documents need to be updated or retired, not that the model needs more prompting.

On embeddings: use a hybrid retrieval strategy — dense vector search for semantic matches plus BM25 or another sparse keyword method for exact-term matches (error codes, function names, ticket IDs rarely embed well semantically). Re-rank the combined candidate set with a cross-encoder before handing the top 5-8 chunks to the LLM. Skipping the re-rank step and just dumping the top-20 vector hits into the context window is a common shortcut that quietly degrades answer quality, because irrelevant chunks dilute the model's attention even when the right chunk is technically present.

Permissions Are Not an Afterthought

This is the section most tutorials skip and the one that will actually get your project killed by security review. An internal knowledge agent that ignores document-level permissions is a data leak waiting to happen. If your compensation bands live in a restricted Notion space, and your vector index doesn't track that restriction, any engineer who asks the agent the right question can exfiltrate it through a summarization request.

The fix is to carry permission metadata through the entire pipeline, not bolt it on at the UI layer:

def search_knowledge_base(query: str, user_id: str, source_filter: str = "all") -> str:
    user_groups = get_user_permission_groups(user_id)

    candidates = vector_store.query(
        query_text=query,
        top_k=25,
        filter={"source_type": source_filter} if source_filter != "all" else None,
    )

    allowed = [
        c for c in candidates
        if set(c["metadata"]["allowed_groups"]) & set(user_groups)
    ]

    reranked = rerank(query, allowed)[:8]

    if not reranked:
        return "No accessible documents matched this query."

    return format_chunks_for_context(reranked)
}

The critical property here: filtering happens after retrieval against the actual requesting user's groups, using metadata that was captured at ingestion time and kept in sync with the source system's ACLs. If someone's access to a Confluence space is revoked, your ingestion job needs to catch that and update the index — a stale permission cache is just as dangerous as no permission check at all. Run a periodic reconciliation job that re-checks source ACLs against your index, not just an on-write hook, because on-write hooks miss out-of-band permission changes made directly in the source system.

Designing Tools Beyond Plain Search

Search retrieval solves "find relevant text." It doesn't solve "tell me who is on-call right now" or "what's the current status of ticket ENG-4521" — those need live data, not indexed snapshots. A knowledge agent that's actually useful to engineers combines retrieval with a small set of well-scoped tools:

• `get_ticket(ticket_id)` — live fetch from Jira/Linear, not from the stale indexed copy, since ticket status changes constantly
• `get_service_owner(service_name)` — queries your service catalog (Backstage, or a simple internal mapping) so "who owns the auth service" doesn't require a document at all
• `search_code(query, repo)` — a dedicated code-search tool (grep-style or symbol-aware) separate from the general knowledge search, because code search benefits from exact-match and regex semantics that vector search handles poorly
• `get_recent_deploys(service_name)` — pulls from your CI/CD system, useful for "did something ship to this service in the last 24 hours"

The design principle: give the agent narrow, single-purpose tools with tight schemas rather than one giant "run arbitrary query" tool. A narrow tool is easier to permission-check, easier to log, and far easier for the model to use correctly, because the input schema itself constrains what the model can even attempt.

def dispatch_tool(name: str, tool_input: dict, user_id: str) -> str:
    handlers = {
        "search_knowledge_base": lambda i: search_knowledge_base(
            i["query"], user_id, i.get("source_filter", "all")
        ),
        "get_ticket": lambda i: get_ticket_if_authorized(i["ticket_id"], user_id),
        "get_service_owner": lambda i: get_service_owner(i["service_name"]),
    }

    handler = handlers.get(name)
    if handler is None:
        return f"Unknown tool: {name}"

    try:
        return handler(tool_input)
    except PermissionError:
        return "You don't have access to this resource."
    except Exception as e:
        return f"Tool error: {str(e)}"

Note the explicit PermissionError handling returned as a message the model can relay honestly to the user, rather than silently failing or leaking a stack trace. The model should be able to say "I don't have permission to check that" as a legitimate, expected answer — not a bug.

Prompting the Agent to Say "I Don't Know"

The single highest-leverage system prompt change for an internal knowledge agent is instructing it, explicitly and repeatedly, to refuse to answer when retrieval comes back empty or ambiguous. Left to its own devices, an LLM under pressure to be helpful will paper over gaps with plausible-sounding filler. For an engineering team, a confidently wrong answer about a deploy procedure is worse than no answer at all.

You are an internal engineering knowledge agent. You answer questions using
ONLY information retrieved via your tools. You have no other source of truth.

Rules:
1. Always search before answering unless the question is purely conversational.
2. If retrieved documents don't contain enough information to answer confidently,
   say so explicitly. Do not fill gaps with general knowledge about how such
   systems "usually" work.
3. Always cite the source of each claim (document title or ticket ID) inline.
4. If sources conflict (e.g., an old doc contradicts a recent one), surface
   the conflict rather than silently picking one.
5. If a query touches a restricted resource the user cannot access, say so
   plainly instead of declining vaguely.

That instruction to surface conflicts (rule 4) matters more than it sounds. Engineering docs frequently go stale — a design doc says "we use synchronous replication," but the postmortem from three months later says the team switched to async after an incident. An agent that just picks whichever chunk scored higher in retrieval and presents it as fact is actively harmful. Surfacing the conflict and letting the engineer decide which is current is the honest answer.

Evaluation: How You Know It's Actually Working

You cannot ship a knowledge agent to an engineering org without a real evaluation set, because "it felt right in the demo" does not survive contact with fifty engineers asking questions you never anticipated. Build an eval set from actual historical questions — mine your support Slack channel or your onboarding buddy's inbox for real questions engineers have asked, and pair each with a known-correct answer and its source documents.

eval_cases = [
    {
        "question": "Why did we move payment retries to exponential backoff?",
        "expected_sources": ["postmortem-2025-11-payment-timeout.md"],
        "must_mention": ["thundering herd", "downstream rate limit"],
    },
    {
        "question": "Who owns the notification service?",
        "expected_sources": ["service-catalog"],
        "must_mention": ["platform-team"],
    },
]

def run_eval(agent_fn, cases):
    results = []
    for case in cases:
        answer, sources_used = agent_fn(case["question"])
        source_hit = any(s in sources_used for s in case["expected_sources"])
        content_hit = all(
            phrase.lower() in answer.lower() for phrase in case["must_mention"]
        )
        results.append({
            "question": case["question"],
            "source_recall": source_hit,
            "content_match": content_hit,
            "passed": source_hit and content_hit,
        })
    return results
}

Track two numbers over time as you change chunking strategy, retrieval parameters, or the system prompt: retrieval recall (did the right document even get retrieved) and answer correctness (given the right document, did the model use it correctly). Splitting these two metrics apart is what tells you whether a regression is a retrieval problem or a reasoning problem — conflating them into one "did it get the answer right" score makes debugging painfully slow, because you'll spend a week tuning prompts to fix what was actually a chunking bug.

Also run a quarterly "stale answer" audit: pick twenty answers the agent gave three months ago and re-ask the same questions today. Documentation drifts, services get renamed, and an agent's index needs the same continuous-integration discipline you'd apply to code — otherwise confidence in the tool erodes the first time it confidently repeats something that stopped being true two reorgs ago.

One more evaluation habit worth building early: log every tool call the agent makes alongside the final answer, not just the final answer itself. When a support engineer flags a wrong answer, you want to know immediately whether the agent searched for the wrong thing, retrieved the right chunk but misread it, or never called a tool at all and answered from general knowledge despite the system prompt telling it not to. Without call-level logging, every bug report turns into an hour of guesswork; with it, most turn into a five-minute diagnosis. Store these traces the same way you'd store request logs for any production service — with enough retention to look back weeks, and enough structure (question, tool calls, tool outputs, final answer, model, prompt version) to diff two runs against each other after you change something.

Rollout: Start Narrow, Expand Deliberately

The teams that get burned by internal knowledge agents are the ones that connect every source system on day one and open it to the whole company. Instead:

1. Pick one team and one source (say, the platform team's on-call runbooks and postmortems) and get retrieval quality genuinely good there first
2. Build the eval set from that team's real questions before expanding sources
3. Add a feedback mechanism — a simple thumbs up/down on each answer — and actually review the thumbs-down cases weekly, not just collect them
4. Expand to a second source only once the first is stable, so you can attribute quality changes to the right cause
5. Roll out company-wide only after permission filtering has been explicitly tested with a user who should NOT have access to a specific restricted doc, confirming the agent correctly withholds it

This staged rollout is slower than a one-week hackathon demo, but a knowledge agent that gives wrong answers confidently in its first week will get uninstalled from muscle memory even after you fix the bug, because trust is far more expensive to rebuild than it is to establish.

Where This Fits Into the Bigger Agent-Building Picture

Building an internal knowledge agent well touches nearly every hard problem in applied AI engineering: retrieval architecture, tool design, permission modeling, prompt engineering for calibrated uncertainty, and evaluation discipline that actually catches regressions before your engineers do. None of these are solved by picking a slightly better model — they're solved by the systems and habits you build around the model.

If you want to go deeper on exactly this kind of agent engineering — designing tool schemas that models use correctly, building retrieval pipelines that don't silently degrade, and setting up evals that catch regressions before your users do — that's the core of what we teach in 30 Days of Hermes Agent on teachyou.ai. It's a hands-on, project-based course where you build a real agent from scratch across 30 days, covering exactly the architecture patterns in this article: RAG pipelines, permission-aware tools, multi-step reasoning loops, and evaluation harnesses you can actually trust. If your team is staring down the "we should really build an internal knowledge agent" backlog item, it's a solid place to start.