RAG

RAG for Internal Wikis: A Confluence/Notion Knowledge Base Bot

Pramod Dutta · May 12, 2026 · 14 min read

Every company with more than twenty engineers has the same problem: the answer to "how do I get prod access" exists somewhere, but nobody remembers if it's in Confluence, a pinned Slack message, or a Notion page that got moved into an archive folder eighteen months ago. Search bars in these tools are keyword matchers from 2012. They don't understand that "deploy pipeline" and "release process" mean the same thing. So people just ask in Slack instead, and the same three senior engineers answer the same ten questions every single week. This is exactly the kind of problem retrieval-augmented generation was built for, and it's one of the highest-leverage internal tools you can ship. Unlike a customer-facing chatbot, an internal wiki bot has a forgiving audience, a bounded document set, and a very clear success metric: did it stop someone from pinging a human. Let's build one properly, using Confluence or Notion as the source of truth.

Why Internal Wikis Are a Different RAG Problem

Most RAG tutorials use static PDFs or a single Notion export as the corpus. Internal wikis are messier in ways that actually matter for design decisions.

First, the content changes constantly. A pricing page in Confluence gets edited three times a week during a launch. If your embeddings are stale, the bot confidently tells someone the old pricing. This means sync strategy is not an afterthought — it's the first thing you design, before you even pick an embedding model.

Second, wikis have structure that carries meaning: parent-child page hierarchies, spaces, labels, permissions. A flat chunk-and-embed pipeline throws all of that away. If you don't preserve at least the page title, space name, and breadcrumb path, your retrieved chunks become orphaned paragraphs with no context — the model has no idea if a snippet about "rotating API keys" is from the security runbook or from an outdated 2022 onboarding doc.

Third, permissions are real. Confluence and Notion both have page-level and space-level access controls. An internal RAG bot that doesn't respect those is a security incident waiting to happen. I've seen teams skip this because "it's just internal," then realize the HR space with salary bands got indexed into a bot every engineer can query.

Fourth, staleness and duplication are rampant. Wikis accumulate five versions of "onboarding guide," half of them abandoned drafts. Naive retrieval will happily surface the wrong one with full confidence, which is worse than surfacing nothing.

Keep these four in your head — sync, structure, permissions, staleness — because every design choice below traces back to one of them.

Architecture Overview

The system has five stages, and it's worth sketching them before writing code:

Ingestion — pull pages from Confluence/Notion via their APIs on a schedule or webhook
Normalization — convert each platform's native format (Confluence storage format XML, Notion's block JSON) into clean markdown
Chunking — split into retrieval-sized units while preserving hierarchy metadata
Embedding and indexing — generate vectors and store them alongside metadata (space, permissions, last-updated) in a vector database
Retrieval and generation — at query time, embed the question, retrieve candidates, filter by the asker's permissions, then generate an answer with citations

The trap most people fall into is treating this as a single "load documents, embed, done" script. It works fine as a demo and falls apart within a month because there's no re-sync logic and no permission filter. Build the sync loop first; the embedding logic is the easy 20%.

Pulling Content from Confluence and Notion

Both platforms expose REST APIs that let you list spaces/databases and fetch page content with pagination. The shapes differ, so I normalize both into a common WikiDocument structure before anything else touches the data.

from dataclasses import dataclass, field
from datetime import datetime

@dataclass
class WikiDocument:
    source: str          # "confluence" or "notion"
    doc_id: str
    title: str
    space: str
    url: str
    breadcrumb: list[str]
    content_markdown: str
    updated_at: datetime
    allowed_groups: list[str] = field(default_factory=list)


def fetch_confluence_pages(base_url, space_key, session, cursor=None):
    params = {
        "spaceKey": space_key,
        "expand": "body.storage,ancestors,version",
        "limit": 50,
    }
    if cursor:
        params["start"] = cursor

    resp = session.get(f"{base_url}/rest/api/content", params=params)
    resp.raise_for_status()
    data = resp.json()

    docs = []
    for page in data["results"]:
        breadcrumb = [a["title"] for a in page.get("ancestors", [])]
        docs.append(WikiDocument(
            source="confluence",
            doc_id=page["id"],
            title=page["title"],
            space=space_key,
            url=f"{base_url}/wiki{page['_links']['webui']}",
            breadcrumb=breadcrumb,
            content_markdown=storage_format_to_markdown(page["body"]["storage"]["value"]),
            updated_at=datetime.fromisoformat(page["version"]["when"]),
        ))
    return docs, data.get("_links", {}).get("next")

Notion works differently — it's block-based rather than a single HTML blob, so you recursively walk children:

def fetch_notion_page_markdown(page_id, client):
    blocks = []
    cursor = None
    while True:
        resp = client.blocks.children.list(block_id=page_id, start_cursor=cursor)
        blocks.extend(resp["results"])
        if not resp["has_more"]:
            break
        cursor = resp["next_cursor"]

    lines = []
    for block in blocks:
        lines.append(notion_block_to_markdown(block))
        if block.get("has_children"):
            child_md = fetch_notion_page_markdown(block["id"], client)
            lines.append(child_md)
    return "\n\n".join(lines)

Whatever the source, the output is markdown with headings preserved. That matters a lot for the next step.

Chunking Strategy: Respect the Document's Own Structure

The single biggest quality lever in a wiki RAG system is chunking, and the biggest mistake is fixed-size chunking that ignores headings. A runbook page usually has sections like "Symptoms," "Diagnosis," "Fix." If you slice at 500 characters regardless of structure, you'll cut a fix instruction in half between two chunks and retrieval will surface neither one usefully.

Instead, chunk along markdown headings first, then split oversized sections by size as a fallback:

import re

def chunk_wiki_document(doc: WikiDocument, max_tokens=400):
    sections = re.split(r"(?=^#{1,3} .+$)", doc.content_markdown, flags=re.MULTILINE)
    chunks = []

    for section in sections:
        section = section.strip()
        if not section:
            continue

        heading_match = re.match(r"^(#{1,3}) (.+)$", section)
        heading = heading_match.group(2) if heading_match else doc.title

        if estimate_tokens(section) <= max_tokens:
            chunks.append(build_chunk(doc, section, heading))
        else:
            for sub in split_by_tokens(section, max_tokens, overlap=50):
                chunks.append(build_chunk(doc, sub, heading))

    return chunks


def build_chunk(doc: WikiDocument, text: str, heading: str):
    context_prefix = f"Page: {doc.title}\nSection: {heading}\nSpace: {doc.space}\n\n"
    return {
        "text": context_prefix + text,
        "metadata": {
            "doc_id": doc.doc_id,
            "title": doc.title,
            "heading": heading,
            "space": doc.space,
            "url": doc.url,
            "breadcrumb": doc.breadcrumb,
            "updated_at": doc.updated_at.isoformat(),
            "allowed_groups": doc.allowed_groups,
        },
    }

Notice the context_prefix. Embedding a bare paragraph like "Restart the service with systemctl restart worker" loses all context about which service, which environment, which runbook. Prepending the page title and section heading into the embedded text dramatically improves retrieval precision, because the embedding model now encodes "this is about the payments worker restart procedure" rather than an ambiguous shell command.

Overlap between sub-chunks (I use 50 tokens) prevents you from losing a sentence that straddles a chunk boundary — a classic cause of "the bot almost answered but missed the last step."

Embedding and Indexing with Metadata Filters

Once you have chunks, embedding is the boring part — call an embedding model, store vectors plus metadata. The part people skip is designing the metadata schema so permission filtering and staleness checks are cheap at query time, not an afterthought bolted onto retrieval.

import hashlib

def index_chunks(chunks, embed_fn, vector_store):
    texts = [c["text"] for c in chunks]
    vectors = embed_fn(texts)  # batched embedding call

    records = []
    for chunk, vector in zip(chunks, vectors):
        chunk_id = hashlib.sha256(
            (chunk["metadata"]["doc_id"] + chunk["text"]).encode()
        ).hexdigest()
        records.append({
            "id": chunk_id,
            "vector": vector,
            "text": chunk["text"],
            "metadata": chunk["metadata"],
        })

    vector_store.upsert(records)
    return len(records)

Using a deterministic ID derived from doc_id plus content means re-indexing an unchanged page is a no-op upsert, and re-indexing a changed page naturally creates new chunk IDs while old ones become orphaned. You still need a cleanup pass that deletes vectors whose doc_id no longer appears in the latest crawl — otherwise deleted or moved pages linger in the index forever, and six months later the bot cites a runbook that was deleted after an incident.

For the vector store itself, pick based on what you already run — pgvector on existing Postgres, or a managed service like Pinecone or Qdrant. For an internal tool serving a few hundred people, pgvector is genuinely fine and keeps your stack simpler; you don't need a dedicated vector database until you're past a few million chunks.

Respecting Permissions at Query Time

This is the step that separates a toy demo from something you can actually deploy without your security team blocking the launch. Confluence space permissions and Notion page-sharing settings both resolve down to "which groups or users can see this page." Capture that during ingestion, and enforce it as a hard filter during retrieval — never as a post-hoc step where the LLM "chooses" not to mention it.

def retrieve(query, user_groups, vector_store, embed_fn, top_k=8):
    query_vector = embed_fn([query])[0]

    results = vector_store.query(
        vector=query_vector,
        top_k=top_k * 3,  # overfetch, then filter
        filter={"allowed_groups": {"$in": user_groups + ["public"]}},
    )

    return results[:top_k]

Overfetching before filtering matters because a naive top_k=8 with a permission filter applied after retrieval can silently return fewer than 8 results, or worse, zero, if the top matches all belong to a restricted space the user can't see. Filtering inside the vector query (most vector DBs support metadata filters natively) is both faster and correct.

If your wiki has genuinely complex permission inheritance — a page inheriting from a parent that inherits from a space default — resolve that at ingestion time into a flat allowed_groups list per page, not at query time. Query time should be a simple set-membership check, nothing more.

Building the Retrieval-Augmented Answer

With permission-filtered chunks in hand, the generation step is where you decide how much the bot hedges versus asserts, and how it cites sources.

SYSTEM_PROMPT = """You are an internal knowledge assistant for {company}.
Answer using ONLY the provided wiki excerpts. Each excerpt has a Page and
Section label — cite them inline like [Page: Deploy Runbook].
If the excerpts don't contain a clear answer, say so explicitly and suggest
who might know (search for an owner in the metadata if present).
Never guess at commands, credentials, or config values not present in the
excerpts."""

def answer_question(query, user_groups, vector_store, embed_fn, llm_fn):
    chunks = retrieve(query, user_groups, vector_store, embed_fn)

    if not chunks:
        return {
            "answer": "I couldn't find anything relevant in the wiki for that. "
                      "Try rephrasing, or ask in #eng-help.",
            "sources": [],
        }

    context = "\n\n---\n\n".join(
        f"[Page: {c['metadata']['title']} | Section: {c['metadata']['heading']}]\n{c['text']}"
        for c in chunks
    )

    messages = [
        {"role": "system", "content": SYSTEM_PROMPT.format(company="Acme")},
        {"role": "user", "content": f"Wiki excerpts:\n\n{context}\n\nQuestion: {query}"},
    ]

    response = llm_fn(messages)

    return {
        "answer": response,
        "sources": [
            {"title": c["metadata"]["title"], "url": c["metadata"]["url"]}
            for c in chunks
        ],
    }

Two details matter more than they look. First, the explicit instruction to admit uncertainty rather than fabricate — internal wikis have gaps, and a bot that confidently invents a deploy command because it pattern-matched from a similar-sounding page is actively dangerous. Second, always return the source list separately from the answer text, even if your UI renders them as a footer. Engineers trust an answer more when they can click through and verify it, and that trust is the entire point of the tool — if people stop clicking through and stop double-checking, that's fine once the bot has a track record, but you want that to be earned, not assumed on day one.

Keeping the Index in Sync

This is the unglamorous 40% of the project that determines whether it's still useful in six months. Two approaches, and most teams end up using both.

Scheduled full re-crawl. Run a job every few hours that pages through all spaces/databases, compares updated_at timestamps against what's in your index, and re-processes only the pages that changed.

def sync_space(space_key, base_url, session, vector_store, last_sync_state):
    cursor = None
    changed_docs = []

    while True:
        docs, cursor = fetch_confluence_pages(base_url, space_key, session, cursor)
        for doc in docs:
            last_known = last_sync_state.get(doc.doc_id)
            if last_known is None or doc.updated_at > last_known:
                changed_docs.append(doc)
                last_sync_state[doc.doc_id] = doc.updated_at
        if not cursor:
            break

    for doc in changed_docs:
        chunks = chunk_wiki_document(doc)
        index_chunks(chunks, embed_fn, vector_store)

    return len(changed_docs)

Webhooks for near-real-time updates. Confluence supports webhook events like page_updated; Notion doesn't have native webhooks on all plans, so a shorter polling interval (every 10-15 minutes) on recently-edited pages is the practical substitute. Either way, treat the webhook as a trigger to re-fetch and re-chunk that single page, not as the source of truth for content — always re-fetch from the API rather than trusting the payload, since webhook bodies are often truncated or missing the full rendered content.

Also budget for a nightly reconciliation job that diffs the full page list against your index and removes chunks for deleted or archived pages. Without this, your bot will eventually cite a page that a 404 greets the user with, which erodes trust fast.

Evaluating Whether It's Actually Working

Before rolling this out beyond a pilot team, build a small evaluation set — twenty to thirty real questions people have actually asked in Slack, with the correct source page noted by hand. Run retrieval against this set whenever you change chunking or embedding models, and check two things: did the correct page show up in the top-k results, and did the generated answer actually reflect it rather than a plausible-sounding hallucination.

def eval_retrieval(eval_set, vector_store, embed_fn):
    hits = 0
    for item in eval_set:
        results = retrieve(item["query"], item["user_groups"], vector_store, embed_fn)
        retrieved_ids = {r["metadata"]["doc_id"] for r in results}
        if item["expected_doc_id"] in retrieved_ids:
            hits += 1
    return hits / len(eval_set)

This is a lightweight version of what you'd build for any production RAG system, but for an internal tool it's often skipped entirely because "it's just for us." Don't skip it — a 60% retrieval hit rate feels fine in a demo and is miserable in daily use, because the failures cluster around exactly the ambiguous, multi-page topics people ask about most (deploy processes, on-call procedures, access requests).

Common Failure Modes and How to Catch Them

A few patterns show up in almost every internal wiki bot I've reviewed or built:

Duplicate pages confusing retrieval. Someone copies a runbook into a new space "temporarily" and never deletes the original. Both get embedded, both get retrieved, and the answer contradicts itself. Fix with a periodic duplicate-detection pass using embedding similarity between page titles plus content, and flag near-duplicates for a human to consolidate.
Stale pages ranking above current ones. If two pages both discuss "VPN setup" and one is from 2021, pure semantic similarity doesn't know which is current. Add a mild recency boost in your ranking, or better, deprioritize/exclude pages past a staleness threshold unless nothing newer exists.
Over-chunking losing procedural context. A five-step deployment procedure split across three chunks means retrieval might grab step 3 without steps 1-2. Keep numbered procedures as a single chunk even if it exceeds your normal token budget — procedural content is one of the few cases where it's worth an explicit exception in the chunker.
Silent permission leaks after reorganizations. When a space gets restructured or renamed, permission metadata can go stale before your next sync. Re-verify allowed_groups on every re-crawl, not just on first ingestion.
The bot answering questions it shouldn't. Someone asks about a competitor's product or something unrelated to internal docs, and the bot tries to answer from general knowledge instead of admitting the wiki has nothing on it. The system prompt's "answer using ONLY the provided excerpts" instruction needs to be tested adversarially, not assumed to hold.

Rolling It Out Without Breaking Trust

Ship this to one team first — ideally a support or platform team that fields a high volume of repetitive questions. Log every question and whether the person clicked a source link or immediately re-asked in Slack; that re-ask signal is your best proxy for "the bot got it wrong" without needing explicit thumbs-down feedback. Review those failures weekly for the first month; almost all of them trace back to either a chunking gap or a page that was never indexed because it lived in a space your crawler didn't have access to.

Once retrieval hit-rate on your eval set stabilizes above your comfort threshold and the failure log stops surfacing new categories of mistakes, widen the rollout. Resist the urge to add scope early — Slack integration, a browser extension, auto-updating tickets — before the core retrieval loop is solid. A wiki bot that answers correctly through a plain web form beats one with five integrations and shaky retrieval.

Building a solid RAG pipeline over messy, permission-gated, constantly-changing internal content is genuinely one of the better ways to learn how retrieval systems behave outside of clean tutorial datasets, and it's a pattern that generalizes well beyond wikis — the same sync-chunk-index-filter loop shows up in support ticket search, codebase Q&A, and customer-facing help centers. If this is your first RAG build, it pairs well with a step back to first principles in our Introduction to RAG course before you tackle the sync and permission layers described here.