Graph RAG Explained: When Knowledge Graphs Beat Vector Search

You've shipped a RAG chatbot. It answers "what is our refund policy" beautifully. Then someone asks "which vendors supply components used in products that failed QA last quarter, and who approved those vendors" — and your vector store returns five semi-relevant chunks that don't connect to each other at all. This is the wall every RAG builder eventually hits, and it's the reason graph rag has moved from research curiosity to production pattern in the last two years. This article walks through what graph RAG actually is, when it earns its complexity budget, and how to build a minimal version yourself so you can judge for your own use case instead of taking anyone's word for it.

What vector search actually gets wrong

Standard RAG chunks your documents, embeds them, and retrieves the top-k nearest neighbors to a query embedding. This works well when the answer lives inside one chunk or a small cluster of semantically similar chunks. It breaks down in three specific ways.

First, multi-hop questions. "Who manages the team that owns the service that the payments API depends on?" requires traversing four separate facts. Cosine similarity on a single query embedding has no mechanism for chaining facts together — it just finds text that looks like the question, and there's no guarantee any single chunk contains the full chain.

Second, entity disambiguation and aggregation. Ask "list every incident involving the auth service in the last six months" and vector search will return the chunks most semantically similar to that sentence — which biases toward incidents that literally describe themselves that way, and misses ones phrased differently, or ones where "auth service" is referred to as "identity gateway" three documents later.

Third, global summarization over a corpus. "What are the main themes across all of our customer complaint tickets?" is not a similarity-search question at all — it's an aggregation question. Vector search will hand you the k tickets most similar to "main themes," which is nearly meaningless.

Graph RAG doesn't replace embeddings. It adds a structured layer — a knowledge graph of entities and relationships — that retrieval can traverse explicitly, so the system can answer questions that are shaped like graphs even when the corpus wasn't chunked that way.

It helps to be precise about why chunking itself is the root cause here, because it's tempting to blame the embedding model instead. A chunker splits documents into fixed-size (or semantically-split) windows so each piece fits inside an embedding model's context and stays topically coherent enough to retrieve well. That's a reasonable trade-off for single-topic passages, but it actively destroys cross-document relationships. If your org chart lives in one wiki page and your incident postmortems live in another system entirely, no amount of embedding-model quality will let a similarity search infer that the VP named in the postmortem is the same person mentioned three reorgs ago in the org chart. The information to answer that question exists in your corpus — it's just been fragmented by the retrieval architecture itself. Graph RAG's real contribution is giving you a data structure that survives chunking, because relationships are extracted and stored independently of which chunk they originally came from.

What a knowledge graph actually is, concretely

Strip away the marketing and a knowledge graph is just nodes and edges: entities (people, companies, products, concepts) as nodes, and typed relationships (works_at, depends_on, caused_by, supplies) as edges, usually with properties attached to both. If you've used Neo4j, Amazon Neptune, or even a well-normalized Postgres schema with join tables, you already understand the data structure. What's new in graph RAG is not the graph — it's using an LLM to (a) extract that graph automatically from unstructured text, and (b) use the graph as a retrieval mechanism alongside or instead of vector similarity.

A minimal triple extracted from a sentence like "Acme Corp acquired Widgetron in 2023, and Widgetron's CTO Maria Chen stayed on as VP of Engineering" looks like:

(Acme Corp) -[ACQUIRED {year: 2023}]-> (Widgetron)
(Maria Chen) -[WAS_CTO_OF]-> (Widgetron)
(Maria Chen) -[BECAME]-> (VP of Engineering at Acme Corp)

Once you have thousands of these triples across a corpus, you get something a vector index fundamentally cannot give you: the ability to start at one node and walk outward to find everything connected to it, regardless of whether those connections were ever stated in the same paragraph, same document, or same phrasing.

The two retrieval patterns: local search and global search

Most production graph RAG systems support two distinct query modes, and conflating them is the single most common design mistake.

Local search answers questions anchored to specific entities. "What incidents has the billing service had?" — you identify "billing service" as an entity in the graph, pull its immediate neighborhood (one or two hops out), and feed that subgraph plus the source text chunks associated with those nodes into the LLM as context. This is essentially vector search with an extra join: you still need to find the right starting entity (often via embedding similarity on entity names/descriptions), but the expansion from there is graph traversal, not similarity.

Global search answers questions about the corpus as a whole. "What are the recurring root causes across our incident postmortems?" Here you can't anchor to one entity — you need a summary of the entire graph's structure. The common approach (popularized by Microsoft's GraphRAG research) is to pre-compute community detection over the graph — clustering densely connected subgraphs into "communities" — and generate an LLM summary of each community in advance. At query time, you retrieve the relevant community summaries and synthesize across them, rather than searching raw text at all.

This distinction matters for cost and latency. Local search is cheap and fast — it's a graph query plus one LLM call. Global search requires expensive offline preprocessing (community detection, summarization of every cluster) but makes otherwise-impossible aggregation questions answerable in a single query at runtime.

It's worth being honest about what community detection actually buys you versus what it costs. Algorithms like Leiden or Louvain partition a graph into clusters that are densely connected internally and sparsely connected to the rest of the graph — in practice, this tends to group "all the incidents, services, and people around the payments team" into one community and "all the incidents, services, and people around the mobile app team" into another, even if nobody explicitly labeled them that way. You then run an LLM summarization pass over each community once, offline, and cache the result. The payoff is that a query like "summarize recurring themes in payments-related incidents" becomes a lookup against a handful of pre-written summaries instead of an attempt to stuff hundreds of raw documents into a context window. The cost is that this preprocessing has to be redone (at least incrementally) every time the graph changes meaningfully, and tuning the clustering resolution so communities are neither too broad nor too fragmented takes real iteration — it is not a set-and-forget step.

Building the extraction pipeline

The unglamorous truth about graph RAG is that 80% of the engineering effort is in extraction quality, not retrieval logic. Here's a simplified but functional extraction step using an LLM to pull triples from text chunks, storing them for later graph construction.

import json
from openai import OpenAI

client = OpenAI()

EXTRACTION_PROMPT = """Extract entities and relationships from the text below.
Return JSON with this exact shape:
{
  "entities": [{"name": str, "type": str, "description": str}],
  "relationships": [
    {"source": str, "target": str, "relation": str, "description": str}
  ]
}
Only extract relationships explicitly supported by the text. Do not infer
facts that aren't stated. Use canonical entity names (e.g. "Acme Corp" not
"the company" or "they").

TEXT:
{chunk}
"""

def extract_graph_from_chunk(chunk: str) -> dict:
    response = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[{"role": "user", "content": EXTRACTION_PROMPT.format(chunk=chunk)}],
        response_format={"type": "json_object"},
        temperature=0,
    )
    return json.loads(response.choices[0].message.content)

# Run across every chunk in your corpus, accumulating entities/relationships
all_entities, all_relationships = [], []
for chunk in document_chunks:
    graph_piece = extract_graph_from_chunk(chunk)
    all_entities.extend(graph_piece["entities"])
    all_relationships.extend(graph_piece["relationships"])

Two things will bite you immediately if you skip them. The first is entity resolution: your extraction will produce "Acme Corp," "Acme," and "Acme Corporation" as three separate nodes unless you add a normalization pass (embedding similarity on entity names plus an LLM-based merge decision works reasonably well at small-to-medium scale). The second is temperature and prompt discipline — extraction is exactly the kind of task where an under-constrained prompt invents relationships that sound plausible but aren't in the source text, so temperature=0 and an explicit "don't infer" instruction are not optional.

A third issue shows up once you're past a toy dataset: entity typing consistency. If your prompt doesn't pin down a fixed vocabulary of entity types, the model will invent new ones per chunk — "Person," "Human," "Individual," and "Employee" all showing up for what should be one category. This quietly breaks any downstream filtering or type-aware traversal you try to build later ("show me all people connected to this incident" stops working if half your person-nodes are typed as something else). The fix is mechanical but essential: pass an explicit enum of allowed entity types and relationship types into the extraction prompt, reject or flag any extraction that falls outside it, and periodically audit a sample of extracted triples against the source text — not just for hallucinated facts, but for silently drifting schema. Teams that skip this step usually discover it only after the graph has grown too large to clean up by hand, which is a much more expensive place to catch it than a 50-document pilot.

Loading the graph and running a traversal query

Once you have entities and relationships, load them into an actual graph store. Neo4j is the most common choice because Cypher makes multi-hop traversal readable. Here's a loading step and a query that answers a genuine multi-hop question.

from neo4j import GraphDatabase

driver = GraphDatabase.driver("bolt://localhost:7687", auth=("neo4j", "password"))

def load_graph(entities, relationships):
    with driver.session() as session:
        for e in entities:
            session.run(
                """
                MERGE (n:Entity {name: $name})
                SET n.type = $type, n.description = $description
                """,
                name=e["name"], type=e["type"], description=e["description"],
            )
        for r in relationships:
            session.run(
                """
                MATCH (a:Entity {name: $source}), (b:Entity {name: $target})
                MERGE (a)-[rel:RELATED {type: $relation}]->(b)
                SET rel.description = $description
                """,
                source=r["source"], target=r["target"],
                relation=r["relation"], description=r["description"],
            )

def multi_hop_query(start_entity: str, max_hops: int = 3):
    with driver.session() as session:
        result = session.run(
            """
            MATCH path = (start:Entity {name: $start})-[*1..$hops]-(connected)
            RETURN path
            """,
            start=start_entity, hops=max_hops,
        )
        return [record["path"] for record in result]

Given the earlier example, a query starting at "Widgetron" with max_hops=2 will surface Maria Chen and her new role at Acme Corp — a fact that never appeared in the same sentence, or even necessarily the same document, as "Widgetron." That's the entire value proposition of graph RAG in one traversal.

Hybrid retrieval: combining graph and vector search

In production, you almost never use graph traversal alone. The pattern that actually works is hybrid: use vector similarity to find your entry point(s) into the graph, then traverse from there, then optionally re-rank the combined context before it hits the LLM.

def hybrid_retrieve(query: str, top_k_entities: int = 3, hops: int = 2):
    # Step 1: vector search over entity descriptions to find anchor nodes
    query_embedding = embed(query)
    anchor_entities = vector_index.search(query_embedding, top_k=top_k_entities)

    # Step 2: graph traversal from each anchor
    subgraphs = []
    for entity in anchor_entities:
        subgraphs.extend(multi_hop_query(entity.name, max_hops=hops))

    # Step 3: pull source text chunks tied to nodes in the subgraph
    context_chunks = fetch_source_chunks(subgraphs)

    # Step 4: also run plain vector search as a fallback/supplement
    vector_chunks = vector_index.search(query_embedding, top_k=5)

    combined = deduplicate(context_chunks + vector_chunks)
    return rerank(query, combined)

This hybrid shape matters because pure graph traversal has its own failure mode: if your extraction missed a relationship (and it will — LLM extraction is not perfectly recall-complete), graph-only retrieval silently returns nothing where vector search would have at least found the raw text. Treat the graph as an additional retrieval path, not a replacement.

The re-ranking step deserves a specific note, because it's easy to skip and it's where a lot of the perceived "graph RAG is smarter" quality gain actually comes from. When you combine graph-traversal chunks with vector-search chunks, you often end up with 15-30 candidate passages, many of them only tangentially relevant to the specific question asked (a two-hop traversal picks up a lot of context that's structurally connected but not actually useful for answering this particular query). Passing all of that unranked into the LLM's context window degrades answer quality — the model has to do the work of figuring out what's relevant, and it often gets distracted by the wrong passage. A cross-encoder re-ranker (or even a cheap LLM call asking "rate 0-10 how relevant is this passage to the question") run over the combined candidate set before final generation is a small addition that measurably improves output quality, and it's worth budgeting the extra latency for it rather than skipping straight from retrieval to generation.

When graph RAG is worth the complexity — and when it isn't

This is the section most graph RAG content skips, and it's the one that actually saves you time. Graph RAG adds real cost: an extraction pipeline to build and monitor, a graph database to operate, entity resolution to maintain as your corpus grows, and materially higher indexing latency. None of that is free, and none of it is justified for every RAG project.

Reach for graph RAG when:

• Your questions are genuinely relational — "who reports to whom," "which services depend on which," "what caused what" — and multi-hop reasoning is a frequent query pattern, not an edge case.
• Your corpus has a small, well-defined set of entity types (people, orgs, products, incidents) that recur across many documents, making extraction and resolution tractable.
• You need corpus-level aggregation ("summarize themes across all X") in addition to point lookups.
• You've already tried tuning chunking, metadata filtering, and re-ranking on plain vector RAG and multi-hop questions are still failing.

Stick with plain vector RAG (or vector RAG plus metadata filters) when:

• Most queries are "find the passage that answers this" — single-hop, semantically-anchored lookups. This is the majority of support-doc and internal-wiki RAG use cases, and vector search handles it well and cheaply.
• Your corpus doesn't have clean, extractable entities — narrative text, marketing copy, and legal prose often resist clean triple extraction without heavy custom prompting.
• You can't tolerate the extraction and maintenance overhead — a small team maintaining a graph pipeline on top of an already-complex RAG stack is a real ongoing cost, not a one-time setup.

A practical middle ground many teams land on: keep vector RAG as the default path, and add a lightweight graph layer only for a specific sub-domain where relational questions actually show up in your query logs. You don't need to graph your entire corpus to get the benefit — you need to graph the 20% of it where relationships matter.

Common failure modes when teams build this

A few patterns show up repeatedly enough to call out directly.

• Over-extraction: prompting the LLM to "extract all entities and relationships" with no type constraints produces a graph so dense and noisy it's barely more useful than the raw text. Constrain entity types and relationship types explicitly in your extraction schema.
• Skipping entity resolution: without it, your graph fragments into thousands of near-duplicate nodes and traversal queries miss connections because "Acme Corp" and "Acme" are different nodes.
• No incremental update strategy: teams build the graph once against a static corpus snapshot, then the corpus changes and the graph silently goes stale because nobody built a re-extraction pipeline for new or edited documents.
• Treating community summaries as free: global search's precomputed community summaries need to be regenerated whenever the graph structure shifts meaningfully, which is a real recurring compute cost, not a one-time index build.
• Skipping evaluation: teams eyeball a handful of demo queries and ship. Graph RAG needs the same eval discipline as any RAG system — a labeled set of multi-hop questions with known-correct answers, checked against both the graph-only and hybrid retrieval paths, so you can actually see whether the added complexity is buying you accuracy.

A minimal decision checklist before you build

Before committing engineering time to a graph RAG build, run through this quickly with your actual query logs, not hypothetical ones:

1. Pull the last 100 real user queries against your RAG system (or your best guess at expected queries if pre-launch).
2. Classify each as single-hop (answerable from one chunk) versus multi-hop (requires connecting facts across chunks) versus aggregation (requires summarizing across many documents).
3. If multi-hop and aggregation together are under roughly 15-20% of queries, plain vector RAG with better chunking and re-ranking will likely outperform graph RAG on cost-per-correct-answer.
4. If they're a third or more, prototype the extraction pipeline against a 50-document sample first, and manually audit the extracted triples for precision before building anything else. Bad extraction poisons every downstream query, so this is the step to get right before investing in the graph database, the traversal logic, or the hybrid retrieval layer.
5. Only after extraction quality looks solid should you invest in community detection and global search — it's the most expensive piece and the least necessary for most teams.

Closing thoughts

Graph RAG isn't a universal upgrade over vector search — it's a different tool for a different shape of question, and the right call depends entirely on whether your users are asking "find me the passage" or "connect these facts for me." The extraction pipeline is where almost all the real engineering effort lives, and it's worth prototyping on a small sample before you commit to a graph database and a traversal layer. If you're still getting comfortable with the fundamentals of chunking, embeddings, and retrieval quality before tackling graphs, our Introduction to RAG course on teachyou.ai walks through the full vector RAG pipeline from first principles, which is the right foundation to have solid before you layer graph retrieval on top of it.