Building Graph RAG with Neo4j

Graph rag combines language models with a knowledge graph so retrieval can follow relationships, not only match similar text. Neo4j is a practical foundation because it stores entities and connections directly, supports expressive Cypher queries, and can keep vectors beside structured data. This guide builds a working pipeline for ingestion, entity linking, multi-hop retrieval, prompt assembly, and evaluation.

Why Graph RAG Changes Retrieval

Conventional vector RAG divides documents into chunks, embeds them, and returns chunks that resemble a question. That works when an answer is stated within one passage. It struggles when answering requires joining facts across documents, resolving names, applying filters, or tracing dependencies.

Suppose an engineer asks, "Which services owned by the payments team depend on libraries affected by incident INC-104?" Similarity search may retrieve the incident report, a service catalog entry, and dependency notes. It does not inherently know how to join them. A graph expresses the path explicitly:

(Team)-[:OWNS]->(Service)-[:DEPENDS_ON]->(Library)-[:AFFECTED_BY]->(Incident)

Retrieval becomes a constrained traversal. The system can identify paths, collect supporting properties, and give the model compact evidence. Vector search remains valuable for interpreting natural language and locating unstructured evidence. The graph adds identity, topology, and deterministic joins.

Use graph retrieval when questions contain relationship verbs such as owns, depends on, supplied, approved, reported, or caused. It is useful for multi-hop questions, authorization-aware retrieval, recommendation chains, root-cause analysis, and domains where explaining the evidence path matters.

Design the Graph RAG Schema First

A useful graph begins with the questions it must answer. Avoid modeling every noun in the source material. Start with representative questions, identify the entities and relationships needed for each answer, then design the smallest schema supporting those paths.

For a software operations example, use:

• Document nodes for source provenance
• Chunk nodes for searchable text
• Team, Person, Service, and Library entity nodes
• Incident nodes for operational events
• HAS_CHUNK relationships between documents and chunks
• MENTIONS relationships from chunks to entities
• OWNS, DEPENDS_ON, and AFFECTED_BY domain relationships

Give every entity a stable identifier. Names change and collide, so a display name should not be the primary key. A service might use serviceId, a person an employee identifier, and a library a normalized package coordinate. If the source lacks identifiers, generate a canonical key from normalized type, namespace, and name, while retaining the original label for display.

Every extracted fact needs provenance. Add sourceChunkId, observedAt, confidence, and extractorVersion to relationships, or connect facts to evidence nodes. Provenance lets you show citations, investigate incorrect edges, and rebuild facts produced by an outdated extractor.

Create constraints before ingestion:

CREATE CONSTRAINT document_id IF NOT EXISTS
FOR (d:Document) REQUIRE d.id IS UNIQUE;

CREATE CONSTRAINT chunk_id IF NOT EXISTS
FOR (c:Chunk) REQUIRE c.id IS UNIQUE;

CREATE CONSTRAINT service_id IF NOT EXISTS
FOR (s:Service) REQUIRE s.id IS UNIQUE;

CREATE VECTOR INDEX chunk_embedding IF NOT EXISTS
FOR (c:Chunk) ON c.embedding
OPTIONS {indexConfig: {
  `vector.dimensions`: 1536,
  `vector.similarity_function`: 'cosine'
}};

Match dimensions to the embedding model you use. Treat a model change as a data migration because embeddings from different models should not share an index.

Set Up Neo4j and Python

Run Neo4j in a managed service or local container. Keep credentials in environment variables and use a dedicated application account with only the permissions needed for ingestion or retrieval.

python -m pip install neo4j openai pydantic python-dotenv

Create a small connection module:

import os
from neo4j import GraphDatabase

driver = GraphDatabase.driver(
    os.environ["NEO4J_URI"],
    auth=(os.environ["NEO4J_USERNAME"], os.environ["NEO4J_PASSWORD"]),
)

driver.verify_connectivity()

Reuse one driver for the process. It manages a connection pool, so creating a driver per request wastes resources. Close it during shutdown. Set query timeouts for user-facing retrieval and retry transient failures, not every exception.

Keep transactions small enough to retry safely. For bulk loading, send batches with UNWIND instead of one query per row. Parameterize all values. Never concatenate model-generated text into Cypher.

Ingest Documents and Embeddings

Chunking remains important because chunks carry original wording. Use boundaries respecting headings and paragraphs. Several hundred tokens per chunk with limited overlap is a reasonable starting point, but evaluate it against actual documents and questions.

Assign deterministic chunk identifiers so rerunning ingestion updates nodes rather than duplicating them. Hash the document identifier, section path, position, and normalized content. Store the content hash separately to detect changes.

def write_chunks(tx, document, chunks):
    tx.run(
        """
        MERGE (d:Document {id: $document_id})
        SET d.title = $title, d.updatedAt = datetime()
        WITH d
        UNWIND $chunks AS row
        MERGE (c:Chunk {id: row.id})
        SET c.text = row.text,
            c.position = row.position,
            c.embedding = row.embedding,
            c.contentHash = row.content_hash
        MERGE (d)-[:HAS_CHUNK]->(c)
        """,
        document_id=document["id"],
        title=document["title"],
        chunks=chunks,
    )

Generate embeddings in batches, respect rate limits, and cache by content hash. Store the embedding model name and version on each chunk. If privacy rules prevent sending text to a hosted model, use an approved local model and configure the index dimensions accordingly.

Do not delete old chunks immediately when a document changes. Write the new version, switch the active document relationship, verify the load, then remove orphaned chunks through a controlled cleanup job. This keeps ingestion recoverable.

Extract Entities and Relationships

Entity and relationship extraction turns text into structure. An LLM can propose facts, but the application must validate output. Ask for strict JSON conforming to a closed schema. Reject unknown types, missing evidence, and malformed identifiers.

{
  "entities": [
    {"type": "Service", "id": "checkout-api", "name": "Checkout API"},
    {"type": "Team", "id": "payments", "name": "Payments"}
  ],
  "relationships": [
    {
      "type": "OWNS",
      "from": "payments",
      "to": "checkout-api",
      "evidence": "The Payments team owns Checkout API",
      "confidence": 0.96
    }
  ]
}

Never accept arbitrary labels or relationship names from the model and interpolate them into queries. Map every allowed type to predefined Cypher. This prevents injection and keeps vocabulary consistent.

Entity resolution is harder than extraction. "Checkout", "checkout-api", and "Checkout Service" may refer to one node. Resolve with stable external identifiers first, normalized aliases second, and similarity only as fallback. Put uncertain matches into review instead of merging aggressively. A false merge contaminates every traversal through that node.

Write verified facts idempotently:

MATCH (t:Team {id: $team_id})
MATCH (s:Service {id: $service_id})
MERGE (t)-[r:OWNS]->(s)
SET r.sourceChunkId = $chunk_id,
    r.confidence = $confidence,
    r.extractorVersion = $extractor_version,
    r.updatedAt = datetime()

When several sources support one fact, a single source property is insufficient. Model each assertion as a fact node, attach evidence chunks, and derive the domain edge from accepted assertions. This costs storage but supports conflict analysis and temporal history.

Build Graph RAG Retrieval

A strong graph rag retriever has stages. Classify the question and identify entities. Find seed nodes through exact identifiers, aliases, full-text search, or vector search. Traverse only allowed paths. Rank and serialize evidence within a token budget.

Vector search can locate chunks and entities they mention:

CALL db.index.vector.queryNodes('chunk_embedding', $k, $embedding)
YIELD node AS chunk, score
OPTIONAL MATCH (chunk)-[:MENTIONS]->(entity)
RETURN chunk.id AS chunkId,
       chunk.text AS text,
       score,
       collect({labels: labels(entity), id: entity.id, name: entity.name}) AS entities
ORDER BY score DESC

Returned entities become seeds for constrained Cypher:

MATCH (i:Incident {id: $incident_id})<-[:AFFECTED_BY]-(l:Library)
MATCH (s:Service)-[:DEPENDS_ON]->(l)
MATCH (t:Team)-[:OWNS]->(s)
WHERE toLower(t.name) = toLower($team_name)
RETURN t.name AS team,
       s.name AS service,
       l.name AS library,
       i.id AS incident
ORDER BY service, library
LIMIT 50

Templates are predictable, testable, and easy to authorize. An LLM may select a template and fill parameters. Free-form text-to-Cypher is an advanced feature. If used, provide the schema, restrict the account to read-only access, inspect generated queries, reject mutation clauses, enforce timeouts and limits, and maintain adversarial tests.

Avoid unconstrained variable-length patterns. They can explore huge graph regions and return irrelevant paths. Specify relationship types, direction, maximum depth, labels, tenant filters, and limits. Useful paths in business graphs are usually short.

Rank Graph RAG Paths

More results do not guarantee better answers. Rank paths using semantic similarity, entity match quality, path length, relationship confidence, recency, and source authority. Prefer direct, well-supported paths over long chains containing weak edges.

Deduplicate paths communicating the same fact. Fetch source chunks attached to relationships or entities. Final context should include structured statements and selected original text. Structured data makes joins clear, while source text preserves nuance.

QUESTION
Which payments services were affected by INC-104?

GRAPH FACTS
Payments -> OWNS -> Checkout API
Checkout API -> DEPENDS_ON -> pricing-core
pricing-core -> AFFECTED_BY -> INC-104

SOURCE EVIDENCE
[chunk ops-104-7]
The incident affected pricing-core versions used by Checkout API.

Tell the answer model to use supplied evidence only, distinguish facts from inference, and mention evidence identifiers for important claims. If evidence is incomplete or conflicting, it should say so. Clearly delimit retrieved sources to reduce prompt injection risk.

Add Multi-Hop Retrieval Safely

Multi-hop retrieval should be deliberate, not graph-wide search. Define approved path patterns for common intents. Dependency impact might allow Service -> Library -> Incident, while ownership might allow Person -> Team -> Service. Each pattern becomes a small retrieval tool with typed parameters.

When the starting entity is ambiguous, clarify it. If "Mercury" matches a project, service, and customer, returning all paths confuses ranking. Present options or use conversational context only when it identifies one confidently.

Apply access control inside every query. Tenant, department, document visibility, and user permissions must constrain seeds and traversal. Filtering afterward can leak sensitive properties through counts, paths, or model context.

Limit depth, fan-out, execution time, and context size. Log the template, parameters, returned node identifiers, scores, and latency. These controls make graph traversal observable rather than mysterious.

Evaluate Graph RAG End to End

Evaluate retrieval separately from generation. Build a test set containing questions, expected entities, expected paths, supporting chunks, and acceptable answers. Include lookups, multi-hop questions, ambiguous entities, missing evidence, stale facts, conflicting sources, and unauthorized data.

Useful retrieval measures include:

• Seed entity accuracy
• Expected path recall
• Evidence precision
• Unsupported edge rate
• Retrieval latency
• Context token count

For generation, measure correctness, completeness, faithfulness, citation accuracy, and appropriate refusal. Review failures by stage. A wrong answer may come from missing ingestion, bad resolution, incorrect template selection, excessive pruning, or generation ignoring good evidence.

Maintain deterministic regression tests in continuous integration. Run a larger model-scored evaluation when extraction prompts, embedding models, schemas, ranking weights, or answer prompts change. Human review matters in high-impact domains because automatic judges can reward fluent but unsupported answers.

Compare graph rag against a vector-only baseline. Some questions need no traversal, and routing them directly to vector retrieval reduces cost and latency. Use structure only when structure adds evidence.

Operate Graph RAG in Production

Record the document version, chunker version, embedding model, extractor version, schema version, and ingestion run identifier. These fields allow selective reprocessing.

Monitor query latency, vector latency, traversal fan-out, empty retrieval rate, resolution confidence, extraction rejection rate, and answer refusal rate. Sudden changes often reveal a source format change, broken extractor, or query that lost an index.

Use EXPLAIN and PROFILE on representative queries. Confirm lookups use indexes rather than label scans. Put limits near expansions, return only needed properties, and batch writes. Large text belongs on chunks, not repeated entity nodes.

Plan for deletion and correction. When a source is removed, deactivate its chunks, retract assertions supported only by those chunks, then recompute derived relationships. Soft deletion helps with audits, but retrieval must consistently exclude inactive data.

Back up the database and test restoration. Rebuilding from sources may be possible, but recreating embeddings and extractions can be slow and may differ after model updates. Keep raw sources and extraction artifacts in durable, controlled storage.

Common Graph RAG Mistakes

The first mistake is building an ontology before collecting real questions. This produces complexity with little retrieval value. Work backward from answer paths and expand only when evaluation shows a need.

The second is treating LLM extraction as truth. Output is a candidate assertion. Validate types, identifiers, evidence, confidence, and domain rules. Preserve provenance so every edge can be explained or removed.

The third is merging entities too eagerly. Conservative resolution may leave duplicates, but duplicates can be reviewed later. Incorrect merges silently connect unrelated evidence and are harder to repair.

The fourth is sending raw subgraphs to the answer model. Nodes and relationships need ranking, deduplication, provenance, and token-aware serialization. Otherwise the model receives noise.

The fifth is allowing generated Cypher broad permissions. Prefer templates. When dynamic Cypher is necessary, combine read-only credentials, validation, timeouts, limits, and monitoring.

A Practical Implementation Sequence

Start with one high-value multi-hop question family. Model only the entities and relationships it needs. Load a small trusted corpus, create constraints, write deterministic chunks, and build the vector index.

Implement structured extraction with strict validation and provenance. Manually inspect a sample of entities and edges. Add conservative resolution, then create one parameterized Cypher template.

Combine vector seed discovery with the template, serialize paths and source chunks, and prompt the answer model to cite evidence identifiers. Build an evaluation set before adding more query types.

Then route between vector-only and graph-assisted retrieval. Expand one intent at a time, measure quality and latency, and embed permissions in every query. Graph RAG succeeds when the graph carries stable identity, vectors locate relevant language, and the model explains only what evidence supports.

Before release, run a shadow test against real questions without showing generated answers to users. Compare the selected seeds, traversed paths, evidence chunks, latency, and final response with the current retrieval system. Review empty results and unexpectedly large subgraphs first, because they often expose identifier gaps or missing query constraints. Promote the new route gradually, keep a fast fallback for database or model failures, and sample production traces for human review. Feedback should connect to the retrieval trace, not only the final answer, so engineers can correct the specific node, edge, ranking rule, or source document responsible for an error.

FAQ

Does graph RAG replace vector search?

No. Vector search locates semantically related text, while graph traversal joins entities and follows explicit relationships. Effective systems combine both and route simple questions through the cheaper path.

How many hops should retrieval allow?

Use the shortest path answering the question. Two or three relationships are often enough. Set explicit depths and relationship types instead of arbitrary traversal.

Should an LLM generate Cypher directly?

Prefer tested templates for known intents. If dynamic Cypher is required, use a read-only account, validate queries, reject writes, enforce timeouts and limits, and log execution.

Where should embeddings be stored?

Keeping chunk embeddings in Neo4j places semantic discovery near traversal. A separate vector store also works when stable identifiers connect results to graph nodes.

How do I update facts when documents change?

Version documents and extraction runs, preserve provenance, load new chunks idempotently, retract assertions losing all evidence, and rebuild derived edges in a controlled job.

What quality control matters most?

Maintain an evaluation set with expected entities, paths, and evidence. It reveals whether failures arise in extraction, resolution, retrieval, ranking, or generation.