Choosing a Vector Database for RAG

Picking a vector database for RAG is less about finding the "best" engine and more about matching a small number of hard constraints, data volume, latency budget, filtering needs, and operational appetite, to the systems that satisfy them. Every vector database on the market today can do approximate nearest neighbor search reasonably well. What actually separates them in production is metadata filtering performance, how they handle updates and deletes, and whether you want to run infrastructure yourself or pay someone else to. This guide walks through the decision in the order it actually comes up when you're building a RAG system, not in marketing-brochure order.

What a vector database for RAG actually needs to do

Before comparing products, it helps to separate the five jobs a vector store does in a RAG pipeline, because most selection mistakes come from optimizing for the wrong one.

1. Store embeddings and their source text/metadata. Every chunk needs an embedding vector plus enough metadata (source URL, document ID, chunk index, timestamp, ACL tags) to reconstruct context and enforce access control.
2. Run approximate nearest neighbor (ANN) search. This is the search-a-billion-vectors-in-milliseconds part everyone talks about. HNSW (Hierarchical Navigable Small World) is the dominant algorithm; IVF-based indexes show up in a few engines too.
3. Filter by metadata alongside the vector search. "Find the closest chunks where tenant_id = 42 and status = 'published'" is the query you'll actually run, not bare ANN search. This is where products diverge the most.
4. Handle upserts and deletes cleanly. Documents get edited and removed. Rebuilding an entire index on every change doesn't scale.
5. Scale storage and query throughput independently. At some point you need more query capacity without paying for more storage, or vice versa.

If you keep this list in front of you, the vendor comparison gets much shorter, because most vendors are strong on (1) and (2) and differentiate on (3) through (5).

Start with the question that eliminates 80% of the options: do you already have Postgres?

If your application already runs on Postgres, pgvector should be your default choice, and you should have a specific reason to move away from it rather than a specific reason to adopt it.

CREATE EXTENSION IF NOT EXISTS vector;

CREATE TABLE document_chunks (
    id BIGSERIAL PRIMARY KEY,
    document_id UUID NOT NULL,
    tenant_id INT NOT NULL,
    chunk_text TEXT NOT NULL,
    embedding VECTOR(1536),
    created_at TIMESTAMPTZ DEFAULT now()
);

CREATE INDEX ON document_chunks
    USING hnsw (embedding vector_cosine_ops)
    WITH (m = 16, ef_construction = 64);

CREATE INDEX ON document_chunks (tenant_id);

A query that combines vector search with metadata filtering is just SQL:

SELECT id, chunk_text, 1 - (embedding <=> $1) AS similarity
FROM document_chunks
WHERE tenant_id = $2
ORDER BY embedding <=> $1
LIMIT 10;

The reasons to stay on pgvector: you keep one database to operate, one backup strategy, one connection pool, and transactional consistency between your application data and your embeddings (inserting a document and its chunks in the same transaction is genuinely useful). Postgres's query planner also handles the filter-then-search and search-then-filter tradeoff reasonably well as of recent versions, especially with the iterative scan mode that newer pgvector releases support for HNSW.

The reasons to leave: once you're past roughly 10-50 million vectors, or your filter predicates are highly selective (filtering out 99% of rows before the vector search even runs), pgvector's recall can degrade unless you tune ef_search carefully, and a single Postgres instance becomes a scaling bottleneck for query throughput. If you're already sharding Postgres for other reasons, this is a natural point to reconsider.

If you need a dedicated engine, decide on managed vs. self-hosted first

This is the second fork, and it matters more than picking between HNSW variants.

Managed (Pinecone, Zilliz Cloud, Weaviate Cloud, MongoDB Atlas Vector Search): you get index tuning, replication, and scaling handled for you, at a real dollar cost per GB and per query, and with data residency constraints you need to check against your compliance requirements. This is the right call when your team doesn't want to own another stateful service, or when you need to move fast and revisit infrastructure later.

Self-hosted (Qdrant, Milvus, Weaviate open source, Chroma for smaller workloads): you run the cluster, which means you own upgrades, backups, and capacity planning, but you avoid vendor lock-in and per-query pricing, and you can colocate the vector store with your inference infrastructure to cut network latency. This is the right call when you have infra capacity already, when data can't leave your VPC, or when query volume is high enough that managed pricing stops making sense.

There's no universally correct answer here; it's a build-vs-buy decision like any other, and the deciding factor is almost always team capacity, not raw feature comparison.

Comparing the dedicated engines on the things that actually differ

Assuming you've decided you need a dedicated vector database for rag rather than pgvector, here's where Qdrant, Weaviate, Milvus, and Pinecone genuinely diverge.

Filtering architecture. Qdrant built filtering into the HNSW graph traversal itself (it skips non-matching nodes during graph walk rather than filtering after retrieval), which keeps recall high even under selective filters. This matters a lot for multi-tenant RAG where every query is scoped to one tenant. Milvus and Weaviate both support filtered search but historically leaned more on pre-filtering or post-filtering strategies that can hurt recall on very selective filters; check current benchmarks for your filter selectivity before committing, since this is an area vendors actively improve.

Multi-tenancy model. If you're building RAG for multiple customers, ask each candidate specifically how it isolates tenants: separate collections per tenant, a single collection with a tenant field, or dedicated namespaces (Pinecone's approach). Namespaces and payload-based multi-tenancy (Qdrant) tend to scale to thousands of tenants more gracefully than one-collection-per-tenant, which runs into per-collection overhead at scale.

Hybrid search (dense + sparse/keyword). Pure semantic search misses exact-match queries, product codes, and rare terms. Weaviate, Qdrant, and Milvus all support hybrid dense+sparse retrieval natively now (BM25 or SPLADE fused with vector scores); this used to require bolting on Elasticsearch separately. If your RAG corpus includes technical documentation with specific identifiers, hybrid search is not optional, it's table stakes.

Update and delete cost. Milvus historically favored append-heavy, batch-rebuilt workloads well suited to large static corpora. Qdrant and Weaviate handle point-level upserts and deletes more cheaply, which matters if your source documents change frequently, think a support-ticket knowledge base updated hourly versus a quarterly compliance manual.

Operational maturity at your scale. Below about 10 million vectors, almost every engine performs similarly and the choice is really about developer experience and existing team familiarity. Above 100 million vectors, you want to look specifically at sharding and replication docs, not just the ANN benchmark numbers, since operational behavior under real write load is where engines separate.

A concrete decision framework

Answer these in order; the first "yes" tells you where to look.

1. Already on Postgres, and under ~20M vectors with moderate filter selectivity? Use pgvector. Stop here.
2. Need zero infrastructure ownership and can accept per-query/per-GB pricing? Use a managed service (Pinecone, Weaviate Cloud, Zilliz Cloud, or your cloud provider's native offering). Pick based on region availability and existing cloud contracts.
3. Multi-tenant SaaS with tenant-scoped filters on every query? Prioritize Qdrant or a managed service with strong namespace isolation; benchmark filtered recall specifically, don't trust unfiltered ANN benchmarks.
4. Corpus mixes exact-match terms (SKUs, IDs, jargon) with natural language? Prioritize an engine with mature hybrid search: Weaviate, Qdrant, or Milvus with BM25/sparse fusion enabled.
5. Hundreds of millions of vectors, write-heavy, need to self-host for compliance? Milvus or Qdrant, and budget real engineering time for cluster operations; this tier is not a weekend setup regardless of which engine you pick.

Testing the finalists before you commit

Don't pick a vector database for rag off a comparison table. Load a representative sample of your actual corpus (not a synthetic benchmark dataset) into your top two candidates and measure three things: filtered query latency at your expected concurrency, recall against a hand-labeled set of query/relevant-chunk pairs, and index build/rebuild time as your corpus grows. Here's a minimal harness using Qdrant's Python client as an example; the same shape works for any client library.

import time
from qdrant_client import QdrantClient
from qdrant_client.models import Distance, VectorParams, PointStruct, Filter, FieldCondition, MatchValue

client = QdrantClient(url="http://localhost:6333")

client.recreate_collection(
    collection_name="rag_eval",
    vectors_config=VectorParams(size=1536, distance=Distance.COSINE),
)

def upsert_batch(chunks):
    points = [
        PointStruct(
            id=chunk["id"],
            vector=chunk["embedding"],
            payload={"tenant_id": chunk["tenant_id"], "text": chunk["text"]},
        )
        for chunk in chunks
    ]
    client.upsert(collection_name="rag_eval", points=points)

def timed_filtered_search(query_vector, tenant_id, top_k=10):
    start = time.perf_counter()
    results = client.search(
        collection_name="rag_eval",
        query_vector=query_vector,
        query_filter=Filter(
            must=[FieldCondition(key="tenant_id", match=MatchValue(value=tenant_id))]
        ),
        limit=top_k,
    )
    elapsed_ms = (time.perf_counter() - start) * 1000
    return results, elapsed_ms

def recall_at_k(labeled_queries, k=10):
    hits = 0
    for query in labeled_queries:
        results, _ = timed_filtered_search(query["embedding"], query["tenant_id"], top_k=k)
        returned_ids = {r.id for r in results}
        if query["relevant_id"] in returned_ids:
            hits += 1
    return hits / len(labeled_queries)

Run this against 500-1000 real queries with known relevant chunks (you can bootstrap the labeled set by having your team mark a handful of "this chunk should answer this question" pairs), and you'll have actual latency and recall numbers instead of vendor claims. Do this before committing to a schema or a contract, migrating a populated vector index later is far more painful than migrating an empty one.

Common mistakes worth naming directly

Choosing based on unfiltered ANN benchmarks. Public benchmarks (ann-benchmarks.com and similar) mostly measure raw ANN speed without metadata filters. Your production queries almost always carry a filter. Benchmark with your filters, not without them.

Ignoring embedding dimensionality cost. A 3072-dimension embedding model roughly doubles storage and search cost versus a 1536-dimension one for marginal recall gains in many domains. Check whether a smaller embedding model meets your recall bar before assuming you need the largest available model.

Treating the vector database as the whole RAG system. Retrieval quality depends at least as much on chunking strategy and reranking as on which ANN index you chose. A mediocre vector database with good chunking and a reranker will usually beat a great vector database with naive fixed-size chunking.

Skipping a reindex plan. Embedding models get upgraded. Plan for how you'll re-embed and re-index your corpus without downtime before you need to do it under pressure; some engines support blue-green collection swaps natively (create a new collection, backfill, then atomically switch an alias), which is worth checking for up front.

FAQ

Is pgvector good enough for production RAG? Yes, for most applications under roughly 10-50 million vectors with moderate query volume, especially if you're already running Postgres. It stops being sufficient when you need very high query throughput, extremely selective filters at scale, or you want to decouple vector search scaling from your primary database.

Do I need a dedicated vector database if I'm just prototyping? No. Use pgvector, SQLite with a vector extension, or even an in-memory FAISS index for prototyping. Switching vector stores later is a schema migration, not a rewrite, as long as you keep your embedding generation and chunking logic decoupled from the storage layer.

How much does metadata filtering actually matter? A lot, more than most teams expect before they build multi-tenant RAG. If every query in production carries a filter (tenant ID, document type, date range, access control), filtered recall and filtered latency are the numbers that matter, not raw unfiltered ANN speed.

Should I pick the vector database based on which embedding model I use? No, keep these decoupled. Any modern vector database accepts arbitrary-dimension float vectors and doesn't care which model produced them. Pick the embedding model based on retrieval quality for your domain, and pick the vector database based on the operational criteria in this guide.

Is hybrid search (keyword plus vector) worth the added complexity? For most real-world RAG corpora, yes. Pure semantic search regularly misses exact identifiers, product codes, and rare technical terms that a BM25 or sparse-vector component catches. Most engines now support hybrid search natively, so the added complexity is usually a config flag rather than a second system to run.

Can I switch vector databases later without redoing everything? Mostly yes, if you've kept chunking, embedding generation, and metadata schema in your own application code rather than baked into a vendor's ingestion pipeline. The migration cost is re-embedding (if you don't keep raw vectors archived separately) and re-upserting, not rewriting your retrieval logic, provided you built a thin abstraction layer over the vector store client from the start.