Sharding and Scaling Vector Databases

Vector database sharding means splitting one logical collection of embeddings across many machines so that no single node holds the whole index. You reach for it when your vectors no longer fit in one node's RAM, when insert throughput saturates a single writer, or when query latency climbs because one machine is scanning too much. This guide covers the two sharding strategies that actually matter (partition by document ID versus partition by vector cluster), how queries fan out and merge across shards, and the operational traps that quietly wreck recall.

Why vector database sharding is different from row sharding

If you have sharded a relational database, you know the drill: pick a shard key, hash it, route each row to a node, and a lookup by that key hits exactly one shard. Vector search breaks this model because the query is not a key lookup. A nearest-neighbor query asks "which of my billion vectors are closest to this one," and the answer can live on any shard. There is no key to route on.

That single fact drives every decision in vector database sharding. Because the nearest neighbors can be scattered anywhere, most sharded vector systems have to send each query to every shard, gather partial results, and merge them. This is a scatter-gather (also called fan-out) pattern. Understanding when you can avoid the full fan-out is where the real engineering lives.

The second difference is memory. An approximate nearest neighbor (ANN) index like HNSW keeps its graph in RAM for speed. A 768-dimension float32 vector is about 3 KB raw, and the HNSW graph adds pointer overhead on top. One billion of those vectors is multiple terabytes of RAM before you count replicas. No single server holds that, so sharding is not optional at that scale. It is the only way to fit the index at all.

The third difference is that recall is probabilistic. When you split an index and query it approximately, you can lose neighbors that a single unified index would have found. A big part of doing vector database sharding well is making sure the split does not silently degrade result quality.

When you actually need to shard

Do not shard early. A single modern node with a lot of RAM handles tens of millions of vectors comfortably, and replication (read replicas of the same full index) solves query throughput without any of the fan-out complexity. Shard only when one of these is true:

• The index no longer fits in one node's RAM, even after quantization. This is the hard limit. Once your working set spills to disk, tail latency falls off a cliff.
• Write throughput saturates a single primary. HNSW inserts are CPU-heavy because each insert walks the graph. If one writer cannot keep up with your ingest rate, splitting writes across shards helps.
• Your dataset is naturally multi-tenant and you want isolation, so tenant A's queries never touch tenant B's data.

Notice that pure read QPS is usually NOT a reason to shard. Replication is simpler and gives you linear read scaling without splitting the index. Reach for sharding when the data itself is too big or the writes are too fast, and lean on replicas for read load. Real production clusters combine both: N shards, each replicated M times.

Strategy one: shard by document ID (random partitioning)

The default strategy in Milvus, Weaviate, Qdrant, and Elasticsearch/OpenSearch is to partition vectors by a hash of their ID. Each incoming vector gets assigned to a shard by hash(id) % num_shards, so vectors are spread evenly and each shard builds its own independent ANN index over roughly total / num_shards vectors.

The appeal is balance. Data spreads uniformly, every shard is the same size, and writes distribute evenly. The cost is that every query must fan out to every shard, because a query's true neighbors could be on any of them.

Here is the query path with ID-based sharding, in pseudocode:

def search(query_vec, k, shards):
    partials = []
    for shard in shards:            # runs in parallel, one RPC per shard
        partials += shard.search(query_vec, k)   # each returns its own top-k
    # merge: global top-k across all partial results
    merged = sorted(partials, key=lambda hit: hit.distance)[:k]
    return merged

Two subtle points make or break this. First, each shard must return k results, not k / num_shards. If you have 10 shards and you want the global top 10, you cannot ask each shard for 1 result. The 10 true nearest neighbors might all live on a single shard. Every shard returns its own top-k, and the coordinator merges num_shards * k candidates down to the final k. That means the coordinator's merge cost grows with shard count.

Second, latency is governed by the slowest shard. A query finishes only when the last shard replies, so p99 latency across a fan-out is worse than any single shard's p99. This is the tax of scatter-gather. More shards means more chances that one of them is having a slow moment (garbage collection pause, a hot query, a cold cache), and your query waits for it.

To keep ID-sharding healthy:

• Keep k per shard equal to the requested global k. Do not divide.
• Set a per-shard timeout and decide your policy: return partial results, or fail the whole query. For RAG, partial results are usually acceptable; for exact recall guarantees, they are not.
• Watch the merge step. With very large k and many shards, sorting num_shards * k candidates on the coordinator becomes non-trivial.

Strategy two: shard by cluster (semantic partitioning)

The more advanced strategy is to partition by where vectors live in the embedding space, not by ID. You run a coarse clustering step (k-means over a sample of your vectors) to learn, say, 256 centroids. Each vector is assigned to the shard that owns its nearest centroid. Now vectors that are close together in embedding space tend to land on the same shard.

The payoff is query pruning. At query time, you compute the query's distance to every centroid, pick the few closest centroids (this is the nprobe parameter in IVF-style indexes), and only route the query to the shards that own those centroids. Instead of fanning out to all 256 shards, you touch maybe 8. That cuts the work per query dramatically and sidesteps the slowest-shard tax because fewer shards are involved.

This is exactly the idea behind IVF (inverted file) indexes, scaled out across machines. FAISS implements IVF in a single process; cluster-based sharding is IVF where each inverted list (or group of lists) lives on its own node.

The catch is threefold, and each one bites in production:

• Load imbalance. Real embedding distributions are lumpy. Some clusters are dense (a popular topic) and some are sparse. Cluster-based shards end up unequal in size and query traffic, so one shard becomes a hot spot while others idle.
• Boundary recall loss. A query near a cluster boundary has true neighbors sitting just across the line in a shard you did not probe. If you probe too few clusters, recall drops. You tune nprobe up to recover recall, which erodes the pruning benefit.
• Rebalancing pain. As data drifts (new topics arrive, embeddings from a new model version), the old centroids stop matching the data. You periodically re-cluster and reshuffle vectors, which is expensive and needs careful online migration.

Cluster-based sharding shines when your data is large, relatively stable, and your latency budget is tight enough that fanning out to every shard is unacceptable. If your data churns fast or you cannot tolerate periodic re-clustering, ID-based sharding is less clever but far more robust.

Choosing between the two

A practical rule of thumb:

• Default to ID-based (random) sharding. It is what the mainstream engines do out of the box, it stays balanced automatically, and its failure modes are predictable. Scatter-gather to all shards is fine up to a few dozen shards.
• Move to cluster-based sharding only when fan-out cost becomes the bottleneck, typically at hundreds of shards or when p99 latency under full fan-out blows your budget, and only if your data is stable enough to re-cluster occasionally.
• If you are multi-tenant, consider a third axis entirely: shard by tenant. Give large tenants their own shard(s) and pack many small tenants into shared shards. This gives isolation and lets a tenant's query touch only its own data, which is both faster and safer.

Replication sits on top of sharding

Sharding splits the data; replication copies each shard. You want both. A shard with no replica is a single point of failure and a throughput ceiling. The usual layout is a primary per shard that takes writes, plus one or more replicas that serve reads and stand by for failover.

The tension is consistency. When a write lands on a shard's primary, the replica has to catch up. Most vector databases replicate asynchronously for speed, which means a freshly inserted vector may not be searchable on a replica for a short window. For RAG and search, this eventual consistency is usually fine. If you need read-your-writes (you insert a document and immediately query for it), check whether your engine supports routing that query to the primary or offers a consistency level knob. Milvus, for example, exposes tunable consistency levels precisely for this reason.

A concrete layout for one billion vectors might be:

• 20 shards, roughly 50 million vectors each, sized so each shard's index fits in one node's RAM after quantization.
• 3 replicas per shard for availability and read throughput.
• 60 index-serving nodes total, plus a coordinator/query tier that fans out and merges.

Keeping recall honest across shards

The quiet danger of sharding is that recall degrades and nobody notices, because the queries still return results, just slightly worse ones. Guard against it:

• Measure recall against a brute-force baseline. Take a sample of real queries, compute the exact top-k with a flat (brute-force) search over the full dataset offline, then compare your sharded ANN results against that ground truth. Recall at 10 is the fraction of the true top-10 you actually returned. Track it as a first-class metric, not an afterthought.
• Do not over-shard. Every shard boundary is a chance to miss a neighbor. If ten shards give you the recall you need, do not jump to a hundred for marginal latency gains.
• Tune per-shard search width. HNSW's ef_search (the size of the candidate list explored at query time) trades latency for recall. When you shard, you may need to raise ef_search per shard to keep global recall where it was on a single index, because each shard now sees fewer vectors and the graph is smaller.
• For cluster sharding, tune nprobe against recall, not against a guess. Sweep it, plot recall versus latency, and pick the knee of the curve.

Here is a minimal recall check you can run against any sharded system that speaks a search API:

import numpy as np

def recall_at_k(query_vecs, sharded_search, exact_search, k=10):
    hits = 0
    total = 0
    for q in query_vecs:
        approx = set(h.id for h in sharded_search(q, k))
        truth  = set(h.id for h in exact_search(q, k))   # brute force
        hits  += len(approx & truth)
        total += k
    return hits / total

# exact_search does a full linear scan over all vectors, no ANN, no shards.
# Run it offline on a sample. If recall_at_k drops after you add shards,
# raise ef_search / nprobe before shipping.

Routing, coordinators, and the write path

Someone has to decide which shard a query or write goes to. That is the coordinator (Milvus calls its components proxy and query coordinator; other systems fold it into a smart client or a gateway). For ID-based sharding the coordinator fans a query to all shards. For cluster-based sharding it first scores the query against the centroids, then routes only to the relevant shards.

The write path matters just as much. When a new vector arrives:

• The coordinator computes its target shard (hash(id) for random, nearest-centroid for cluster).
• The vector goes to that shard's primary, which inserts it into its ANN index.
• The primary streams the change to replicas.

HNSW insertion is not free. Building the graph edges for a new node means running a search inside the graph, so ingest is CPU-bound. High-ingest systems often decouple this: writes go to a fast append-only segment first, and a background process periodically builds or merges the ANN index over sealed segments. This is why many vector databases have a notion of "growing" versus "sealed" segments. Fresh data is searchable via a slower brute-force scan of the small growing segment, while the bulk of the data is served from optimized sealed indexes. When you shard, this segment lifecycle happens independently on each shard.

Quantization: shard less by shrinking vectors first

Before you add shards, shrink the vectors. Sharding exists partly to fit the index in RAM, and quantization attacks the same problem from the other side. Product quantization (PQ) compresses each vector into a compact code, often cutting memory by 8x to 32x with a modest recall hit. Scalar quantization to int8 roughly quarters memory versus float32 with very little recall loss for many embedding models. Binary quantization is even more aggressive and works surprisingly well for some high-dimensional embeddings when paired with a re-ranking pass.

The practical order of operations:

• Quantize first. If int8 lets your index fit in one node, you may not need to shard at all.
• If it still does not fit, shard, and quantize each shard's index too. The two are complementary.
• Keep full-precision vectors on disk (or in an object store) for an optional re-rank step: retrieve k * some_factor candidates using the quantized index, then re-score the top candidates against full-precision vectors to recover recall.

This re-rank pattern is why many production stacks separate the "coarse" quantized index (fast, in RAM, sharded) from the "fine" exact vectors (on disk, used only to re-score a handful of finalists).

A checklist before you shard in production

• Confirm you actually need it. Have you quantized? Have you tried a bigger single node with read replicas? Sharding adds a coordinator, fan-out latency, and rebalancing chores. Earn it.
• Pick the strategy deliberately. ID-based for balance and simplicity, cluster-based for fan-out reduction on stable data, tenant-based for isolation.
• Size shards to fit in RAM after quantization, with headroom for the growing segment and query working memory.
• Replicate every shard at least once. No naked shards in production.
• Instrument recall against a brute-force baseline, and alert on it. Recall regressions are silent.
• Decide your partial-result policy. Under a shard timeout, do you return what you have or fail? RAG usually tolerates partial; compliance search may not.
• Plan rebalancing before you need it. For cluster sharding, how and when do you re-cluster? For ID sharding, how do you add a shard without rehashing everything (consistent hashing helps here)?

FAQ

Does sharding a vector database reduce recall? It can, if you do it carelessly. Each shard runs its own approximate index, and merging partial results can miss neighbors that a single unified index would have found, especially with cluster-based sharding near cluster boundaries. You keep recall high by returning a full top-k from every shard (never k / num_shards), raising ef_search or nprobe to compensate for smaller per-shard indexes, and continuously measuring recall against a brute-force baseline.

How many shards should I start with? Start with the fewest that let each shard's index fit in one node's RAM after quantization, then add replicas for throughput. Over-sharding hurts: it raises fan-out latency (you wait for the slowest shard), grows the coordinator's merge cost, and adds more boundaries where recall can leak. Ten well-sized shards usually beat a hundred tiny ones.

Should I shard or just add read replicas? If your problem is query throughput and the index fits in one node, use read replicas. Replication scales reads linearly without any fan-out complexity. Shard only when the data is too big for one node's RAM or a single writer cannot keep up with ingest. Real clusters do both: shard for size, replicate each shard for availability and read load.

What is the difference between ID-based and cluster-based sharding? ID-based (random) sharding hashes each vector's ID to a shard, keeping shards balanced but forcing every query to fan out to all shards. Cluster-based sharding groups vectors by their position in embedding space (via k-means centroids) so a query can skip shards that hold no nearby vectors, cutting fan-out at the cost of load imbalance, boundary recall loss, and periodic re-clustering. Default to ID-based; move to cluster-based only when full fan-out becomes your bottleneck.

Do I need a coordinator node? Yes, logically. Something must route writes to the right shard and, for reads, fan queries out and merge the partial results into a global top-k. Some systems run this as a dedicated tier (a proxy plus a query coordinator), others fold it into a smart client library. Either way, the merge step is real work: it sorts up to num_shards * k candidates, so watch its cost as you add shards.

How does quantization interact with sharding? They solve the same RAM problem from opposite ends and stack cleanly. Quantize first (int8 scalar quantization often quarters memory with little recall loss); you may fit in one node and skip sharding entirely. If you still overflow, shard and quantize each shard. Keep full-precision vectors on disk for an optional re-rank pass: pull extra candidates from the quantized index, then re-score the finalists against exact vectors to recover the recall quantization cost you.

Can I add a shard later without rebuilding everything? With plain hash(id) % num_shards, adding a shard changes the modulus and forces almost every vector to move. Use consistent hashing (or a virtual-node scheme) so adding a shard only relocates a fraction of vectors. For cluster-based sharding, adding capacity usually means re-clustering, which is a heavier operation, so plan the migration path before you commit to that strategy.