Metadata Indexing in Vector Databases

Vector metadata indexing is the practice of attaching structured, filterable fields (tenant ID, timestamp, document type, permission level, category) to each vector you store, then building indexes over those fields so a query can combine similarity search with exact filters in one pass. Without it, a vector database can only tell you "these chunks are semantically close," and you're stuck writing separate logic to figure out which chunks the current user is even allowed to see or which chunks belong to the right document. If you're building a RAG pipeline, a multi-tenant search product, or anything that mixes "find similar things" with "find similar things that match these conditions," metadata indexing is not optional polish, it's the mechanism that keeps retrieval correct.

This article walks through why metadata indexing matters, how it works differently from a normal database index, the filtering strategies (pre-filter, post-filter, single-stage), and hands-on examples across Pinecone, Qdrant, Weaviate, and pgvector so you can pick the right pattern for your stack.

Why plain vector search isn't enough

A vector index (HNSW, IVF, or a flat index) is built to answer one question fast: "give me the k nearest vectors to this query vector." It knows nothing about who owns a document, when it was created, or what category it belongs to. If you store embeddings for every customer's support tickets in one collection and a query returns the top 10 nearest neighbors, there's a real chance those neighbors belong to a different customer entirely. That's a data leak, not a bug you can patch after the fact.

The naive fix is to fetch a large number of nearest neighbors (say 200) and filter the list in application code, discarding anything that doesn't match the tenant or permission you need. This works at small scale and breaks at real scale for two reasons:

• You waste compute and network fetching hundreds of vectors just to throw most of them away.
• Recall degrades. If your actual matches are ranked 250th and 400th because the collection is dominated by other tenants' data, a top-200 fetch never surfaces them, and the user sees an empty result even though good matches exist.

Metadata indexing solves this by pushing the filter into the search itself, so the nearest-neighbor algorithm only considers vectors that already satisfy the filter, or reliably converges to enough vectors that do.

How metadata indexing actually works

Under the hood there are three broad strategies vector databases use to combine filtering with ANN (approximate nearest neighbor) search. Which one your database implements affects both correctness and speed, so it's worth knowing the difference.

Pre-filtering. The database first evaluates the metadata filter (often using a standard inverted index or B-tree style structure, the same kind used in traditional databases) to produce a candidate set of vector IDs, then restricts the ANN search to that subset. This gives exact recall relative to the filter but can be slow if the candidate set is large, because graph-based indexes like HNSW aren't naturally built to search over an arbitrary subset efficiently.

Post-filtering. The database runs the normal ANN search first, retrieves more results than requested (an "overfetch"), then filters by metadata afterward and truncates to the requested k. This is fast when the filter matches most of the data, but degrades badly (as described above) when the filter is selective, i.e., matches a small fraction of the collection.

Single-stage filtered search (integrated filtering). The newest and most sophisticated approach, used by Qdrant and increasingly by others, weaves the metadata check directly into the graph traversal. As HNSW walks the graph looking for close neighbors, it checks the filter condition on each candidate node and skips ones that don't match, while still following edges to unmatched nodes so it doesn't get stuck in a filtered-out region of the graph. This gives both good recall and good latency even for highly selective filters, because it avoids the "search a huge candidate set" cost of pre-filtering and the "throw away most of what we fetched" cost of post-filtering.

When you evaluate a vector database, ask specifically which strategy it uses for filtered search, and whether that behavior changes based on filter selectivity. Some databases pick a strategy automatically (falling back to pre-filtering when the estimated candidate set is small, and post-filtering otherwise), which is the ideal but not universal behavior.

Designing your metadata schema

Before you touch code, decide what fields actually need to be indexed versus what can just ride along as payload. Every field you index costs memory and write latency; every field you don't index can't be filtered on efficiently. A reasonable rule: index anything you will filter or sort by in a query, and leave everything else (raw text, source URLs, display titles) as unindexed payload that you fetch alongside the vector but never filter on.

Typical fields worth indexing in a RAG or search system:

• tenant_id or org_id: almost always required for multi-tenant isolation, and should be a mandatory filter on every query, not an optional one.
• document_id and chunk_index: lets you fetch all chunks belonging to one document, or exclude a document being re-indexed.
• created_at / updated_at: for time-windowed queries ("only search documents from the last 90 days") and for invalidating stale chunks after a re-embed.
• category or doc_type: lets you scope a query to, say, "policy documents" versus "meeting notes."
• access_level or role_tags: enforces permission-aware retrieval so a query never returns chunks the requesting user shouldn't see.
• language: useful once you have multilingual content and want to avoid mixing languages in one result set.

Keep field types simple and consistent. Most vector databases support keyword/string filters, numeric range filters, and boolean filters well; nested objects and arrays are supported by some (Qdrant and Weaviate both handle arrays of tags reasonably) but add complexity, so only reach for them when a flat schema genuinely doesn't fit.

Hands-on: Pinecone metadata filtering

Pinecone stores metadata as a JSON-like object alongside each vector and lets you filter with a Mongo-style query syntax at search time. Metadata fields are automatically indexed unless you explicitly configure selective metadata indexing on a serverless index to reduce index size.

from pinecone import Pinecone

pc = Pinecone(api_key="YOUR_API_KEY")
index = pc.Index("support-tickets")

# Upsert with metadata
index.upsert(vectors=[
    {
        "id": "chunk-001",
        "values": [0.12, 0.98, -0.44, 0.05],
        "metadata": {
            "tenant_id": "acme-corp",
            "doc_type": "ticket",
            "created_at": 1732147200,
            "access_level": "internal"
        }
    }
])

# Query with a metadata filter
results = index.query(
    vector=[0.10, 0.95, -0.40, 0.02],
    top_k=5,
    filter={
        "tenant_id": {"$eq": "acme-corp"},
        "created_at": {"$gte": 1729468800},
        "access_level": {"$in": ["internal", "public"]}
    },
    include_metadata=True
)

for match in results["matches"]:
    print(match["id"], match["score"], match["metadata"])

Pinecone's serverless tier lets you mark specific metadata fields as indexed versus not, which matters at scale: unindexed fields still travel with the vector for display purposes but aren't usable in a filter clause, and they don't consume the same indexing overhead.

Hands-on: Qdrant payload indexing

Qdrant calls metadata "payload" and requires you to explicitly create a payload index on any field you intend to filter, which gives you fine control and lets Qdrant use its integrated filtering during HNSW traversal.

from qdrant_client import QdrantClient
from qdrant_client.models import (
    PointStruct, Filter, FieldCondition, MatchValue, Range,
    PayloadSchemaType
)

client = QdrantClient(url="http://localhost:6333")

# Create the payload index once, up front
client.create_payload_index(
    collection_name="support_tickets",
    field_name="tenant_id",
    field_schema=PayloadSchemaType.KEYWORD
)
client.create_payload_index(
    collection_name="support_tickets",
    field_name="created_at",
    field_schema=PayloadSchemaType.INTEGER
)

# Upsert a point with payload
client.upsert(
    collection_name="support_tickets",
    points=[
        PointStruct(
            id=1,
            vector=[0.12, 0.98, -0.44, 0.05],
            payload={
                "tenant_id": "acme-corp",
                "doc_type": "ticket",
                "created_at": 1732147200
            }
        )
    ]
)

# Filtered search
hits = client.search(
    collection_name="support_tickets",
    query_vector=[0.10, 0.95, -0.40, 0.02],
    query_filter=Filter(
        must=[
            FieldCondition(key="tenant_id", match=MatchValue(value="acme-corp")),
            FieldCondition(key="created_at", range=Range(gte=1729468800))
        ]
    ),
    limit=5
)

for hit in hits:
    print(hit.id, hit.score, hit.payload)

Because Qdrant indexes payload fields explicitly, a query on an unindexed field falls back to a full scan of matching filter conditions, which is fine for small collections but will hurt you once you're past a few hundred thousand points. Always index the fields your application actually filters on, and check Qdrant's collection info endpoint periodically to confirm the indexes you expect are actually present.

Hands-on: Weaviate where filters

Weaviate exposes metadata as regular schema properties and lets you combine vector search with a where filter, with the option to run pre-filtering (default) or a hybrid mode that blends vector and keyword scoring.

import weaviate
import weaviate.classes as wvc

client = weaviate.connect_to_local()
collection = client.collections.get("SupportTicket")

# Insert an object with properties (Weaviate auto-vectorizes or accepts your own vector)
collection.data.insert(
    properties={
        "tenantId": "acme-corp",
        "docType": "ticket",
        "createdAt": "2026-06-01T00:00:00Z"
    },
    vector=[0.12, 0.98, -0.44, 0.05]
)

# Filtered near-vector search
response = collection.query.near_vector(
    near_vector=[0.10, 0.95, -0.40, 0.02],
    limit=5,
    filters=(
        wvc.query.Filter.by_property("tenantId").equal("acme-corp")
        & wvc.query.Filter.by_property("createdAt").greater_or_equal("2026-03-01T00:00:00Z")
    )
)

for obj in response.objects:
    print(obj.properties)

client.close()

Weaviate's multi-tenancy feature is worth calling out separately from ordinary metadata filtering: if tenant_id is your primary isolation boundary, enabling native multi-tenancy (each tenant gets its own physical shard) is usually a better fit than filtering a shared collection, because it guarantees isolation at the storage layer instead of relying on every query remembering to include the filter.

Hands-on: pgvector with a composite index

If you're already on Postgres, pgvector lets you keep vectors and relational metadata in the same table, which means you get standard B-tree and GIN indexes on metadata columns for free, combined with an HNSW or IVFFlat index on the vector column.

CREATE TABLE support_chunks (
    id BIGSERIAL PRIMARY KEY,
    tenant_id TEXT NOT NULL,
    doc_type TEXT NOT NULL,
    created_at TIMESTAMPTZ NOT NULL,
    content TEXT NOT NULL,
    embedding VECTOR(768) NOT NULL
);

-- Vector index for ANN search
CREATE INDEX ON support_chunks USING hnsw (embedding vector_cosine_ops);

-- Metadata indexes for filtering
CREATE INDEX idx_chunks_tenant ON support_chunks (tenant_id);
CREATE INDEX idx_chunks_tenant_created ON support_chunks (tenant_id, created_at);

SELECT id, content, embedding <=> '[0.10,0.95,-0.40,0.02]' AS distance
FROM support_chunks
WHERE tenant_id = 'acme-corp'
  AND created_at >= now() - interval '90 days'
ORDER BY embedding <=> '[0.10,0.95,-0.40,0.02]'
LIMIT 5;

Postgres's query planner decides whether to use the metadata B-tree index first (effectively pre-filtering) or scan the HNSW index and filter afterward, based on selectivity estimates from ANALYZE. Run EXPLAIN ANALYZE on your actual filtered queries after loading realistic data volumes, because the planner's choice can flip as your table grows, and a plan that was fast at 10,000 rows can silently degrade at 10 million.

Common mistakes to avoid

• Filtering after the fact in application code. If you're fetching 500 vectors and filtering down to 5 in Python, you're paying for the overfetch and still risking under-recall. Push the filter into the query.
• Forgetting to index a field you filter on. In Qdrant and similar databases, an unindexed filter field silently works but scans linearly, which only shows up as a problem once your collection is large enough to hurt.
• Treating `tenant_id` as optional. Every single query against a shared collection should include the tenant filter, with no code path that skips it. Enforce this at the query-builder or ORM layer, not by convention.
• Over-indexing metadata that never gets filtered. Indexing every field "just in case" bloats memory and slows writes. Index what you query, and keep the rest as plain payload.
• Ignoring filter selectivity when choosing a database. If your filters are usually highly selective (a tenant with only 200 vectors out of 50 million), post-filtering databases will give you poor recall unless you tune the overfetch factor aggressively. Test with your real data distribution before committing to a vendor.
• Re-embedding without invalidating old chunks. When a document changes, delete or mark the old chunks as stale using a metadata flag (is_current: false) rather than leaving orphaned vectors in the index; otherwise search results start surfacing outdated content next to the new version.

FAQ

What's the difference between metadata filtering and hybrid search? Metadata filtering restricts results to those matching exact structured conditions (equality, range, membership) and doesn't affect ranking by itself. Hybrid search blends vector similarity scores with keyword/BM25 scores to influence ranking. You often use both together: filter by metadata to scope the candidate pool, then rank with a hybrid score within that pool.

Does adding metadata indexes slow down inserts? Yes, somewhat. Every indexed field adds write overhead similar to a normal database index, since the index structure has to be updated on every insert or upsert. The cost is usually small relative to the cost of computing the embedding itself, so it rarely becomes the bottleneck, but it's still a reason to index only fields you actually filter on.

Can I filter on nested or array metadata fields? Most modern vector databases support this to some degree. Qdrant supports arrays natively and lets you match "any value in this array equals X." Weaviate supports array properties with similar semantics. Pinecone supports array metadata for membership checks ($in). Deeply nested objects are less universally supported, so flatten your schema where you can.

How do I handle permission changes without re-embedding everything? Store access control as metadata (access_level, allowed_roles) rather than baking it into the embedding, and update that metadata field directly when permissions change. Most vector databases support a metadata-only update (a "patch" or partial upsert) that doesn't require recomputing the vector, which is exactly what you want for permission changes that happen far more often than content changes.

Should I use one collection per tenant or one shared collection with a tenant_id filter? It depends on tenant count and isolation requirements. A handful of large tenants often justifies per-tenant collections or native multi-tenancy features (like Weaviate's), which gives hard isolation and lets you scale or delete a tenant independently. Hundreds or thousands of small tenants usually favor a shared collection with an indexed tenant_id field, since per-tenant collections at that scale add operational overhead (connection pooling, index warm-up, monitoring) that outweighs the isolation benefit.

What happens if I filter on a field with very low cardinality, like a boolean? Low-cardinality filters (few distinct values, each matching a large fraction of the data) are the case post-filtering handles well, since the candidate set from ANN search will usually already contain enough matches. The risk case is the opposite: high-cardinality, highly selective filters where the candidate set from ANN search often contains zero matches, which is where pre-filtering or integrated filtering earns its keep.