A Practical Guide to Milvus

This Milvus guide walks through everything you need to run a real vector search workload: standing up a local instance, designing a collection schema, choosing an index, and wiring it into a retrieval-augmented generation (RAG) pipeline. Milvus is a purpose-built vector database for storing and querying high-dimensional embeddings at scale, and by the end of this guide you will have a working collection, a populated index, and queries returning ranked results.

What Milvus Is and When to Use It

Milvus is an open source vector database designed to store embedding vectors (the numeric representations produced by models like text-embedding-3-large or open source sentence transformers) and search them by similarity instead of exact match. Traditional databases answer "give me rows where id = 5." Milvus answers "give me the 10 vectors closest to this one," using distance metrics like cosine similarity, inner product, or Euclidean distance.

You reach for Milvus when:

• You are building semantic search over documents, images, or audio embeddings.
• You need RAG for a chatbot or agent and want sub-100ms retrieval over millions of chunks.
• You are doing recommendation systems, anomaly detection, or deduplication based on vector similarity.
• You have outgrown a simple in-memory index (like a flat NumPy array or FAISS-only setup) and need persistence, filtering, and horizontal scale.

Milvus differs from lighter-weight options like Chroma or LanceDB in that it is built for production scale: it separates storage and compute, supports distributed deployments, and offers multiple index types tuned for different tradeoffs between recall, latency, and memory. If you only have a few thousand vectors, a simpler tool might be enough. Once you are past a few million vectors, or need multi-tenant collections with metadata filtering, Milvus starts to pay off.

Installing Milvus Locally

The fastest way to try Milvus is Milvus Lite, an embedded version that runs in-process with no server to manage. It's ideal for prototyping and small datasets before moving to a full deployment.

pip install pymilvus

Milvus Lite activates automatically when you connect using a local file path instead of a server URI:

from pymilvus import MilvusClient

client = MilvusClient("milvus_demo.db")
print("Connected to Milvus Lite")

For anything beyond local experimentation, run the full server via Docker Compose. Milvus provides a standalone Docker setup that bundles etcd (metadata), MinIO (object storage), and the Milvus server itself.

curl -sfL https://raw.githubusercontent.com/milvus-io/milvus/master/scripts/standalone_embed.sh -o standalone_embed.sh
bash standalone_embed.sh start

This starts Milvus on localhost:19530. Verify it's up:

docker ps

You should see containers for milvus-standalone, milvus-etcd, and milvus-minio. Once you outgrow a single node, Milvus also runs in distributed mode on Kubernetes, and Zilliz Cloud offers a managed hosted version if you'd rather not operate the cluster yourself.

Connecting and Creating a Collection

A collection in Milvus is roughly equivalent to a table: it has a defined schema with typed fields, one of which holds the vector data. Connect to a running server like this:

from pymilvus import MilvusClient

client = MilvusClient(uri="http://localhost:19530")

Milvus's MilvusClient offers a quick-start path that infers a reasonable schema, but for production work you want explicit control. Here's a schema for a document chunk store with metadata fields you'll filter on later:

from pymilvus import MilvusClient, DataType

client = MilvusClient(uri="http://localhost:19530")

schema = client.create_schema(auto_id=True, enable_dynamic_field=False)

schema.add_field(field_name="id", datatype=DataType.INT64, is_primary=True)
schema.add_field(field_name="embedding", datatype=DataType.FLOAT_VECTOR, dim=1536)
schema.add_field(field_name="text", datatype=DataType.VARCHAR, max_length=4000)
schema.add_field(field_name="source", datatype=DataType.VARCHAR, max_length=256)
schema.add_field(field_name="chunk_index", datatype=DataType.INT64)

client.create_collection(collection_name="docs", schema=schema)

The dim value must match your embedding model's output size. text-embedding-3-large produces 3072-dimensional vectors, text-embedding-3-small produces 1536, and common open source models like all-MiniLM-L6-v2 produce 384. Get this wrong and every insert will fail with a dimension mismatch error.

Choosing and Building an Index

A raw collection with no index does brute-force comparison, which is fine for a few thousand vectors but doesn't scale. Milvus supports several index types, and picking the right one is the single biggest lever for balancing speed, recall, and memory.

• FLAT: exact search, no approximation. Use it for small collections (under ~100k vectors) or when you need guaranteed exact recall.
• IVF_FLAT / IVF_SQ8: clusters vectors into buckets (inverted file index) and searches only the closest buckets. Good general-purpose choice, IVF_SQ8 trades a little recall for lower memory via scalar quantization.
• HNSW: a graph-based index that gives excellent recall and low latency for most workloads. This is the default recommendation for most RAG and semantic search use cases in 2026.
• DISKANN: designed for datasets too large to fit in memory, trading some latency for the ability to scale into the hundreds of millions of vectors on disk.

For a typical RAG collection, HNSW is the pragmatic default:

index_params = client.prepare_index_params()

index_params.add_index(
    field_name="embedding",
    index_type="HNSW",
    metric_type="COSINE",
    params={"M": 16, "efConstruction": 200}
)

client.create_index(collection_name="docs", index_params=index_params)

M controls how many neighbor connections each node keeps (higher means better recall, more memory). efConstruction controls the search effort during index build (higher means a better-quality graph, slower build). For most document retrieval workloads, M=16 and efConstruction=200 is a solid starting point that you can tune later based on measured recall.

COSINE is the right metric for most text embedding models, since they're typically normalized and cosine similarity captures semantic closeness well. If your embeddings are unnormalized and magnitude carries meaning, IP (inner product) or L2 (Euclidean) may fit better; check your embedding model's documentation.

Inserting Data

With schema and index in place, load the collection and insert vectors. Here's a minimal end-to-end example using sentence-transformers to generate embeddings for a handful of text chunks:

from sentence_transformers import SentenceTransformer

model = SentenceTransformer("all-MiniLM-L6-v2")

chunks = [
    "Milvus stores vector embeddings for similarity search.",
    "HNSW indexes trade memory for fast approximate nearest neighbor search.",
    "RAG pipelines retrieve relevant chunks before generating an answer.",
]

embeddings = model.encode(chunks).tolist()

data = [
    {"embedding": emb, "text": chunk, "source": "guide.md", "chunk_index": i}
    for i, (emb, chunk) in enumerate(zip(embeddings, chunks))
]

client.insert(collection_name="docs", data=data)

Note the dimension mismatch trap mentioned earlier: all-MiniLM-L6-v2 outputs 384-dimensional vectors, so if you followed the earlier schema example with dim=1536, this insert will fail. Match your schema's dim to whatever embedding model you actually use before creating the collection, since changing dim later requires dropping and recreating the collection.

For production ingestion, batch inserts in chunks of a few thousand rather than inserting one row at a time. This dramatically reduces round-trip overhead:

batch_size = 1000
for i in range(0, len(data), batch_size):
    client.insert(collection_name="docs", data=data[i:i + batch_size])

Querying: Similarity Search and Filtering

Once data is inserted, load the collection into memory and run a search:

client.load_collection(collection_name="docs")

query_text = "How does approximate nearest neighbor search work?"
query_embedding = model.encode([query_text]).tolist()

results = client.search(
    collection_name="docs",
    data=query_embedding,
    limit=5,
    output_fields=["text", "source", "chunk_index"],
    search_params={"metric_type": "COSINE", "params": {"ef": 64}}
)

for hit in results[0]:
    print(hit["distance"], hit["entity"]["text"])

ef at query time controls how many candidates HNSW examines before returning results. Higher ef improves recall at the cost of latency, and unlike efConstruction, you can tune it per query without rebuilding the index.

Milvus also supports scalar filtering alongside vector search, which is essential for real applications where you need to restrict results to a specific tenant, document, or date range:

results = client.search(
    collection_name="docs",
    data=query_embedding,
    limit=5,
    filter='source == "guide.md" and chunk_index < 10',
    output_fields=["text", "source"]
)

This filter expression syntax reads like Python and supports comparisons, and/or, in, and string matching. Filtering happens as part of the same search call rather than as a separate post-processing step, which keeps latency low even with selective filters.

Hybrid Search: Combining Dense and Sparse Vectors

Pure dense vector search sometimes misses exact keyword matches, like product codes, names, or acronyms that an embedding model wasn't trained to distinguish precisely. Milvus supports hybrid search, combining dense vector similarity with sparse (BM25-style) keyword matching in a single query, then fusing the results.

from pymilvus import Function, FunctionType

schema = client.create_schema(auto_id=True)
schema.add_field(field_name="id", datatype=DataType.INT64, is_primary=True)
schema.add_field(field_name="text", datatype=DataType.VARCHAR, max_length=4000, enable_analyzer=True)
schema.add_field(field_name="dense_vector", datatype=DataType.FLOAT_VECTOR, dim=384)
schema.add_field(field_name="sparse_vector", datatype=DataType.SPARSE_FLOAT_VECTOR)

bm25_function = Function(
    name="text_bm25",
    input_field_names=["text"],
    output_field_names=["sparse_vector"],
    function_type=FunctionType.BM25
)
schema.add_function(bm25_function)

Milvus generates the sparse BM25 vector automatically from the text field when enable_analyzer=True is set, so you don't need to run your own BM25 scoring separately. At query time, you provide both a dense query vector and the raw query text, and Milvus fuses the ranked lists using a reranker like reciprocal rank fusion (RRF). This is worth setting up whenever your RAG pipeline handles queries containing exact identifiers, jargon, or names that embeddings alone tend to blur.

Wiring Milvus into a RAG Pipeline

A typical RAG flow looks like: chunk your documents, embed each chunk, store in Milvus, then at query time embed the user's question, retrieve top-k chunks, and pass them to an LLM as context. Here's the retrieval half assembled into a reusable function:

def retrieve_context(question, top_k=5):
    query_vec = model.encode([question]).tolist()
    results = client.search(
        collection_name="docs",
        data=query_vec,
        limit=top_k,
        output_fields=["text", "source"]
    )
    return [hit["entity"]["text"] for hit in results[0]]

def build_prompt(question):
    context_chunks = retrieve_context(question)
    context = "\n\n".join(context_chunks)
    return f"""Answer the question using only the context below.

Context:
{context}

Question: {question}
Answer:"""

Feed the resulting prompt to whatever LLM client you're using (Claude, GPT, or a local model). The key engineering decisions live in chunking strategy (how you split documents), top_k (how many chunks to retrieve), and whether you rerank results with a cross-encoder before passing them to the LLM. Milvus handles the retrieval mechanics; your chunking and prompt construction determine answer quality.

Performance Tuning and Common Pitfalls

A few practical lessons that save debugging time:

1. Always call `load_collection` before searching. A collection that isn't loaded into memory returns errors or empty results, and it's the most common "why is my search failing" mistake for people new to Milvus.
2. Match `dim` exactly to your embedding model. Changing embedding models later means recreating the collection and re-embedding everything, since dim is fixed at schema creation time.
3. Don't over-provision `M` and `efConstruction`. Bigger values improve recall but increase memory and index build time substantially. Start with defaults, measure recall on a labeled sample, then tune.
4. Batch your inserts. Row-by-row inserts are dramatically slower than batched inserts of 500-5000 rows at a time.
5. Use partitions for multi-tenant data. Instead of filtering every query by a tenant_id field, consider Milvus partitions, which physically segment data and let you search only the relevant partition, cutting latency for large multi-tenant collections.
6. Release collections you're not actively querying. client.release_collection() frees memory when a collection is idle, useful if you're juggling many collections on limited hardware.

FAQ

What is Milvus used for? Milvus is used to store and search high-dimensional vector embeddings for use cases like semantic search, RAG retrieval, recommendation systems, image similarity search, and anomaly detection, anywhere you need "find things similar to this" rather than exact-match queries.

Is Milvus free to use? Yes, Milvus is open source and free to self-host, either via Milvus Lite for local prototyping or a full Docker/Kubernetes deployment for production. Zilliz Cloud offers a managed hosted version if you prefer not to operate the infrastructure yourself, which is a paid service.

How is Milvus different from Pinecone or Chroma? Pinecone is a fully managed, closed source vector database with no self-hosting option. Chroma is lightweight and easy to start with but is generally aimed at smaller-scale or prototyping workloads. Milvus sits in between: open source like Chroma, but architected for distributed, production-scale deployments like Pinecone, with more index type options and hybrid search built in.

Which index type should I use by default? HNSW is the sensible default for most semantic search and RAG workloads in 2026, offering strong recall and low query latency. Use FLAT for small collections where exact results matter more than speed, and DISKANN when your dataset is too large to fit in memory.

Can Milvus do keyword search, not just vector similarity? Yes. Milvus supports hybrid search that combines dense vector similarity with sparse BM25-style keyword scoring in a single query, useful when queries contain exact identifiers, product codes, or names that pure embedding similarity handles poorly.

Do I need Docker to try Milvus? No. Milvus Lite runs embedded in your Python process with pip install pymilvus and a local file path, no server or Docker required. It's meant for prototyping; move to the full Docker or Kubernetes deployment once you need persistence at scale, multi-client access, or distributed compute.

What embedding dimension should my collection use? It must exactly match your embedding model's output dimension: 1536 for text-embedding-3-small, 3072 for text-embedding-3-large, 384 for all-MiniLM-L6-v2, and so on. Check your model's documentation before creating the schema, since changing it later requires recreating the collection.