A Practical Guide to Chroma

If you are looking for a chroma guide that actually gets you writing and querying vectors instead of reading marketing copy, this is it. Chroma is an open source embedding database built for retrieval augmented generation (RAG) pipelines: you feed it text, it stores the vector representation alongside your metadata, and you query it with a new piece of text to get back the most semantically similar chunks. It ships as a Python (and JavaScript) library first, with an optional client-server mode for production, which makes it one of the fastest vector databases to get running on a laptop.

This guide walks through installing Chroma, creating collections, choosing an embedding function, running similarity search with filters, persisting data to disk, and moving to a client-server deployment. Every section includes code you can run as-is.

Why Chroma instead of another vector database

Most teams evaluating vector databases end up comparing Chroma against Pinecone, Weaviate, Qdrant, and pgvector. The tradeoffs come down to three things: operational overhead, embedding integration, and scale.

Chroma's biggest advantage is zero-friction local development. You pip install chromadb and you have a working vector store in one line, no Docker container, no API key, no network call. That matters when you are prototyping a RAG pipeline and want fast iteration on chunking strategy or prompt design before you commit to infrastructure.

Pinecone and Weaviate Cloud are managed services, which means you skip ops work but you also need an account and a network round trip for every query, even during local development. Qdrant and pgvector are strong production choices, but pgvector piggybacks on an existing Postgres instance (great if you already run Postgres, awkward if you don't), and Qdrant leans more toward a dedicated ops team managing a Rust service.

Chroma sits in the middle: it can run embedded (in-process, backed by SQLite and a Parquet-like storage layer) for development, and it can run as a standalone server for production, using the same client API in both cases. That means your prototype code and your production code look nearly identical, which shortens the path from notebook to deployed service.

Where Chroma is weaker: very large scale (hundreds of millions of vectors with strict latency SLAs) is better served by Qdrant, Milvus, or a managed service with dedicated sharding. Chroma is improving here, but if you already know you're building for that scale, evaluate accordingly.

Installing Chroma and creating your first collection

Start with a clean virtual environment and install the client.

python -m venv .venv
source .venv/bin/activate
pip install chromadb

Chroma organizes vectors into collections, which are roughly analogous to a table in a relational database. Each collection has a name, an embedding function, and holds documents, their vectors, metadata, and IDs.

import chromadb

# In-memory client: nothing touches disk, useful for quick experiments
client = chromadb.Client()

collection = client.create_collection(name="docs_demo")

collection.add(
    documents=[
        "Chroma is an open source embedding database for RAG.",
        "Postgres with pgvector reuses your existing database.",
        "Qdrant is written in Rust and built for large scale vector search.",
    ],
    metadatas=[
        {"source": "chroma_intro"},
        {"source": "pgvector_intro"},
        {"source": "qdrant_intro"},
    ],
    ids=["doc1", "doc2", "doc3"],
)

results = collection.query(
    query_texts=["Which vector database is written in Rust?"],
    n_results=2,
)

for doc, dist in zip(results["documents"][0], results["distances"][0]):
    print(f"{dist:.4f}  {doc}")

Notice you never called an embedding model directly. By default, Chroma uses a lightweight local sentence-transformers model to turn your text into vectors automatically, both on add and on query. That default model is fine for demos but not for production accuracy, which brings us to embedding functions.

Choosing and configuring an embedding function

An embedding function is the piece that converts text into a vector. Chroma lets you swap this out per collection, so you can use OpenAI's embedding models, Cohere, a local sentence-transformers model, or any custom function that returns a list of floats per input string.

Here is how to wire up a hosted embedding model instead of the default:

import chromadb
from chromadb.utils import embedding_functions

openai_ef = embedding_functions.OpenAIEmbeddingFunction(
    api_key="YOUR_API_KEY",
    model_name="text-embedding-3-small",
)

client = chromadb.Client()
collection = client.create_collection(
    name="support_articles",
    embedding_function=openai_ef,
)

If you want to keep everything local and avoid API calls entirely, use sentence-transformers with a stronger model than the default:

sentence_transformer_ef = embedding_functions.SentenceTransformerEmbeddingFunction(
    model_name="all-mpnet-base-v2"
)

collection = client.create_collection(
    name="local_articles",
    embedding_function=sentence_transformer_ef,
)

Two rules matter here. First, pick one embedding function per collection and stick with it: mixing embedding models in the same collection produces vectors in incompatible spaces, and your similarity search results will be nonsense. Second, if you ever change embedding models, you must re-embed and re-add every document. There is no in-place migration for embeddings, because the whole point of an embedding is that it is tied to the model that produced it.

You can also bring your own precomputed vectors if you are already running embeddings through a pipeline elsewhere:

collection.add(
    embeddings=[[0.01, 0.22, -0.15, 0.4]],  # your precomputed vector
    documents=["Precomputed embedding example."],
    ids=["doc4"],
)

Filtering and metadata queries

Semantic search alone is often not enough. You usually need to combine similarity with structured filters: only search within a date range, a specific customer's documents, or a document type. Chroma supports this through the where clause on metadata and where_document for substring filters on the document text.

collection.add(
    documents=[
        "Refund policy allows returns within 30 days.",
        "Refund policy allows returns within 30 days for EU customers.",
        "Shipping typically takes 3 to 5 business days.",
    ],
    metadatas=[
        {"category": "refunds", "region": "us"},
        {"category": "refunds", "region": "eu"},
        {"category": "shipping", "region": "us"},
    ],
    ids=["r1", "r2", "s1"],
)

results = collection.query(
    query_texts=["how long do refunds take"],
    n_results=3,
    where={"category": "refunds"},
    where_document={"$contains": "30 days"},
)

For more complex filters, Chroma supports comparison and logical operators:

results = collection.query(
    query_texts=["refund policy"],
    where={
        "$and": [
            {"category": {"$eq": "refunds"}},
            {"region": {"$in": ["us", "eu"]}},
        ]
    },
    n_results=5,
)

This is the pattern most RAG systems need in practice: narrow the candidate set with metadata first, then rank by semantic similarity within that narrowed set. It keeps irrelevant documents out of your context window even when they are semantically close to the query.

Persisting data and running Chroma as a server

The in-memory client above loses everything when the process exits. For anything beyond a scratch notebook, use the persistent client, which writes to a local directory.

import chromadb

client = chromadb.PersistentClient(path="./chroma_data")
collection = client.get_or_create_collection(name="docs_demo")

Everything else in your code stays the same, add, query, get, update, delete, they all work identically. The get_or_create_collection call is worth using over create_collection once your code runs more than once, since create_collection raises an error if the collection already exists.

For a real application with multiple services or multiple machines talking to the same vector store, run Chroma as a standalone server. Start it with the CLI:

chroma run --path ./chroma_data --port 8000

Then connect from any client using HttpClient instead of PersistentClient:

import chromadb

client = chromadb.HttpClient(host="localhost", port=8000)
collection = client.get_or_create_collection(name="docs_demo")

You can also run this server in Docker for a reproducible deployment:

docker run -p 8000:8000 -v ./chroma_data:/chroma/chroma chromadb/chroma

The client API does not change between embedded and server mode, which is the main reason teams start with Chroma: your prototype script and your production service call the exact same methods.

Building a small RAG pipeline end to end

Here is a compact example that chunks a document, embeds it, stores it in Chroma, and retrieves context for a question. This is the shape of nearly every RAG pipeline you will build.

import chromadb
from chromadb.utils import embedding_functions

def chunk_text(text, chunk_size=300, overlap=50):
    chunks = []
    start = 0
    while start < len(text):
        end = start + chunk_size
        chunks.append(text[start:end])
        start = end - overlap
    return chunks

raw_text = """
Chroma stores embeddings in collections. Each document you add gets a vector
representation computed by the collection's embedding function. Queries are
embedded the same way and compared against stored vectors using a distance
metric, typically cosine or L2, to find the closest matches. Metadata lets you
filter results before or alongside the similarity ranking, which keeps
retrieval accurate even in large collections.
"""

chunks = chunk_text(raw_text)

client = chromadb.PersistentClient(path="./rag_store")
ef = embedding_functions.SentenceTransformerEmbeddingFunction(model_name="all-mpnet-base-v2")
collection = client.get_or_create_collection(name="rag_pipeline", embedding_function=ef)

collection.add(
    documents=chunks,
    ids=[f"chunk_{i}" for i in range(len(chunks))],
    metadatas=[{"chunk_index": i} for i in range(len(chunks))],
)

def retrieve_context(question, k=3):
    results = collection.query(query_texts=[question], n_results=k)
    return "\n---\n".join(results["documents"][0])

question = "How does Chroma compare a query to stored vectors?"
context = retrieve_context(question)
print(context)

# Feed `context` and `question` into your LLM prompt from here

From here, the missing piece is passing context and question into your language model of choice as a system or user prompt, which is outside Chroma's scope but is the natural next step for any RAG application.

Choosing a distance metric and tuning collection settings

Chroma defaults to L2 (Euclidean) distance, but for most text embedding models, cosine similarity is the better choice because it ignores vector magnitude and focuses on direction, which is what semantic similarity models are trained to produce. Set it explicitly when you create a collection:

collection = client.create_collection(
    name="tuned_collection",
    metadata={"hnsw:space": "cosine"},
)

Other useful HNSW index settings you can pass through the metadata dict include hnsw:construction_ef (higher values improve index quality at the cost of build time) and hnsw:search_ef (higher values improve recall at query time at the cost of latency). Leave these at defaults until you have measured a real recall or latency problem, tuning blind rarely helps.

Updating, deleting, and inspecting data

Documents change, so you need update and delete paths, not just add and query.

# Update an existing document and its metadata
collection.update(
    ids=["r1"],
    documents=["Refund policy now allows returns within 45 days."],
    metadatas=[{"category": "refunds", "region": "us"}],
)

# Delete by ID
collection.delete(ids=["s1"])

# Delete by filter instead of ID
collection.delete(where={"category": "shipping"})

# Inspect what is actually stored
record = collection.get(ids=["r1"], include=["documents", "metadatas", "embeddings"])
print(record)

# Count total documents in a collection
print(collection.count())

collection.get is the tool you reach for when debugging a retrieval pipeline that returns unexpected results. Pull the raw stored document and metadata for a suspicious ID and check whether the chunking or metadata tagging step upstream is the actual bug, rather than assuming the vector search itself is wrong.

Common mistakes to avoid

A few issues come up repeatedly when teams adopt Chroma for the first time.

• Mixing embedding functions across adds. If you switch models mid-project without re-embedding old documents, your query results silently degrade because old and new vectors live in different geometric spaces.
• Chunking too large or too small. Chunks over roughly 500 tokens dilute the embedding's specificity, chunks under 100 tokens lose surrounding context. Start around 300 tokens with an overlap of 15 to 20 percent and adjust based on retrieval quality.
• Ignoring metadata filtering. Pure semantic search over a large, mixed-topic collection returns confidently wrong results. Tag documents with category, source, date, and tenant metadata from day one, even if you don't filter on it immediately.
• Treating the in-memory client as persistent. chromadb.Client() without a path loses everything on process exit. Use PersistentClient or HttpClient outside of throwaway scripts.
• Not measuring distance score meaning. A cosine distance of 0.3 is not inherently "good" or "bad", it depends on your embedding model and data. Look at actual query results for known good and bad matches to calibrate a reasonable threshold before hardcoding a cutoff in production logic.

FAQ

Is Chroma free to use? Yes, Chroma is open source under the Apache 2.0 license. You can run it embedded in a Python process or as a self-hosted server at no licensing cost. A managed Chroma Cloud offering also exists for teams that don't want to operate the server themselves.

Does Chroma support languages other than Python? Chroma ships an official JavaScript/TypeScript client in addition to Python, and the HTTP server mode means any language with an HTTP client can talk to it directly, since the client libraries are thin wrappers around REST calls.

How does Chroma compare to pgvector for a small project? If you already run Postgres, pgvector avoids adding a new service to your stack. If you don't already run Postgres, or you want a purpose-built API for RAG (chunking-friendly metadata filters, built-in embedding function abstraction), Chroma gets you running faster with less boilerplate.

Can I use Chroma in production, or is it only for prototyping? Chroma is used in production by teams at moderate scale, particularly through the client-server deployment mode with persistent storage. For very high query volume or hundreds of millions of vectors, evaluate Qdrant, Milvus, or a managed service alongside Chroma before committing.

What embedding model should I use with Chroma? For quality, a hosted model like OpenAI's text-embedding-3-small or text-embedding-3-large gives strong general-purpose results with minimal setup. For a fully local, no-API-key pipeline, all-mpnet-base-v2 from sentence-transformers is a solid default. Benchmark both against your actual queries before locking one in, since embedding quality varies a lot by domain.

How do I back up a persistent Chroma database? Since PersistentClient writes to a local directory, back it up the same way you'd back up any directory: copy it, snapshot the volume, or sync it to object storage on a schedule. There is no separate export command required beyond copying the storage path.

Can multiple processes write to the same Chroma collection at once? With PersistentClient, concurrent writes from multiple processes on the same machine are not safe. Run Chroma in server mode (HttpClient against a chroma run server) whenever more than one process needs to read or write the same collection concurrently.