RAG

RAG Cost Optimization: Cutting Your Vector DB and LLM Bill

Pramod Dutta · May 13, 2026 · 14 min read

The first time a RAG pipeline goes from prototype to production, the bill shows up before the praise does. A demo that answered ten questions a day for the team suddenly answers ten thousand questions a day for customers, and the invoice from your vector database and your LLM provider stops looking like a rounding error. Most teams respond by reaching for a cheaper model and calling it a day, but that's the least effective lever you have. Rag cost optimization is really a systems problem — it touches chunking, embeddings, index design, retrieval logic, prompt construction, and generation, and the biggest wins usually come from the parts of the pipeline nobody thinks to profile. This article walks through where the money actually goes in a RAG system and what to change first, second, and third.

Where the money actually goes

Before optimizing anything, you need a cost breakdown, not a guess. A typical production RAG stack spends money in four places: embedding generation (one-time and ongoing, as documents change), vector database storage and query costs, retrieval infrastructure (reranking models, hybrid search), and LLM generation (the prompt you send plus the completion you get back).

Here's the part that surprises people: for most RAG applications, generation is the expensive line item, not retrieval. A single query might retrieve five to ten chunks, each 500-800 tokens, plus a system prompt, conversation history, and the user's question. That can easily push your input tokens to 4,000-6,000 per call before the model has generated a single word. If you're paying per-token for a frontier model and doing this at scale, your context window is your bill.

Vector database costs matter too, but differently — they tend to be dominated by storage (how many vectors, what dimensionality) and by the compute tier you provision for query throughput, not by per-query pricing the way LLM calls are. That means vector DB cost problems are solved by architecture decisions made once, while LLM cost problems are solved by decisions made on every single request.

Start any cost audit by logging, for a representative week of traffic: average input tokens per request, average output tokens per request, number of chunks retrieved per query, and your vector index size in millions of vectors. Everything below maps to one of those four numbers.

Chunking strategy is a cost lever, not just a quality lever

Most teams pick a chunk size early — 512 tokens, 1000 tokens, whatever a tutorial suggested — and never revisit it. But chunk size directly determines two costs: how many embedding calls you make when you ingest documents, and how many tokens you stuff into the LLM context at query time.

Smaller chunks mean more vectors to store and more embedding calls, but they let you retrieve more precisely, which sometimes means you need fewer chunks per query to get a complete answer. Larger chunks mean fewer vectors and cheaper ingestion, but you retrieve more tokens than you need per relevant chunk, and you pay for that redundancy on every single query, forever.

A practical approach: chunk by semantic unit (a section, a function, a paragraph cluster) rather than a fixed token count, and cap the size. This tends to reduce the "wasted tokens" problem where you retrieve a 1000-token chunk to get the two sentences that actually answer the question.

def chunk_by_section(document_text, max_tokens=400):
    """Chunk on semantic boundaries (headings/paragraphs),
    falling back to a hard cap so no chunk blows the budget."""
    sections = split_on_headings(document_text)
    chunks = []
    for section in sections:
        if count_tokens(section) <= max_tokens:
            chunks.append(section)
        else:
            # fall back to paragraph-level splitting only when needed
            chunks.extend(split_into_token_windows(section, max_tokens))
    return chunks

The other lever here is deduplication at ingestion time. Internal docs get copy-pasted, FAQs repeat themselves, and product documentation often has three versions of the same paragraph across different pages. Every duplicate is a vector you're storing and a chunk you might retrieve and pay to send to the LLM twice. A simple near-duplicate check (cosine similarity above 0.97 between candidate chunks before you embed) can shrink an index by 15-30% with zero quality loss — I've seen this on customer documentation corpora that had been "cleaned up" three separate times by three separate content teams, and each cleanup added more duplication than it removed.

Choosing embedding models with cost in mind

Embedding model choice affects cost in a way that compounds: dimensionality directly affects storage size in your vector database, and storage tier is usually billed per GB or per million vectors. A 1536-dimension embedding model costs roughly double the storage of a 768-dimension model for the same number of chunks — and unlike generation costs, this cost recurs every month whether you query the index or not.

The temptation is to always use the biggest, best embedding model. But retrieval quality and embedding dimensionality don't scale linearly — many teams see marginal retrieval quality improvement going from 768 to 1536 dimensions on domain-specific corpora (internal support docs, code, product manuals), while storage cost doubles. If you're running evaluation on retrieval quality (and you should be — see below), it's worth explicitly testing a smaller embedding model against your actual corpus and query set before assuming bigger is better.

Matryoshka-style embeddings (models trained so that you can truncate the vector to a shorter length and still get a usable, if slightly less precise, representation) are worth investigating if your vector DB bill is dominated by storage. Truncating a 1536-dim embedding to 512 dims can cut storage by two-thirds; whether that's viable depends on whether your retrieval quality holds up on your own eval set, not a benchmark someone else ran on a different corpus.

Re-embedding is another quiet cost sink. Teams often re-embed their entire corpus every time they update a chunking strategy, add a document, or swap embedding models — even when only 2% of documents actually changed. Track a content hash per chunk, and only re-embed chunks whose hash changed.

def get_chunks_needing_reembedding(chunks, existing_hashes):
    to_embed = []
    for chunk in chunks:
        chunk_hash = hashlib.sha256(chunk.text.encode()).hexdigest()
        if existing_hashes.get(chunk.id) != chunk_hash:
            to_embed.append(chunk)
    return to_embed

Vector database costs: index type and tiering

Vector database pricing usually breaks down into storage, compute (query throughput/pod size), and sometimes a per-operation cost for writes. The single biggest cost mistake here is over-provisioning compute for peak load that rarely happens, or under-using tiered storage for data that's rarely queried.

If your corpus has a long tail — old support tickets, deprecated documentation, last year's product specs — you likely don't need all of it sitting in your highest-performance index tier. Many vector databases (and self-hosted options like pgvector or Qdrant) let you split collections by access frequency: a "hot" index for current, frequently retrieved content on faster (and pricier) infrastructure, and a "cold" index for archival content that gets queried rarely, on cheaper storage with looser latency requirements.

Index type matters too. HNSW (Hierarchical Navigable Small World) graphs give excellent query latency but cost more memory than IVF (Inverted File) indexes, which trade a bit of recall for a much smaller memory footprint. If your application can tolerate slightly lower recall in exchange for materially lower memory costs — and many internal tools can — IVF or a hybrid IVF-PQ (product quantization) index is worth benchmarking.

# Example: pgvector index choice affects cost, not just speed
# HNSW: faster queries, higher memory/storage cost
"CREATE INDEX ON chunks USING hnsw (embedding vector_cosine_ops)"

# IVFFlat: lower memory footprint, requires periodic reindexing
# as data grows, good for cost-sensitive, latency-tolerant workloads
"CREATE INDEX ON chunks USING ivfflat (embedding vector_cosine_ops) WITH (lists = 100)"

Also check whether you're paying for a fully managed vector database when a self-hosted pgvector instance on infrastructure you already run would suffice. For many mid-size RAG deployments (low millions of vectors, moderate query volume), a well-tuned Postgres instance with pgvector is materially cheaper than a dedicated vector DB service, because you're not paying a per-vector SaaS markup on top of infrastructure you're already paying for anyway.

Retrieval efficiency: retrieve less, retrieve better

This is where a lot of RAG cost hides in plain sight. Teams tune top_k (how many chunks to retrieve) once, during development, based on vibes — "10 feels safe" — and never touch it again. But every chunk you retrieve becomes tokens in your LLM prompt. If you can cut top_k from 10 to 5 without hurting answer quality, you've roughly halved your retrieval-driven token cost on every single query.

The fix isn't guessing a smaller number — it's adding a reranking step. Retrieve a wider candidate set cheaply (say, top 20 by vector similarity), then use a lightweight reranker model to score and reorder those 20, and only pass the actual top 3-5 to the LLM. Rerankers are cheap and fast compared to LLM generation calls, and they let you keep recall high at retrieval time while keeping the expensive part — the LLM context — small.

def retrieve_and_rerank(query, vector_index, reranker, final_k=5, candidate_k=20):
    candidates = vector_index.search(query, top_k=candidate_k)
    scored = reranker.score(query, [c.text for c in candidates])
    ranked = sorted(zip(candidates, scored), key=lambda x: x[1], reverse=True)
    return [c for c, score in ranked[:final_k]]

Hybrid search (combining vector similarity with keyword/BM25 search) can also reduce cost indirectly: it often improves precision on queries with specific terms — product names, error codes, SKUs — meaning fewer irrelevant chunks get retrieved and passed to the LLM as noise. Noise in context isn't just a quality problem; it's tokens you paid for that made the answer worse.

Caching is the most underused retrieval optimization. If your application sees repeated or near-repeated queries — common in customer support, internal Q&A, or documentation search — cache the retrieved chunk set and, when appropriate, the final answer. A semantic cache (checking if the incoming query is close enough in embedding space to a previously answered query) can eliminate both the retrieval and the generation cost for a meaningful fraction of traffic, especially in support scenarios where the same handful of questions dominate volume.

Prompt construction: stop paying for repeated context

Once you've retrieved the right chunks, how you build the prompt still affects cost. A few practical habits:

Don't include full conversation history on every turn if you don't need to — summarize older turns instead of replaying them verbatim.
Strip retrieved chunk metadata (source URLs, timestamps, internal IDs) from the prompt unless the model actually needs to cite it; metadata tokens add up over thousands of queries for zero answer-quality benefit.
Use prompt caching if your LLM provider supports it. If your system prompt and retrieved context are stable across a session or across similar queries, caching the static portion of the prompt can cut the effective cost of those repeated tokens substantially — this is one of the highest-leverage, lowest-effort changes available, since it requires no pipeline redesign, only careful ordering of static content before dynamic content in the prompt.
Trim retrieved chunks that are marginally relevant — a reranker score threshold ("drop anything below 0.4") is a cheap filter that prevents low-value chunks from silently inflating every prompt.

def build_prompt(system_prompt, retrieved_chunks, query, min_score=0.4):
    relevant = [c for c in retrieved_chunks if c.score >= min_score]
    context = "\n\n".join(c.text for c in relevant)
    # static content first enables prompt caching on supporting providers
    return f"{system_prompt}\n\nContext:\n{context}\n\nQuestion: {query}"

Right-sizing the generation model

This is the lever everyone reaches for first, and it does matter — but only after the steps above, because a smaller model fed a bloated, redundant, poorly-retrieved context will just give you cheaper wrong answers faster.

Once your retrieval is tight, evaluate whether every query actually needs your most capable model. A large share of RAG traffic — factual lookup, simple summarization, "what does the documentation say about X" — doesn't need frontier-model reasoning. A tiered approach works well in practice: route straightforward retrieval-grounded questions to a smaller, cheaper model, and reserve the larger model for queries that require multi-step reasoning, synthesis across many chunks, or ambiguous questions that need judgment.

def route_query(query, retrieved_chunks):
    complexity_signals = [
        len(retrieved_chunks) > 6,
        requires_multi_hop_reasoning(query),
        is_ambiguous(query),
    ]
    if any(complexity_signals):
        return "large-model"
    return "small-model"

Building this router well is its own small project — it's effectively a lightweight classifier, and it's worth investing real evaluation time in it, because a bad router either sends easy queries to the expensive model (no savings) or sends hard queries to the cheap model (quality complaints). Don't skip evaluating the router itself just because it feels like plumbing rather than "real" AI work.

Also reconsider output length. If your system prompt doesn't constrain response length, models will often produce longer answers than necessary, and output tokens are typically priced higher than input tokens. A simple instruction to be concise, combined with a reasonable max_tokens cap, can meaningfully reduce output-side cost without hurting the answers users actually read.

Measuring what you cut

None of this matters if you can't tell whether a cost-cutting change degraded quality. Every change described above — smaller chunks, a cheaper embedding model, a smaller top_k, IVF instead of HNSW, a smaller generation model — trades some quality for some cost savings, and you need an evaluation set to know where that trade actually lands for your data and your users.

Build a small, representative eval set (50-200 real or realistic queries with known-good answers or known-good source chunks) before you start optimizing, not after. Run it against your current pipeline to get a baseline, then run it again after every change. Track retrieval recall (did the right chunk show up in the retrieved set) separately from answer quality (did the final answer correctly use that chunk), because a cost-cutting change can break either one independently — a smaller top_k breaks recall, while a smaller generation model can break answer quality even when the right chunk was retrieved.

def evaluate_pipeline(eval_set, pipeline):
    results = {"recall": [], "answer_correct": []}
    for item in eval_set:
        retrieved = pipeline.retrieve(item.query)
        recall_hit = item.expected_chunk_id in [c.id for c in retrieved]
        results["recall"].append(recall_hit)

        answer = pipeline.generate(item.query, retrieved)
        results["answer_correct"].append(
            judge_answer(answer, item.expected_answer)
        )
    return {
        "recall_rate": sum(results["recall"]) / len(eval_set),
        "accuracy_rate": sum(results["answer_correct"]) / len(eval_set),
    }

This is the discipline that separates real rag cost optimization from cost-cutting that quietly breaks the product. Treat every change as an experiment with a before/after eval score, not a one-way door.

Monitoring cost drift over time

RAG systems aren't static, and neither is their cost profile. Your corpus grows, so your vector index grows, so your storage bill grows even if query volume doesn't change. Your query mix shifts as users find new use cases, which changes your average retrieved-chunk count and your average generation model routing. A cost audit done once at launch goes stale within a quarter.

Instrument the pipeline to log, per request: tokens in, tokens out, number of chunks retrieved, which generation model was used, and latency. Aggregate this weekly. Watch specifically for "context creep" — the average number of input tokens per request quietly rising over time as chunking strategy drifts, as retrieval top_k gets bumped up by someone trying to fix a quality complaint without root-causing it, or as conversation history handling silently starts including more turns than intended.

The single most common failure mode I see in production RAG systems, cost-wise, is a quality bug getting "fixed" by throwing more context at the model — bump top_k, stop trimming history, add a longer system prompt — without ever confirming that more context was actually the right fix. Six months later nobody remembers why top_k is 15 instead of 5, and the bill reflects it every single day.

Bringing it together

Rag cost optimization isn't a single switch — it's a sequence of decisions, most of which compound with each other. Tight chunking reduces both storage and per-query tokens. A right-sized embedding model cuts storage without necessarily hurting retrieval quality on your actual corpus. Reranking lets you retrieve fewer, better chunks, which shrinks your LLM context on every call. Model routing sends easy queries to cheap models and hard queries to expensive ones. And an evaluation set makes sure none of these changes are secretly making your product worse while your invoice gets smaller.

The teams that get this right treat cost as a first-class metric next to latency and answer quality, not an afterthought they revisit only when finance asks a question. If you're building or maintaining a RAG system and want to go deeper on the fundamentals that make these tradeoffs intuitive rather than guesswork — chunking theory, retrieval architecture, evaluation design — our course Introduction to RAG walks through the full pipeline from first principles, including the cost and quality tradeoffs covered here, with hands-on exercises against real corpora rather than toy examples.