RAG Without a Vector Database: Simpler Alternatives That Work

Every RAG tutorial starts the same way: spin up a vector database, install an embedding model, and start chunking documents. But if you're building a support bot for 200 PDFs, a chatbot over your company wiki, or a Q&A tool for a few hundred pages of documentation, a vector database is often the most complicated part of your stack — and the least necessary one. We've shipped RAG systems at teachyou.ai using nothing more than SQLite, keyword search, and a bit of Python, and they perform just as well as the "proper" pipeline for the corpus sizes most teams actually deal with. This article walks through when you can skip the vector database entirely, what to use instead, and how to reason about the tradeoffs so you don't over-engineer a feature that a for-loop could handle.

Why people reach for a vector database by default

Vector databases became the default RAG recommendation because early tutorials assumed you'd be searching millions of documents with semantic queries where exact keyword matches wouldn't help. Pinecone, Weaviate, Qdrant, and Milvus are genuinely excellent at approximate nearest neighbor search over tens of millions of vectors with sub-100ms latency. That's a real engineering problem, and these tools solve it well.

The issue is that most RAG projects never get close to that scale. A knowledge base of 500 articles, a product manual with 3,000 paragraphs, a customer support archive of 10,000 tickets — none of these need approximate nearest neighbor search. They need "find the 5 most relevant chunks out of a few thousand," which a single machine can do in milliseconds using brute-force comparison or even plain text search.

Reaching for a vector database by default adds real cost:

• Operational overhead — another service to deploy, monitor, back up, and pay for.
• Latency from network hops — an external vector DB call adds round-trip time that an in-process search does not.
• A new failure mode — if the vector DB is down, your RAG pipeline is down, even though your actual data lives elsewhere.
• Vendor lock-in on index format — migrating 2 million embeddings between providers is more painful than it sounds.

None of this means vector databases are bad. It means they're a scaling solution, and you should only pay for a scaling solution once you've actually hit the scaling problem.

There's also a subtler cost that doesn't show up until later: cognitive overhead for whoever maintains the system after you. A teammate debugging a bad answer six months from now has to understand your chunking logic, your embedding model, your retrieval code, *and* the internals of whichever vector database you chose — its index type, its consistency guarantees, its quirks around filtering and metadata. Every extra moving part is something a future maintainer has to hold in their head before they can even start reasoning about why retrieval returned the wrong chunk. Keeping retrieval inside your own application code, even if it's "just" a SQL query or a NumPy array, means the entire system fits in one file that anyone on the team can read top to bottom.

The simplest alternative: brute-force cosine similarity in memory

If your corpus fits comfortably in RAM — which for text embeddings means anywhere from a few thousand to a few hundred thousand chunks — you can skip indexing altogether. Store your embeddings as a NumPy array, and compute cosine similarity against the query vector directly.

import numpy as np

class InMemoryRetriever:
    def __init__(self, embeddings: np.ndarray, chunks: list[str]):
        # Normalize once at load time so similarity = dot product
        norms = np.linalg.norm(embeddings, axis=1, keepdims=True)
        self.embeddings = embeddings / norms
        self.chunks = chunks

    def search(self, query_embedding: np.ndarray, top_k: int = 5):
        query_norm = query_embedding / np.linalg.norm(query_embedding)
        scores = self.embeddings @ query_norm  # cosine similarity
        top_indices = np.argsort(scores)[::-1][:top_k]
        return [(self.chunks[i], float(scores[i])) for i in top_indices]

For 50,000 chunks with 768-dimensional embeddings, this matrix multiplication takes single-digit milliseconds on a laptop CPU. There's no index to build, no approximate results to worry about — every search is exact. You load the embeddings once at startup (from a .npy file or a Parquet column), and the rest is arithmetic.

This is the pattern we teach first in Introduction to RAG, precisely because it forces you to understand what a vector database is actually doing under the hood before you outsource that work to a managed service. Once you've built brute-force search yourself, evaluating whether you need FAISS, HNSW, or a hosted vector DB becomes a much easier decision — you know exactly what capability you'd be buying.

It's worth being precise about where the numbers actually break down. A single 768-dimensional float32 vector is about 3KB. Fifty thousand of them is roughly 150MB — nothing for a modern machine. Even 500,000 chunks lands you around 1.5GB, still well within the memory of a small cloud instance. The matrix multiplication itself scales linearly with the number of chunks, so doubling your corpus roughly doubles your search time, not your infrastructure bill. Most teams overestimate how much data they actually have and underestimate how much a modern CPU can chew through in a single call to np.dot.

Using SQLite with FTS5 for keyword-based retrieval

A large fraction of real-world RAG queries are actually keyword-shaped, even when they're phrased as questions. "What's our refund policy for annual plans?" is going to match documents containing "refund," "policy," and "annual" far more reliably than it needs semantic understanding to find them. SQLite's FTS5 extension gives you full-text search with BM25 ranking, and it ships with Python's standard library.

import sqlite3

conn = sqlite3.connect("knowledge_base.db")
conn.execute("""
    CREATE VIRTUAL TABLE IF NOT EXISTS docs
    USING fts5(chunk_id, content, source_file)
""")

def index_chunk(chunk_id: str, content: str, source_file: str):
    conn.execute(
        "INSERT INTO docs (chunk_id, content, source_file) VALUES (?, ?, ?)",
        (chunk_id, content, source_file),
    )
    conn.commit()

def search(query: str, top_k: int = 5):
    cursor = conn.execute(
        """
        SELECT chunk_id, content, source_file, bm25(docs) AS rank
        FROM docs
        WHERE docs MATCH ?
        ORDER BY rank
        LIMIT ?
        """,
        (query, top_k),
    )
    return cursor.fetchall()

This is a single file on disk. No server process, no Docker container, no API keys for a hosted service. You can scp your entire retrieval system to another machine and it just works. For teams building internal tools, prototypes, or products where the whole app is a single Python process, this is often the correct permanent architecture, not just a stepping stone.

The tradeoff is real: FTS5 won't catch a query like "how do I get my money back" matching a document that only says "refund," because there's no token overlap. That's the exact gap embeddings are designed to fill. Which brings us to the approach most production systems actually converge on.

Hybrid search without a dedicated vector store

You don't have to choose between keyword search and embeddings — you can run both and combine the results, all without a dedicated vector database. Postgres is the easiest place to do this because it can hold your relational data, your full-text index, and your embeddings in the same database, queried with plain SQL.

Postgres's tsvector type handles keyword search, and the pgvector extension adds vector similarity as a column type — not a separate system, just an extension on a database you probably already run.

-- One-time setup
CREATE EXTENSION IF NOT EXISTS vector;

CREATE TABLE chunks (
    id SERIAL PRIMARY KEY,
    content TEXT NOT NULL,
    embedding VECTOR(768),
    search_vector TSVECTOR GENERATED ALWAYS AS (to_tsvector('english', content)) STORED
);

CREATE INDEX idx_chunks_fts ON chunks USING GIN (search_vector);

def hybrid_search(conn, query_text: str, query_embedding: list[float], top_k: int = 5):
    cursor = conn.execute(
        """
        SELECT content,
               (1 - (embedding <=> %s::vector)) AS semantic_score,
               ts_rank(search_vector, plainto_tsquery('english', %s)) AS keyword_score
        FROM chunks
        ORDER BY (0.6 * semantic_score + 0.4 * keyword_score) DESC
        LIMIT %s
        """,
        (query_embedding, query_text, top_k),
    )
    return cursor.fetchall()

Because Postgres is a general-purpose database, this scales into the millions of rows with a plain B-tree or IVFFlat index on the embedding column — no separate infrastructure to run. If your app is already backed by Postgres (which describes most SaaS products), adding retrieval is a migration and a query, not a new subsystem. This is the approach we recommend most often when students ask how to move a RAG prototype into an actual product without adopting a whole new database category.

When file-based indexes (FAISS, Annoy) are enough

Sometimes you want faster-than-brute-force search but still don't want a running server. FAISS (from Meta) and Annoy (from Spotify) are libraries, not services — you build an index in-process and save it to disk as a file, just like you would a serialized model.

import faiss
import numpy as np

dimension = 768
index = faiss.IndexFlatIP(dimension)  # exact inner product search
embeddings = np.load("chunk_embeddings.npy").astype("float32")
faiss.normalize_L2(embeddings)
index.add(embeddings)

faiss.write_index(index, "chunks.faiss")

# Later, at query time
loaded_index = faiss.read_index("chunks.faiss")
query_vec = np.array([query_embedding], dtype="float32")
faiss.normalize_L2(query_vec)
distances, indices = loaded_index.search(query_vec, k=5)

IndexFlatIP is still exact search — the speed comes from FAISS's optimized C++ implementation of the same matrix math you'd write yourself in NumPy, plus SIMD acceleration. For a few hundred thousand vectors, this runs in single-digit milliseconds without needing an approximate index at all. If you eventually need approximate search for tens of millions of vectors, FAISS also gives you IndexIVFFlat or IndexHNSWFlat — but critically, you get there by changing one line, not by adopting new infrastructure.

The point worth internalizing: FAISS is a library you import, not a database you operate. There's no server to keep alive, no network call, no separate deployment. Your retrieval logic lives in the same process as the rest of your application, which also makes debugging dramatically simpler — you can inspect the exact same objects your search function used, in the same Python REPL.

Re-ranking as a substitute for a fancier index

A pattern that consistently punches above its complexity: do a cheap, coarse retrieval step first (keyword search, or even just filtering by metadata), then re-rank the top 50-100 candidates with a cross-encoder model that scores query-document pairs directly.

from sentence_transformers import CrossEncoder

reranker = CrossEncoder("cross-encoder/ms-marco-MiniLM-L-6-v2")

def retrieve_and_rerank(query: str, candidates: list[str], top_k: int = 5):
    pairs = [(query, doc) for doc in candidates]
    scores = reranker.predict(pairs)
    ranked = sorted(zip(candidates, scores), key=lambda x: x[1], reverse=True)
    return ranked[:top_k]

Cross-encoders are more accurate than embedding similarity because they look at the query and document together, rather than comparing two independently-computed vectors. They're too slow to run over your entire corpus (you're doing a full forward pass per document, not a single dot product), but running one over the 50-100 candidates a cheap first pass already narrowed down is fast and often outperforms a pure vector-database pipeline on relevance.

This is a genuinely underused technique. Teams jump straight to "we need better embeddings" or "we need a bigger vector index" when the actual fix is a re-ranking stage bolted onto whatever coarse retrieval they already have — including plain keyword search.

Metadata filtering can replace half your "semantic" queries

A lot of what looks like a semantic search problem is actually a structured filtering problem wearing a disguise. "Show me the pricing page updates from last quarter" doesn't need an embedding model — it needs a WHERE category = 'pricing' AND updated_at > '2026-04-01' clause. If your documents have dates, categories, authors, or tags, push as much of the query as possible into structured filters before you ever touch similarity scoring.

def filtered_search(conn, category: str, after_date: str, query_embedding=None):
    base_query = """
        SELECT content, embedding FROM chunks
        WHERE category = %s AND updated_at > %s
    """
    candidates = conn.execute(base_query, (category, after_date)).fetchall()

    if query_embedding is None or len(candidates) <= 10:
        return candidates[:10]

    # Only rank by similarity within the already-filtered set
    scored = [
        (content, cosine_sim(emb, query_embedding))
        for content, emb in candidates
    ]
    return sorted(scored, key=lambda x: x[1], reverse=True)[:10]

Once you filter first, the set of candidates you need to semantically rank shrinks dramatically — often from thousands down to dozens — which is exactly the regime where brute-force NumPy comparison is trivially fast. This is also usually a *better* user experience than pure semantic search, because it respects the explicit structure the user gave you instead of hoping the embedding model infers it.

Chunking and preprocessing matter more than your retrieval engine

Regardless of which retrieval approach you pick, the quality of your RAG system is dominated by how you chunk and clean your documents, not by whether you used Qdrant or SQLite. A vector database searching over badly-chunked text (arbitrary 500-character splits that cut sentences in half, tables flattened into unreadable strings, headers stripped of their section context) will perform worse than brute-force search over well-chunked text.

A few things worth getting right before you worry about the search backend:

• Chunk on semantic boundaries — paragraphs, sections, or markdown headers, not fixed character counts.
• Keep chunk sizes consistent with what you'll actually feed to the LLM — 200-400 tokens is a reasonable default for most models and use cases.
• Preserve context in each chunk — prepend the document title or section header so a chunk still makes sense in isolation.
• Deduplicate near-identical chunks — repeated boilerplate (headers, footers, legal disclaimers) pollutes your candidate set and wastes retrieval slots.
• Store the source location alongside each chunk — you'll want to cite it in the final answer, and you'll want it for debugging when retrieval goes wrong.

We spend an entire module on this in Introduction to RAG because it's the part every tutorial skips past to get to the exciting vector database demo, yet it's the part that determines whether your system feels reliable or flaky in production.

When you actually do need a vector database

None of this is an argument that vector databases are unnecessary in general — it's an argument that they're unnecessary by default. There are real signals that tell you it's time to adopt one:

1. Your corpus has grown past what fits comfortably in a single machine's RAM (roughly millions of chunks, depending on your embedding dimension and available memory).
2. You need sub-50ms retrieval latency at high query-per-second load, and brute-force comparison is now the bottleneck.
3. You need built-in horizontal scaling because a single Postgres instance or in-memory index can't keep up with write volume from continuous ingestion.
4. You need features like real-time upserts at scale, multi-tenant namespace isolation, or hybrid search with reranking built directly into the retrieval layer, and building those yourself would take longer than adopting a managed product.
5. Your team has the operational maturity to run and monitor another stateful service, and the corpus size actually justifies it.

If none of those apply to you today, they might apply in a year — and the good news is that the migration path is usually straightforward. Your chunks, your embeddings, and your chunking logic don't change; only the storage and search layer does. Starting simple doesn't box you in, it just means you're not paying for capacity you don't need yet.

Closing thoughts

The best RAG systems we've seen in production aren't the ones with the most sophisticated vector database setup — they're the ones where the team deeply understood their retrieval problem and picked the simplest tool that solved it. For a few thousand documents, that's often brute-force cosine similarity in a NumPy array. For a system already built on Postgres, that's pgvector and a hybrid SQL query. For a static knowledge base, that might just be SQLite's FTS5 and nothing more exotic than that.

Start with the simplest retrieval mechanism that could plausibly work, measure whether it's actually giving you relevant chunks, and only add infrastructure when you've identified a specific limitation that infrastructure solves. If you want to build this understanding systematically — from chunking strategy through retrieval architecture to evaluation — that's exactly the path we walk through step by step in Introduction to RAG.