Agent Memory with a Vector Store

An agent memory vector store is a database that turns an agent's past conversations, tool outputs, and facts into embeddings so the agent can retrieve relevant context later, instead of forgetting everything once the context window fills up. Without it, every agent session starts from zero: no memory of user preferences, no record of decisions made three sessions ago, no way to recall a fact learned yesterday. This guide builds a working agent memory vector store from scratch, covers chunking and embedding strategy, retrieval at query time, writing memories back after each turn, and the pruning rules that keep the store from turning into noise.

Why Agents Need Memory Beyond the Context Window

A large language model has no memory between calls. Everything it "knows" about a conversation lives in the context window you send with each request. That works fine for a single chat session, but it breaks down the moment you want an agent that:

• Remembers a user's name, timezone, or preferences across sessions
• Recalls a decision made in a task from last week without re-explaining it
• Builds up domain knowledge from documents or tool outputs over time
• Avoids repeating the same clarifying questions every session

You could try to stuff all of this into the system prompt, but context windows are finite and every extra token costs latency and money. A vector store solves this by acting as external, addressable long-term memory. Instead of loading everything, the agent loads only what's relevant to the current turn, retrieved by semantic similarity rather than exact keyword match.

This is the same idea behind retrieval-augmented generation (RAG), applied to the agent's own history instead of a static document corpus. The agent writes memories out as it works, and reads them back in when they become relevant again.

Choosing a Vector Store for Agent Memory

Any vector database can back agent memory, but the requirements differ slightly from document RAG. Agent memory tends to be:

• Small per-record: a memory is often a sentence or two, not a full document chunk
• High write frequency: memories get added after every turn, sometimes every tool call
• Metadata-heavy: you need to filter by user id, session id, memory type, and timestamp, not just similarity
• Subject to decay: old or superseded memories should be down-weighted or deleted

Popular options that handle this well:

• Chroma: embedded, zero-ops, great for local development and small to mid-size agents
• Qdrant: strong metadata filtering, good for production deployments with payload-based queries
• pgvector: if you already run Postgres, this keeps memory in the same database as your app data, which simplifies joins with user records
• Pinecone or Weaviate: managed options when you want to skip infrastructure entirely

For most single-agent or small multi-agent projects, an embedded store like Chroma is the fastest way to get memory working, and the same retrieval patterns carry over directly to a hosted store later. The rest of this article uses Chroma because it runs in-process with no server to stand up, which keeps the examples runnable end to end.

Building a Minimal Agent Memory Vector Store in Python

Install the dependency first:

pip install chromadb

Now create a memory store wrapper. This uses Chroma's default embedding function so the example runs without an external embedding API key, though in production you'd typically swap in a stronger embedding model from your LLM provider.

import chromadb
from chromadb.config import Settings
import uuid
import time

class AgentMemory:
    def __init__(self, persist_path="./agent_memory_db", collection_name="agent_memories"):
        self.client = chromadb.PersistentClient(path=persist_path)
        self.collection = self.client.get_or_create_collection(
            name=collection_name,
            metadata={"hnsw:space": "cosine"}
        )

    def add_memory(self, text, user_id, memory_type="fact", extra_metadata=None):
        metadata = {
            "user_id": user_id,
            "memory_type": memory_type,
            "created_at": time.time(),
        }
        if extra_metadata:
            metadata.update(extra_metadata)

        self.collection.add(
            documents=[text],
            metadatas=[metadata],
            ids=[str(uuid.uuid4())]
        )

    def recall(self, query, user_id, n_results=5, memory_type=None):
        where_filter = {"user_id": user_id}
        if memory_type:
            where_filter = {
                "$and": [
                    {"user_id": user_id},
                    {"memory_type": memory_type}
                ]
            }

        results = self.collection.query(
            query_texts=[query],
            n_results=n_results,
            where=where_filter
        )

        memories = []
        docs = results.get("documents", [[]])[0]
        metas = results.get("metadatas", [[]])[0]
        for doc, meta in zip(docs, metas):
            memories.append({"text": doc, "metadata": meta})
        return memories

This gives you two operations: add_memory to write, and recall to retrieve by semantic similarity scoped to a specific user. That scoping matters: without a user_id filter, one user's memories can leak into another user's context, which is both a correctness bug and a privacy issue.

Embedding and Chunking Strategy for Memories

Document RAG usually chunks text into 200 to 500 token blocks. Agent memory is different because the source material is already short: a single turn of conversation, a single extracted fact, a single tool result summary. The chunking decision that matters most for agent memory is what counts as one memory unit, not how to split a long document.

A few practical rules:

• One fact per memory. "User prefers dark mode" and "User's timezone is IST" should be two separate records, not one blob. This makes retrieval precision much higher because a query about timezone won't drag in an unrelated preference.
• Summarize before storing, don't store raw transcripts. Raw conversation turns are noisy and full of filler. Run a small summarization step (even a short prompt to the same LLM) to extract the durable fact before writing it to the store.
• Keep memories under roughly 100 words. Short memories embed more precisely and are cheaper to retrieve in bulk.
• Attach a memory_type. Splitting memories into categories like fact, preference, decision, and episodic lets you filter at query time instead of relying purely on similarity.

Here's a minimal extraction step that turns a raw agent turn into a storable memory:

def extract_memory_candidate(llm_client, user_message, agent_response):
    prompt = f"""Given this exchange, extract at most one durable fact worth
remembering long-term. If nothing is worth remembering, respond with NONE.
Keep it under 20 words, third person, no pronouns without antecedents.

User: {user_message}
Agent: {agent_response}

Fact:"""

    response = llm_client.generate(prompt)
    fact = response.strip()

    if fact.upper() == "NONE" or len(fact) < 3:
        return None
    return fact

llm_client.generate here stands in for whatever LLM call your stack already uses, Claude, GPT-4-class models, or a local model. The point of this step is that memory writing is itself a small agentic decision: not everything said in a conversation deserves to become a permanent memory.

Retrieval: Turning Memories Into Context

At query time, the pattern is: embed the current user message, search the memory store for the top-k most similar memories scoped to that user, then inject them into the system prompt before calling the LLM.

def build_context(memory_store, user_id, current_message, base_system_prompt):
    memories = memory_store.recall(current_message, user_id, n_results=5)

    if not memories:
        return base_system_prompt

    memory_lines = "\n".join(f"- {m['text']}" for m in memories)
    memory_block = f"\n\nRelevant memories about this user:\n{memory_lines}\n"

    return base_system_prompt + memory_block

Then the full turn looks like:

def run_agent_turn(memory_store, llm_client, user_id, user_message, base_system_prompt):
    system_prompt = build_context(memory_store, user_id, user_message, base_system_prompt)

    agent_response = llm_client.chat(
        system=system_prompt,
        messages=[{"role": "user", "content": user_message}]
    )

    fact = extract_memory_candidate(llm_client, user_message, agent_response)
    if fact:
        memory_store.add_memory(fact, user_id, memory_type="fact")

    return agent_response

Two things matter for retrieval quality here. First, n_results should stay small, five to ten memories is usually enough; dumping fifty memories into the prompt defeats the purpose of selective retrieval and reintroduces the context bloat problem you were trying to avoid. Second, always re-rank or filter by recency alongside similarity, because a semantically similar but stale memory can mislead the agent just as easily as no memory at all.

Memory Types: Short-Term, Long-Term, and Episodic

Not all memory should live in the vector store, and not all vector-stored memory should be treated the same way.

• Short-term memory is the current conversation's message history. This stays in the context window directly and never touches the vector store; it disappears when the session ends unless explicitly promoted.
• Long-term semantic memory is durable facts and preferences, the kind of thing that stays true across sessions: name, role, stated preferences, project constraints. This is the primary use case for the vector store shown above.
• Episodic memory is a record of specific past events: "on March 3rd the agent recommended switching database providers." This is also vector-stored, but tagged with memory_type="episodic" and usually retrieved less often, since most queries want current facts, not history.

Separating these matters because they decay at different rates. A preference might stay valid for months. An episodic memory about a specific decision might only be relevant for a follow-up question in the same week. Treating everything as one undifferentiated memory type is the single most common mistake in early agent memory designs, it leads to retrieval that surfaces old, superseded information alongside current facts with no way to tell them apart.

Writing Memories Back After Each Turn

Write-back is the step most agent memory tutorials skip, but it's what makes the system actually improve over time instead of staying static. There are three write triggers worth implementing:

1. End-of-turn extraction, shown above, runs after every response and captures small facts as they come up naturally.
2. Explicit user statements, phrases like "remember that" or "from now on" should bypass the extraction heuristic and get written directly, since the user has signaled intent explicitly.
3. Contradiction updates, when a new memory conflicts with an old one (the user says their timezone changed), the old memory should be marked superseded rather than left to compete with the new one during retrieval.

A simple contradiction check before writing:

def add_or_update_memory(memory_store, text, user_id, memory_type="fact"):
    similar = memory_store.recall(text, user_id, n_results=3, memory_type=memory_type)

    for m in similar:
        if is_likely_duplicate_or_conflict(text, m["text"]):
            memory_store.collection.delete(
                where={
                    "$and": [
                        {"user_id": user_id},
                        {"memory_type": memory_type}
                    ]
                }
            )
            break

    memory_store.add_memory(text, user_id, memory_type=memory_type)


def is_likely_duplicate_or_conflict(new_text, existing_text):
    new_words = set(new_text.lower().split())
    existing_words = set(existing_text.lower().split())
    overlap = len(new_words & existing_words) / max(len(new_words), 1)
    return overlap > 0.5

This is a crude overlap heuristic; a production system would use the LLM itself to judge whether two memories conflict, since word overlap alone can't distinguish "user likes tea" from "user does not like tea." The pattern matters more than the exact implementation: never write a new memory without first checking whether it invalidates an old one.

Pruning, Decay, and Forgetting

A vector store that only grows becomes slower and noisier over time. Retrieval quality degrades as near-duplicate and stale memories accumulate, because similarity search starts returning several slightly different versions of the same fact instead of one clean answer. Three pruning strategies keep the store healthy:

• Time-based decay: down-rank or archive memories past a configurable age unless they've been retrieved recently. A memory that hasn't been useful in months is a candidate for removal.
• Access-based reinforcement: track how often a memory gets retrieved and used in a response. Rarely-used memories are safer to prune than frequently-retrieved ones.
• Capacity caps per user: set a hard ceiling, for example 500 memories per user, and evict the lowest-scoring ones (oldest and least-accessed first) when the cap is hit.

def prune_stale_memories(memory_store, user_id, max_age_days=180, max_count=500):
    all_memories = memory_store.collection.get(
        where={"user_id": user_id}
    )

    ids = all_memories["ids"]
    metadatas = all_memories["metadatas"]
    now = time.time()
    cutoff = now - (max_age_days * 86400)

    stale_ids = [
        mid for mid, meta in zip(ids, metadatas)
        if meta.get("created_at", now) < cutoff
    ]

    if stale_ids:
        memory_store.collection.delete(ids=stale_ids)

    remaining = memory_store.collection.get(where={"user_id": user_id})
    if len(remaining["ids"]) > max_count:
        sorted_pairs = sorted(
            zip(remaining["ids"], remaining["metadatas"]),
            key=lambda p: p[1].get("created_at", 0)
        )
        overflow = len(remaining["ids"]) - max_count
        evict_ids = [pid for pid, _ in sorted_pairs[:overflow]]
        memory_store.collection.delete(ids=evict_ids)

Run this as a periodic background job, not on every request, since pruning is a maintenance task rather than something the agent needs synchronously during a conversation.

Common Pitfalls With Agent Memory Vector Stores

• No user scoping on writes or reads. Every add and every query must filter by user id or session id. Skipping this is the fastest way to leak one user's data into another user's context.
• Storing raw transcripts instead of extracted facts. This inflates the store, slows retrieval, and returns noisy context that confuses the agent instead of helping it.
• Retrieving too many memories per turn. More context is not automatically better; irrelevant memories dilute the prompt and can actively mislead the model.
• No contradiction handling. Without it, the store accumulates conflicting facts and retrieval becomes a coin flip about which version the agent sees.
• Treating memory as infinite. Every vector store benefits from pruning; skipping it turns a fast, cheap system into a slow, expensive one within a few months of real usage.
• Using the same embedding model inconsistently. If you change embedding models later, old vectors become incompatible with new ones. Either re-embed the whole store or version your collections by embedding model.

FAQ

What is agent memory in the context of AI agents? Agent memory refers to any mechanism that lets an LLM-based agent retain information across turns or sessions beyond what fits in a single context window. It typically splits into short-term memory (the current conversation) and long-term memory (facts, preferences, and events stored externally, often in a vector store).

Why use a vector store instead of just a database with keyword search? A vector store retrieves by semantic similarity, so a query like "what does the user prefer for notifications" can match a stored memory phrased differently, such as "user turned off email alerts," without requiring exact keyword overlap. Keyword search misses these paraphrases.

How many memories should an agent retrieve per turn? Five to ten is a reasonable starting point for most agents. Retrieving more dilutes the prompt with marginally relevant context and increases token cost without improving response quality.

Should every conversation turn be written to the vector store? No. Write only extracted, durable facts, not raw turns. A short extraction step after each response, deciding whether anything is worth remembering, keeps the store small and the memories precise.

Can the same vector store handle both document RAG and agent memory? Yes, but keep them in separate collections. Document RAG content and agent memory have different chunking strategies, metadata schemas, and lifecycle rules (documents rarely need pruning, memories do), so mixing them in one collection makes both harder to query cleanly.

What happens if two stored memories contradict each other? Without explicit handling, retrieval may surface both, and the agent has no signal about which is current. Implementing a contradiction check before every write, either through similarity plus an LLM judgment call or a simpler overlap heuristic, keeps the store internally consistent.

Does agent memory need a managed vector database, or can it run locally? For prototypes and small-scale agents, an embedded store like Chroma running on local disk is enough. Once you have concurrent users, high write volume, or need the memory store deployed across multiple servers, moving to a managed or server-based store like Qdrant, pgvector, or Pinecone becomes worthwhile.