Long-Term Memory Systems for AI Agents

Agent long-term memory is the layer that lets an AI agent remember facts, preferences, and past decisions across sessions instead of forgetting everything the moment its context window resets. Without it, every conversation with an agent starts from zero, which is fine for a one-off Q&A tool but breaks down fast for anything that needs continuity: a coding assistant that should remember your repo conventions, a support bot that should remember a customer's history, or a research agent that should remember what it already tried. This article walks through the actual architecture of agent long-term memory, with runnable code, so you can build one instead of just reading about the idea.

Why Agent Long-Term Memory Is Different From a Bigger Context Window

The easy answer to "give my agent memory" is "just use a model with a huge context window." That doesn't work for three reasons.

First, cost and latency scale with tokens sent, so stuffing every past interaction into every request gets expensive and slow fast, even when the window technically fits. Second, relevance drops as context grows: models perform worse when the right fact is buried in ten thousand tokens of irrelevant history, an effect sometimes called "lost in the middle." Third, a context window is inherently ephemeral. It resets between sessions, between processes, and whenever you start a new conversation. Agent long-term memory has to live outside the model, in a store the agent can write to and query, so that memory persists independent of any single context window.

This is the same reason humans don't try to hold every fact they've ever learned in working memory. You have working memory (what you're actively thinking about right now), and you have long-term memory (facts, skills, and experiences you can recall when needed). A well-built agent needs the same split.

The Three Layers of Agent Memory

Most production agent memory systems, whether custom-built or from a framework like LangGraph, Mem0, or Zep, converge on the same three-layer model borrowed from cognitive science:

Working memory is the current context window: the system prompt, the last few turns, and whatever the agent pulled in for this specific task. It's fast, fully in-context, and gone when the session ends unless something writes it out.

Episodic memory stores specific events: "on this date, the user asked for X and the agent did Y." This is what lets an agent say "last time we talked about your deployment pipeline, you were using GitHub Actions." It's typically stored as timestamped records with embeddings for semantic search.

Semantic memory stores distilled facts and preferences, stripped of the specific episode that produced them: "the user prefers TypeScript over JavaScript," "the user's production database is Postgres." This is what you get after consolidating many episodes into durable knowledge.

A fourth layer, procedural memory, shows up in more advanced agents: stored strategies or workflows the agent has learned work well for certain tasks, closer to a cached skill than a fact.

Building agent long-term memory means designing storage and retrieval for episodic and semantic memory, and a process for turning the former into the latter.

Core Building Blocks

A minimal agent long-term memory system needs four pieces:

1. A store, where memories live as structured records (not just raw text blobs).
2. An embedding step, so memories can be retrieved by meaning, not just exact keyword match.
3. A retrieval function, that decides which memories are relevant to the current turn.
4. A write policy, that decides what gets remembered in the first place and how it gets consolidated over time.

Let's build each one.

Building a Minimal Memory Store

You don't need a specialized vector database to start. SQLite with a Python embedding library is enough to prototype the whole system and understand the mechanics before you reach for Pinecone, Weaviate, ChromaDB, or pgvector in production.

Install what you need:

pip install sqlite-utils numpy sentence-transformers

Here's a working memory store with add and retrieve operations:

import sqlite3
import numpy as np
import json
import time
from sentence_transformers import SentenceTransformer

model = SentenceTransformer("all-MiniLM-L6-v2")

class MemoryStore:
    def __init__(self, path="agent_memory.db"):
        self.conn = sqlite3.connect(path)
        self.conn.execute("""
            CREATE TABLE IF NOT EXISTS memories (
                id INTEGER PRIMARY KEY AUTOINCREMENT,
                text TEXT NOT NULL,
                kind TEXT NOT NULL,
                embedding BLOB NOT NULL,
                created_at REAL NOT NULL,
                last_accessed REAL NOT NULL,
                importance REAL NOT NULL
            )
        """)
        self.conn.commit()

    def add(self, text: str, kind: str = "episodic", importance: float = 0.5):
        embedding = model.encode(text).astype(np.float32).tobytes()
        now = time.time()
        self.conn.execute(
            "INSERT INTO memories (text, kind, embedding, created_at, last_accessed, importance) "
            "VALUES (?, ?, ?, ?, ?, ?)",
            (text, kind, embedding, now, now, importance),
        )
        self.conn.commit()

    def retrieve(self, query: str, top_k: int = 5):
        query_vec = model.encode(query).astype(np.float32)
        rows = self.conn.execute(
            "SELECT id, text, kind, embedding, created_at, last_accessed, importance FROM memories"
        ).fetchall()

        scored = []
        now = time.time()
        for row_id, text, kind, emb_blob, created_at, last_accessed, importance in rows:
            emb = np.frombuffer(emb_blob, dtype=np.float32)
            similarity = np.dot(query_vec, emb) / (
                np.linalg.norm(query_vec) * np.linalg.norm(emb) + 1e-8
            )
            recency = np.exp(-(now - last_accessed) / (60 * 60 * 24 * 7))  # 7-day decay
            score = 0.6 * similarity + 0.2 * recency + 0.2 * importance
            scored.append((score, row_id, text, kind))

        scored.sort(key=lambda x: x[0], reverse=True)
        top = scored[:top_k]

        for _, row_id, _, _ in top:
            self.conn.execute(
                "UPDATE memories SET last_accessed = ? WHERE id = ?", (now, row_id)
            )
        self.conn.commit()

        return [{"text": t, "kind": k, "score": s} for s, _, t, k in top]


store = MemoryStore()
store.add("User's production stack runs on Postgres and Redis.", kind="semantic", importance=0.9)
store.add("On 2026-06-02, user asked for help debugging a Redis connection pool leak.", kind="episodic", importance=0.6)

results = store.retrieve("what database does the user use in production?")
for r in results:
    print(f"[{r['score']:.2f}] ({r['kind']}) {r['text']}")

This is a real, runnable retrieval system, not a toy. It scores memories on three axes: similarity (does the memory match the current query semantically), recency (has it been accessed lately), and importance (how significant is this fact). That three-factor scoring formula, similarity plus recency plus importance, is the same approach used in well-known generative agent research and it holds up well in practice because pure similarity search alone tends to surface stale or trivial matches.

Deciding What to Write

The write policy matters more than people expect. Two failure modes show up constantly:

Writing everything turns your memory store into a junk drawer. Retrieval quality drops because irrelevant memories dilute the top-k results, and storage costs climb for no benefit.

Writing nothing automatically (relying on explicit "remember this" commands) means the agent forgets useful context the user didn't think to flag.

The practical middle ground is a lightweight importance filter that runs after each turn. You can implement this as a small classification call:

def should_remember(turn_text: str, llm_client) -> tuple[bool, float]:
    prompt = f"""Rate how important this exchange is to remember for future conversations,
on a scale of 0.0 (trivial, forget it) to 1.0 (critical fact or strong preference).
Only output a number.

Exchange:
{turn_text}
"""
    response = llm_client.complete(prompt, max_tokens=5)
    score = float(response.strip())
    return score > 0.3, score

Route the decision through this before writing to the store, and you avoid saving small talk while still catching preferences, decisions, and corrections the user makes mid-conversation.

Consolidation: Turning Episodes Into Semantic Facts

Left alone, episodic memory grows without bound and starts repeating itself: ten separate entries that all boil down to "user prefers Python over Go" is noise, not memory. Consolidation is the background process that periodically summarizes clusters of episodic memories into fewer, denser semantic ones.

A simple consolidation job looks like this:

def consolidate(store: MemoryStore, llm_client, batch_size=20):
    rows = store.conn.execute(
        "SELECT id, text FROM memories WHERE kind = 'episodic' ORDER BY created_at LIMIT ?",
        (batch_size,),
    ).fetchall()

    if len(rows) < batch_size:
        return

    combined = "\n".join(f"- {text}" for _, text in rows)
    prompt = f"""Summarize these interaction logs into 1-5 durable facts or preferences
about the user. Each fact should be a single, self-contained sentence.
Discard anything trivial or one-off.

Logs:
{combined}
"""
    summary = llm_client.complete(prompt, max_tokens=300)
    facts = [line.strip("- ").strip() for line in summary.splitlines() if line.strip()]

    for fact in facts:
        store.add(fact, kind="semantic", importance=0.8)

    ids = [str(r[0]) for r in rows]
    store.conn.execute(f"DELETE FROM memories WHERE id IN ({','.join(ids)})")
    store.conn.commit()

Run this on a schedule, for example nightly via cron, or trigger it once episodic memory count crosses a threshold. Either way, the goal is the same: keep the store small and dense, not large and redundant.

Injecting Memory Into the Agent's Context

Once retrieval works, wiring it into an agent loop is straightforward. Before calling the model, retrieve the top relevant memories and prepend them to the system prompt or as a tool result:

def build_context(user_message: str, store: MemoryStore, system_prompt: str) -> str:
    memories = store.retrieve(user_message, top_k=5)
    if not memories:
        return system_prompt

    memory_block = "\n".join(f"- {m['text']}" for m in memories)
    return f"{system_prompt}\n\nRelevant memory:\n{memory_block}"

Keep this block short. Five to ten memories is usually plenty; dumping fifty retrieved snippets into the prompt reintroduces the "lost in the middle" problem you built the memory system to avoid.

Common Pitfalls in Agent Long-Term Memory

Treating memory as a cache instead of a source of truth. If the underlying fact changes (the user switches from Postgres to MySQL), old memories need to be superseded, not just left to compete with new ones in retrieval. Store a superseded_by reference or run a contradiction check during consolidation.

No decay for episodic memory. Without a recency term in your scoring function, month-old trivia can outrank yesterday's correction if it happens to embed closer to the query. The decay term in the retrieval function above exists specifically to fix this.

Conflating memory with retrieval-augmented generation over documents. RAG over a knowledge base and agent long-term memory over interaction history solve different problems and often need separate stores: one is mostly static reference material, the other is a constantly changing personal record.

Skipping evaluation. Memory systems degrade silently. Build a small test set of "given this query, these memories should be retrieved" pairs and check retrieval precision whenever you change the scoring formula or embedding model.

Storing memories per-agent instead of per-user or per-project. If you have multiple agents (a coding agent, a support agent) that should share context about the same user, keep memory keyed by user or workspace ID, not by which agent wrote the memory.

Build vs. Buy: Picking a Memory Layer

For prototypes, the SQLite approach above is enough. For production, most teams pick one of these paths:

• Managed memory frameworks: Mem0 and Zep are purpose-built for agent memory, handling extraction, consolidation, and retrieval scoring out of the box, which saves you from reimplementing the pipeline above.
• Vector database plus custom logic: pgvector (if you're already on Postgres), ChromaDB, Weaviate, or Pinecone handle the embedding and similarity search, and you layer your own write/consolidation policy on top, similar to what's shown here.
• Framework-native memory: LangGraph's persistence layer and LlamaIndex's memory modules integrate memory directly into the agent loop if you're already building on one of those frameworks.
• Redis for short-lived session memory: useful as a fast working-memory layer that feeds into a slower long-term store, rather than as long-term storage itself.

None of these replace the design decisions above. A vector database gives you similarity search; it doesn't decide your write policy, consolidation schedule, or scoring formula. Pick the storage layer based on what you're already running, then build the memory logic on top regardless of which one you choose.

FAQ

What's the difference between agent long-term memory and RAG? RAG typically retrieves from a static or slowly-changing external knowledge base (documents, docs, code). Agent long-term memory retrieves from a dynamic record of the agent's own interactions and derived facts about a specific user or task. The retrieval mechanics overlap heavily, but the write path is different: RAG content is usually ingested in bulk ahead of time, while agent memory is written continuously as the agent operates.

Do I need a vector database, or is keyword search enough? Keyword search works for exact terms but misses paraphrases ("what DB does the user use" vs. "database the user runs in prod"). Embedding-based similarity search handles that gap. For small memory stores (a few thousand records), brute-force cosine similarity like the example above is fast enough; you only need an indexed vector database once you're past tens of thousands of records per user.

How do I stop the agent from remembering things the user wants forgotten? Add an explicit deletion path: a "forget" or "delete memory" tool the agent can call, plus a hard delete (not a soft flag) in the store. If you're handling anything sensitive, log deletions separately from the memory table itself so you can audit that a delete request was honored.

Should memory be shared across all of a user's sessions or scoped per project? Depends on the product. A coding agent probably wants memory scoped per repository (coding conventions for project A shouldn't leak into project B), while a personal assistant probably wants memory scoped per user across all contexts. Store a scope key (user_id, project_id, or both) on every memory record so you can filter at retrieval time without redesigning the schema later.

How often should consolidation run? There's no fixed number that works everywhere; it depends on interaction volume. A reasonable starting rule is to trigger consolidation once episodic memory for a given scope crosses a fixed count (20-50 records) or once a fixed time window (daily) has passed, whichever comes first, then tune based on how much your retrieval precision test set shows things drifting.

Can I just increase context window size instead of building this? For short-lived, single-session tasks, yes, that's simpler. Long-term memory becomes necessary once you need continuity across separate sessions or processes, since context windows reset between them regardless of how large the window is.