Agent Memory Architectures Compared: Buffer, Vector, Graph

Why your agent forgets things it should remember

Ask a long-running agent about a decision it made three sessions ago and you'll usually get a shrug dressed up as a hallucination. It'll invent a plausible answer, contradict itself, or quietly re-do work it already finished. This isn't a model problem. It's a memory architecture problem, and it's one of the least glamorous, most consequential decisions you'll make when building an agent that needs to operate over more than a single conversation turn.

Most tutorials treat "memory" as one thing: shove the conversation into a list, truncate when it gets too long, done. But production agents — the ones doing customer support across weeks, managing a codebase across months, or running a research assistant across a project's lifetime — need something more deliberate. There isn't one correct memory architecture. There are three dominant families, each with different retrieval semantics, different failure modes, and different costs. Buffer memory keeps things in order. Vector memory finds things by meaning. Graph memory finds things by relationship. Most serious agent systems end up using a blend of at least two.

This article walks through each architecture with working code, explains where each one breaks down, and gives you a decision framework for picking (or combining) them for your own agent.

It helps to first be precise about what "memory" even means in an agent context, because the word gets used loosely. There's working memory — the stuff actively in the model's context window on this call. There's episodic memory — a record of specific things that happened, in order, that you might want to replay or reference later. There's semantic memory — general facts and relationships extracted out of specific episodes, independent of when they were learned. Buffer memory is a direct implementation of working memory. Vector memory typically implements episodic memory. Graph memory typically implements semantic memory. Keeping these distinct in your head makes it much easier to reason about which one you're missing when an agent behaves oddly, instead of reaching for "add more RAG" as a universal fix.

Buffer memory: the conversation as a list

Buffer memory is the default because it's the simplest possible thing that works: keep an ordered list of messages and pass some window of it back to the model on every call. It's not "dumb" so much as it is the correct choice for a huge fraction of agent use cases — anything where recency dominates relevance.

The core idea is a sliding window, usually bounded by token count rather than message count, since a single tool-call result can blow past a naive message limit.

class BufferMemory:
    def __init__(self, max_tokens=4000, token_counter=None):
        self.messages = []
        self.max_tokens = max_tokens
        self.token_counter = token_counter or (lambda s: len(s) // 4)

    def add(self, role, content):
        self.messages.append({"role": role, "content": content})
        self._trim()

    def _trim(self):
        total = sum(self.token_counter(m["content"]) for m in self.messages)
        while total > self.max_tokens and len(self.messages) > 1:
            removed = self.messages.pop(0)
            total -= self.token_counter(removed["content"])

    def as_context(self):
        return self.messages

The trim strategy above is the naive version: drop the oldest message. In practice you want to protect the system prompt and the most recent user turn from eviction, and you often want to summarize what falls off the front rather than discard it outright. That gives you a hybrid pattern — a rolling summary plus a raw tail:

class SummarizingBufferMemory(BufferMemory):
    def __init__(self, max_tokens=4000, summarizer=None, **kwargs):
        super().__init__(max_tokens=max_tokens, **kwargs)
        self.summary = ""
        self.summarizer = summarizer  # callable: (summary, messages) -> new_summary

    def _trim(self):
        total = sum(self.token_counter(m["content"]) for m in self.messages)
        overflow = []
        while total > self.max_tokens and len(self.messages) > 2:
            overflow.append(self.messages.pop(0))
            total = sum(self.token_counter(m["content"]) for m in self.messages)
        if overflow and self.summarizer:
            self.summary = self.summarizer(self.summary, overflow)

    def as_context(self):
        prefix = [{"role": "system", "content": f"Prior context: {self.summary}"}] if self.summary else []
        return prefix + self.messages

Buffer memory's strength is faithfulness to sequence and recency — great for a coding session where "what did we just try" matters more than "what did we discuss three weeks ago." Its weakness is exactly that: it has no notion of importance. A throwaway aside and a critical architectural decision get evicted in the same first-in-first-out order. It also doesn't scale — once a project spans hundreds of thousands of tokens of history, no window size makes buffer memory the right primary store. It's necessary but not sufficient for anything long-lived.

Vector memory: retrieval by meaning

Vector memory solves the scale problem by giving up strict ordering in exchange for relevance. Instead of keeping everything in a list, you embed each memory as a vector and retrieve the nearest neighbors to your current query. This is the architecture behind most "RAG for agent memory" implementations, and it's genuinely good at one thing: finding semantically similar past information regardless of when it happened.

Here's a minimal implementation using an in-process vector store, which is enough to reason about the mechanics before you reach for Pinecone, Weaviate, or pgvector:

import numpy as np
from dataclasses import dataclass, field
import time

@dataclass
class MemoryItem:
    text: str
    embedding: np.ndarray
    timestamp: float = field(default_factory=time.time)
    metadata: dict = field(default_factory=dict)

class VectorMemory:
    def __init__(self, embed_fn):
        self.embed_fn = embed_fn
        self.items = []

    def add(self, text, metadata=None):
        vec = self.embed_fn(text)
        self.items.append(MemoryItem(text=text, embedding=vec, metadata=metadata or {}))

    def search(self, query, k=5, recency_weight=0.0):
        if not self.items:
            return []
        q = self.embed_fn(query)
        now = time.time()
        scored = []
        for item in self.items:
            sim = np.dot(q, item.embedding) / (
                np.linalg.norm(q) * np.linalg.norm(item.embedding) + 1e-8
            )
            if recency_weight:
                age_days = (now - item.timestamp) / 86400
                sim = sim * (1 - recency_weight) + recency_weight * np.exp(-age_days / 30)
            scored.append((sim, item))
        scored.sort(key=lambda x: x[0], reverse=True)
        return [item for _, item in scored[:k]]

The recency_weight parameter matters more than it looks. Pure cosine similarity has no concept of "this happened yesterday vs. a year ago," which means a stale but semantically close memory can outrank a fresh, more relevant one. Most production systems blend similarity with a decay function, exactly like the exponential term above.

Where vector memory earns its keep: a support agent that's seen ten thousand tickets can pull up "the three most similar past resolutions" without anyone hand-tagging categories. Where it falls apart: multi-hop reasoning. If the answer to "why did we choose Postgres over Mongo for this project" depends on connecting a decision made in week 1 to a constraint discovered in week 6 to a benchmark run in week 9, vector search will happily return three unrelated-looking chunks and never surface the causal chain between them. Similarity is not the same as relationship, and that gap is exactly what graph memory is built to close.

It's also worth being honest about cost and latency. Every write is an embedding call; every read is an embedding call plus a similarity search. At small scale this is trivial. At the scale where you're storing hundreds of thousands of memories, you need an actual vector index (HNSW or IVF, not brute-force cosine), or your search() call turns into a linear scan that gets slower every day the agent runs.

Graph memory: retrieval by relationship

Graph memory represents facts as nodes and edges instead of chunks of text. Instead of "here's a document that mentions X," you get "X is-a Y, X depends-on Z, X was-decided-by-conversation-123." This structure lets you answer questions vector search structurally cannot: multi-hop queries, contradiction detection, and "what else does this affect if I change it."

A lightweight graph memory doesn't need a full graph database to start — you can prototype the concept with an adjacency structure and upgrade to Neo4j or a similar store once the shape proves useful:

from collections import defaultdict

class GraphMemory:
    def __init__(self):
        self.nodes = {}          # id -> {"type": ..., "attrs": {...}}
        self.edges = defaultdict(list)  # source_id -> [(relation, target_id, attrs)]

    def add_node(self, node_id, node_type, **attrs):
        self.nodes[node_id] = {"type": node_type, "attrs": attrs}

    def add_edge(self, source_id, relation, target_id, **attrs):
        self.edges[source_id].append((relation, target_id, attrs))

    def neighbors(self, node_id, relation=None):
        edges = self.edges.get(node_id, [])
        if relation:
            edges = [e for e in edges if e[0] == relation]
        return [(rel, self.nodes.get(tgt), tgt) for rel, tgt, _ in edges]

    def traverse(self, start_id, relation, max_hops=3):
        visited = {start_id}
        frontier = [start_id]
        path = []
        for _ in range(max_hops):
            next_frontier = []
            for node_id in frontier:
                for rel, target, attrs in self.edges.get(node_id, []):
                    if rel == relation and target not in visited:
                        path.append((node_id, rel, target))
                        visited.add(target)
                        next_frontier.append(target)
            if not next_frontier:
                break
            frontier = next_frontier
        return path

A concrete example: an agent working on a codebase decides to switch a service from REST to gRPC. That's one node (decision:rest_to_grpc). It connects via affects edges to component:auth_service, component:billing_service, and component:client_sdk. Three weeks later, someone asks the agent "what would break if we roll back the gRPC migration," and a graph traversal from that decision node returns exactly the three affected components — deterministically, in one hop, with no risk of the embedding model deciding those components aren't "similar enough" to the query.

The tradeoff is upfront cost. Someone — a human, an extraction LLM call, or both — has to decide what counts as a node, what counts as an edge, and what the relation vocabulary is. Unlike vector memory, which accepts arbitrary unstructured text with zero schema design, graph memory demands you commit to a structure before it's useful. Get the schema wrong and you'll spend more time refactoring the graph than you saved by having one. It also doesn't handle fuzzy, "kind of related" recall well at all — that's vector memory's job, not graph memory's.

Head-to-head: what each one is actually good at

Putting them side by side by failure mode rather than marketing copy:

• Buffer memory is correct by construction for recency and turn order, cheap to implement, and cheap to run, but it hits a hard wall the moment history exceeds your context budget, and it has zero concept of importance — a passing comment and a hard requirement are equally disposable.
• Vector memory scales to enormous unstructured histories and needs no schema, but it optimizes for surface-level semantic similarity, which quietly diverges from actual relevance, and it cannot reliably support multi-hop reasoning across facts that were never stated near each other.
• Graph memory is the only one of the three that supports deterministic multi-hop retrieval and contradiction/impact analysis, but it requires schema design up front, needs an extraction step to populate (usually another LLM call, which introduces its own error rate), and is overkill for anything that doesn't have genuine relational structure.

A rule of thumb that holds up in practice: use buffer memory for what happened *just now*, vector memory for what's *similar to now*, and graph memory for what's *connected to now*. An agent that only needs one of those three questions answered can genuinely ship with a single architecture. An agent that needs all three — which describes most agents doing real, long-horizon work — needs a layered system.

Building a layered memory system

The pattern that shows up repeatedly in production agent systems is a three-tier memory manager that routes writes to all layers and merges reads across them, letting each layer do the part it's actually good at.

class LayeredMemory:
    def __init__(self, buffer, vector, graph, extractor):
        self.buffer = buffer      # BufferMemory
        self.vector = vector      # VectorMemory
        self.graph = graph        # GraphMemory
        self.extractor = extractor  # LLM call: text -> list of (node, relation, node)

    def observe(self, role, content):
        self.buffer.add(role, content)
        if role in ("user", "assistant"):
            self.vector.add(content, metadata={"role": role})
        for src, relation, tgt in self.extractor(content):
            self.graph.add_node(src, "entity")
            self.graph.add_node(tgt, "entity")
            self.graph.add_edge(src, relation, tgt)

    def recall(self, query, entity_hint=None, k=5):
        context = {
            "recent": self.buffer.as_context(),
            "similar": [m.text for m in self.vector.search(query, k=k)],
            "connected": [],
        }
        if entity_hint:
            context["connected"] = self.graph.traverse(entity_hint, "affects", max_hops=2)
        return context

The extractor here is doing real work and deserves a moment of caution: turning free text into graph triples reliably is itself a hard NLP problem, and every extraction is a place errors can enter your knowledge base silently. A common mitigation is to only extract graph edges for high-confidence, explicitly stated relationships ("X depends on Y," "we decided to use Z instead of W") rather than trying to graph-ify every sentence. Let vector memory absorb the ambiguous stuff; reserve the graph for facts you'd be willing to bet on.

Retrieval-side, the entity_hint parameter matters: you generally don't want to run a graph traversal on every single turn, because most turns don't need one. A cheap heuristic — named entity detection, or just checking if the query references something already in graph.nodes — decides whether the extra traversal is worth doing. Cost control matters here as much as architecture; every layer you query on every turn is another few hundred milliseconds and another API call if embeddings or extraction are remote.

Consolidation and forgetting

An underrated piece of every one of these architectures is that memory needs to shrink, not just grow. Agents that never forget anything eventually drown in their own history — retrieval quality degrades as your vector store fills with near-duplicate memories, and your graph accumulates dead nodes referring to decisions that were later reversed.

Three consolidation strategies worth building in from day one rather than bolting on later:

• Time-based decay: weight older vector memories down (as shown in the recency_weight example) or archive them out of the active index after N days, keeping only a summary.
• Deduplication: before writing a new vector memory, check its similarity against existing ones; if it's above a threshold (say 0.95 cosine similarity), update the existing entry's timestamp instead of writing a near-duplicate.
• Graph pruning: when a decision node is superseded — the gRPC migration gets rolled back, say — don't delete the node, mark it status: superseded and add a new node with a supersedes edge. This preserves the history of *why* something changed, which is often more valuable than the current state alone.

def consolidate_vector_memory(vector_memory, similarity_threshold=0.95):
    kept = []
    for item in vector_memory.items:
        is_dup = False
        for other in kept:
            sim = np.dot(item.embedding, other.embedding) / (
                np.linalg.norm(item.embedding) * np.linalg.norm(other.embedding) + 1e-8
            )
            if sim > similarity_threshold:
                other.timestamp = max(other.timestamp, item.timestamp)
                is_dup = True
                break
        if not is_dup:
            kept.append(item)
    vector_memory.items = kept

Run something like this on a schedule — nightly for a long-lived agent, or after every N observations for a busier one — rather than trying to consolidate synchronously on every write. Consolidation is a batch job, not a request-path concern.

It's worth calling out a subtler failure mode here too: silent drift. Because consolidation runs unattended, it's easy for a dedup threshold that's slightly too aggressive to quietly merge two memories that were actually distinct, or for a decay function that's slightly too fast to make an important but rarely-referenced fact effectively unretrievable. Treat your consolidation job the way you'd treat a database migration — test it against a fixed set of memories with known expected outcomes before you let it run on live data, and log what it merges or archives so you can audit it after the fact. A memory system that forgets the wrong thing is worse than one that forgets nothing, because the failure is invisible until someone asks a question the agent should be able to answer and can't.

Evaluating whether your memory system is actually working

It's tempting to ship a memory layer and judge it by vibes — "the agent seems to remember things now." That's not a real evaluation, and it will not catch regressions when you change an embedding model, tweak a decay constant, or swap your graph extraction prompt. A more disciplined approach is to build a small, fixed set of memory-recall test cases before you need them, structured as (setup interactions, query, expected retrieval). For buffer memory, that might be "after these 12 turns, does the model still know the user's stated constraint from turn 2." For vector memory, "given these 50 stored tickets, does a query about a rare edge case return the 2 genuinely similar prior tickets in the top 5." For graph memory, "given this decision graph, does a 2-hop traversal from the migration node return all three affected services."

Running this kind of eval set after every meaningful change to your memory pipeline is cheap insurance. It also gives you a concrete way to compare architectures instead of arguing about them in the abstract — if your eval set shows vector memory alone misses 40% of the multi-hop questions your agent actually gets asked, that's a much stronger argument for adding a graph layer than intuition ever will be.

Choosing an architecture for your agent

A few concrete signals to decide with, rather than defaulting to whatever's trendy:

• If your agent's sessions are short-lived and self-contained (a single support ticket, a single coding task under an hour), buffer memory alone is probably sufficient. Don't add vector or graph layers you don't need — they're operational cost and failure surface for no benefit.
• If your agent needs to recall unstructured information across a large history — past conversations, documents, prior tickets — and "similar to this" is a good enough proxy for "relevant to this," add vector memory.
• If your agent needs to reason about how decisions, entities, or components relate to each other, and getting that wrong has real consequences (a code change breaking a dependent service, a policy change affecting three other policies), add graph memory. Don't add it speculatively; add it when you can name the multi-hop question you need answered.
• If you're not sure which you need, instrument first. Log what your agent actually gets wrong — stale context, missed similar cases, or missed relationships — and let the failure mode tell you which layer is missing, rather than guessing upfront.

None of this is exotic engineering. It's careful engineering: understanding what each data structure is actually good at, being honest about what it costs to build and maintain, and resisting the urge to reach for a knowledge graph when a sliding window would have done the job. The agents that hold up over weeks and months of real use aren't the ones with the fanciest memory system — they're the ones where the memory architecture actually matches the shape of what needs remembering.

If you want to build this kind of production-grade agent architecture hands-on — not just the memory layer, but the full stack of planning, tool use, evaluation, and deployment — that's exactly what we cover step by step in 30 Days of Hermes Agent, our project-based course on building real, deployable AI agents from scratch.