AI Agent Memory: Beyond RAG to State That Persists

A bigger context window is not memory. Paste your last hundred conversations into the prompt and the model can read them, but it still has no idea which fact is current, which was retracted, or which one you moved on from months ago. It re-derives your whole history from scratch every turn.

Memory is the opposite bet: keep a small, structured store of what is true now and update it as facts change. That sounds like RAG, and people use the words interchangeably, but they are two different jobs — and getting it wrong is why a chatbot still congratulates you on a job you quit. This post walks the mechanics behind YeetCode's interactive memory deep dive: why similarity retrieval and state-tracking diverge, the three ways a plain markdown memory file breaks, the write-time decision that keeps a store honest, and the decay and latency math behind real deployments.

Memory vs RAG: two jobs behind one word#

RAG retrieves by similarity. Given a query, it embeds it, scores every stored chunk by cosine similarity, and returns the closest match. Nothing in that computation knows which chunk is recent — a turn from six months ago scores exactly the same as one from this morning.

Memory tracks state over time. Each fact opens a time window; a later, contradicting fact closes the previous one. The answer to "what is true now?" is whichever window is still open. No embeddings, no similarity — ordering alone decides.

Feed both engines the identical history and the split shows, run over three turns with the query "what framework do I use for my work":

RAG returns "I love PyTorch" — the only turn that shares any words with the query, while the other two tie at zero overlap. Memory returns "moved to JAX", its window the only one still open. Same input, opposite answers: RAG chose by similarity, memory by recency.

How the simplest memory file breaks#

The primitive used across Claude Code, Codex, and Hermes is identical: a plain markdown file loaded at session start, no embeddings — relevance comes from grep or an LLM side-call. It is a great starting point, and it fails in exactly three ways.

1. The cap. Only a prefix of the file reaches the prompt:

With a 200-line cap and a 350-line file, max(0, 350 − 200) = 150 lines sit on disk, unread. Nothing errors — the file just grows until one day your most important fact scrolls past the ceiling and silently vanishes.

2. The semantic miss. A markdown file retrieves with grep — does this exact token appear? Store bash scripts/cpt-train.sh configs/sec-10k.yaml, then ask "how do I start training?": grep returns 0 matches, because "training" is not the token "train.sh". Word-level cosine also scores ~0, since different tokens give orthogonal vectors. Only sub-word structure bridges it — char-trigram cosine slices each string into overlapping 3-character windows (" tr", "tra", "rai", "ain"), and train and training share most of theirs, real similarity where grep saw nothing.

3. No sense of time. A flat file stores facts but not their order. Give it two contradicting facts — Based in Delhi (Jan) and Based in Bangalore (Jun) — and ask where the user lived in March: a no-time engine assigns both weight 1.0 and returns both. Add timestamps and each fact owns an interval [start, end); March falls inside [Jan, Jun), resolving deterministically to Delhi. The half-open window sends a query exactly at the move month to the new fact.

Write-time intelligence: ADD, UPDATE, DELETE, NOOP#

Reading is only half the system. The harder problem is what to do when a new fact arrives. Naive memory appends everything and drowns in duplicates and contradictions. Better systems — the mem0 model — run a small decision at write time, picking one of four operations by comparing the incoming fact to the store:

The defining property is that UPDATE is non-destructive. It does not overwrite; it closes the old fact's window (event_end = today) and opens a new one (event_end = null). The retracted fact survives as closed history, so "where did the user work last spring?" still resolves through the closed window.

Walkthrough: writing "Based in Bangalore"#

Start with a fixed store and thresholds add τ = 0.30, noop τ = 0.85. The incoming fact is "Based in Bangalore".

Swap the incoming fact and the classifier flips. "I review flashcards every morning" scores ~0.0 against everything → ADD. "the NeurIPS submission plan is off" carries the cue off → DELETE the matching plan (similarity 0.365, but the cue wins). An exact duplicate scores 1.0 ≥ noop τ → NOOP. Move the thresholds and a borderline fact like "Builds AI" (~0.63) reclassifies between NOOP and ADD — the op is computed, not scripted.

Forgetting without losing it: memory decay#

A store that only grows becomes noise, so you have to forget — but how you forget matters. Two functions of time since last access, both driven by a half-life knob τ:

At t = 0 both equal 1.0. From there deleteWeight collapses toward zero and the memory is gone for good, while downWeight sinks only to the 0.3 floor and holds. τ sets the pace — a large τ forgets slowly, a small τ drives the delete curve to near-zero fast — but the asymmetry never changes: down-weighting keeps a recoverable trace, deletion throws the information away. The lineage runs from Ebbinghaus's 1885 forgetting curve through MemoryBank to modern decay schemes, and the takeaway holds at every step: prefer the reversible operation.

The voice latency budget: a three-tier stack#

On voice, memory design is dictated by a stopwatch. A spoken round trip is dominated by stages memory cannot touch — end-of-speech detection (VAD), transcription (STT), the LLM's first token (TTFT), and audio synthesis (TTS). Memory gets only the leftover:

With a 700 ms target and a VAD 100 + STT 50 + LLM 300 + TTS 50 = 500 ms pipeline, the residual is max(0, 700 − 500) = 200 ms — comfortable (green). Thresholds: ≥100 ms green, ≥50 ms amber, >0 ms red, 0 ms over budget. Slow the LLM to 420 ms and the residual drops into amber; push the fixed stages past the target and memory gets 0 ms.

Because the budget is a residual, speed is set by what you prepared, not what you fetch — which forces a three-tier stack, each tier assigned by whether its cost fits the residual:

Drag the LLM stage up and background retrieval gets shoved out of budget — the assignment re-derives from the arithmetic. When the residual is tiny, you cannot fetch at request time at all; the work must already be done.

Keep going#

• What Is RAG? Retrieval-Augmented Generation from the Ground Up — the retrieval half of the story, in depth.
• Advanced RAG: Hybrid Search, Reranking & Query Transformation — how similarity retrieval gets sharper before you reach for memory.
• LangGraph Explained: Agents as State Machines — where a memory store plugs into an agent's control flow.
• Recursive Language Models: Context as a Variable — treating context itself as something you manipulate, not just fill.

Then open the interactive deep dive and run all of it live: watch RAG and memory disagree on the same turns, break the markdown file three ways, flip the write classifier by moving a threshold, and drag the voice pipeline until memory falls out of budget.

FAQ#

What is the difference between AI agent memory and RAG?#

Think of it as search versus bookkeeping. RAG is a ranked index: it measures how closely each stored passage resembles your question and returns the top hit, treating a note from last year and one from this morning as equally valid. Memory is a ledger people keep posting corrections to — it cares about the order of edits, not their wording, so a newer entry marks an older, conflicting one as spent. Ask RAG for your current tool and it may hand back a note that gushed about one you dropped; ask memory and it reports the entry still standing. Many production stacks run both: RAG to pull documents, memory to decide which facts are live.

What do ADD, UPDATE, DELETE, and NOOP do in a memory system?#

They are the four write-time operations, chosen by comparing an incoming fact to the store. ADD saves a genuinely new fact. UPDATE fires when a fact fills the same slot with a different value — it closes the old window and opens a fresh one, erasing nothing. DELETE fires on an invalidation cue like "off" or "cancel" and closes the fact as retracted history. NOOP is the no-change path for near-duplicates. Because the choice is computed from similarity and cues, moving the thresholds can reclassify the same fact.

Why down-weight old memories instead of deleting them?#

Because ranking a memory down is reversible and erasing it is not. Down-weighting only pushes a fact lower — its score follows 0.3 + 0.7·e^(−t/τ) and bottoms out at 0.3, never zero — so a fact that goes quiet for months can climb back the moment it matters again. Hard deletion follows e^(−t/τ) to zero, after which nothing is left to recover. The same half-life τ tunes how fast each fades; the difference is one keeps an option open and the other spends it.

How does latency shape memory for voice agents?#

Voice hands memory only the slack after the unavoidable stages — max(0, target − Σ fixed stages). Budget 700 ms, spend 500 on VAD, STT, the LLM, and TTS, and 200 ms of headroom remains; a slower model slides that into an amber or red band, and if the fixed stages alone overrun the target, memory's share is zero. So retrieval cannot be request-time work: hold a hot cache read in roughly 5 ms, run heavier retrieval in the background across a few turns, and defer writes to after the response, off the caller's clock.