"Agent memory: beyond the context window"

The context window is a whiteboard: generous, fast, and wiped clean between conversations. Give users an agent that forgets their name, their preferences, and last week's decisions, and no amount of reasoning ability saves the experience. So every serious agent grows a memory system — and most of them grow the wrong one first.

The three tiers

It helps to name the layers, because they solve different problems:

Session (the transcript). What happened in this conversation: messages, tool calls, results. Frameworks give you this for free — ADK's Session holds the event log and a state dict, and its key-prefix convention (user: for cross-session user facts, app: for shared config, plain keys for this-session scratch) is a nice forcing function for deciding what outlives the chat.

Working state (the scratchpad). Structured facts the agent needs now: the plan, IDs collected so far, flags. Keep this out of prose and in a typed dict — an agent that re-derives the customer ID from conversation history on every turn is wasting tokens and inviting drift.

Long-term memory (the archive). Facts that must survive across sessions: preferences, past decisions, project context. This is the tier everyone rushes to build with a vector database, and it's where most of the mistakes live.

Retrieval is the easy half

Semantic search over past conversations is a solved problem. The hard half is writing: deciding what deserves to be remembered at all.

Storing raw transcripts and retrieving "relevant chunks" fails in practice — you retrieve the moment someone mentioned a deadline, not the later moment it moved. What works is distillation: at session end (or at meaningful moments), have the model extract discrete facts — "prefers staging deploys on Fridays", "project X targets the EU launch, decided 2026-04-12" — and store those as individual records with provenance. Small, atomic, dated facts retrieve cleanly and, crucially, can be corrected one at a time when the world changes.

Which is the part nobody budgets for: forgetting. Facts go stale. A memory system without updates and deletions doesn't make your agent smarter, it makes it confidently out of date. Treat contradiction as a first-class event — new fact supersedes old, old one gets tombstoned, and the agent should trust recency.

Practical rules I've settled on

1. Distill at write time, not read time. Store facts, not transcripts.
2. Date everything, and convert "next Friday" to an absolute date before storing. Relative time is a memory poison.
3. Retrieved memories are hints, not instructions. Pipe them into context clearly labeled as background — an old memory that says "always deploy to prod immediately" should not outrank the current user's explicit ask.
4. Cap the tier sizes. A dozen crisp user facts beat three hundred vague ones; retrieval quality degrades with hoarding.
5. Let the user see and edit it. Memory that can't be inspected is memory that can't be trusted — by them or by you.

The pattern across all five: memory is an editorial process, not a storage problem. The database was never the hard part.