Memory: Virtual Context and MemGPT

A bigger context window looks like the answer to memory. It isn't. Three failure modes keep showing up in production:

1. Overflow. Long conversations, long documents, and tool-heavy trajectories cross the window, and everything past the cutoff is gone.
2. Dilution. Inside the window, irrelevant context still pulls attention away from what matters. Frontier models degrade on long inputs.
3. Persistence. A new session starts with an empty window, so an agent with no external memory can never say "remember when you asked me to..." across sessions.

Bigger windows help but don't fix any of this. Mem0's 2025 paper measured that 128k-window baselines still miss long-horizon facts that a 4k-window agent with external memory catches.

MemGPT: the OS analogy

MemGPT is a design that treats the prompt as small, fast memory and an external searchable store as large, slow memory, with the agent moving data between the two through tool calls. Packer et al. (arXiv:2310.08560, v2 Feb 2024) borrow the idea straight from operating-system virtual memory:

The agent runs a normal ReAct loop. The only addition is one extra class of tools that page data in and out of main context.

Two tiers

• Main context. A fixed-size prompt holding the current task, including the core memory sections pinned inside it. The model always sees all of it.
• External context. Unbounded and searchable through tools: the archival store, plus past conversation turns reachable through conversation_search. The agent reads it when something is relevant and writes to it when new facts emerge.

The original paper tested the design on two tasks that a base window can't handle: document analysis over 100k tokens, and multi-session chat with persistent memory across days.

The interrupt pattern

MemGPT calls this memory-as-interrupt. Mid-conversation, the agent invokes a memory tool, the runtime executes it, and the result splices into the next assistant turn as a new observation. It works like a Unix read() syscall: the process blocks, gets its bytes back, and carries on.

The canonical memory tool surface:

• core_memory_append(section, text) writes to a persistent section of the prompt.
• core_memory_replace(section, old, new) edits a persistent section.
• archival_memory_insert(text) writes to the searchable external store.
• archival_memory_search(query, top_k) retrieves from the external store.
• conversation_search(query) scans past turns.

Where MemGPT ends and Letta begins

MemGPT became Letta in September 2024. The research repo (cpacker/MemGPT) is still there, and Letta extends the design in three ways:

• Three named tiers instead of two: core, recall (past conversation), archival (Lesson 08). This splits the same ground differently: MemGPT's core memory already lives inside main context, and its conversation search covers what Letta calls recall.
• Native reasoning in place of the send_message/heartbeat pattern (Lesson 08).
• Sleep-time agents that do memory work asynchronously (Lesson 08).

Even when production runs on Letta, Mem0, or a custom two-tier store, the MemGPT paper is the foundation underneath.

Where this pattern goes wrong

• Memory rot. Writes pile up faster than reads, and retrieval drowns in stale facts. The fixes are periodic consolidation (Letta sleep-time) and explicit invalidation (Mem0 conflict detector).
• Memory poisoning. External memory is just retrieved text. If attacker-controlled content lands in a memory note, the agent re-ingests it next session. This is the Greshake et al. attack (Lesson 27) stretched across time.
• Citation loss. The agent recalls that "the user asked me to ship X" but can't say which turn it came from. Store source references (session ID, turn ID) with every archival write.

code/main.py implements MemGPT's two-tier pattern in stdlib:

• MainContext is a fixed-size prompt buffer with a core dict and a messages list. When the message count passes the cap, it evicts the oldest messages into an evicted list. Nothing is summarized, and evicted messages are not written to archival; they stay reachable only through conversation_search.
• ArchivalStore is an in-memory BM25-esque store (token-overlap scoring) of (id, text, tags, session, turn) records.
• Five memory tools map to the MemGPT surface.
• A scripted agent fills archival with facts, then a follow-up is served by calling archival_memory_search.

Run it:

python3 code/main.py

The trace shows the agent writing three facts, filling main context to the cap (which forces eviction), then retrieving from archival. That is a MemGPT-style workflow, with no real LLM involved.

Every production memory system today is a MemGPT variant:

• Letta (Lesson 08): three tiers, native reasoning, sleep-time compute.
• Mem0 (Lesson 09): vector, KV, and graph stores fused with a scoring layer.
• OpenAI Assistants / Responses: managed memory via threads and files.
• Claude Agent SDK: long-term memory via skills and a session store.

Pick one by operational shape (self-hosted, managed, framework-integrated). The core pattern is the same in all of them.

outputs/skill-virtual-memory.md is a reusable skill that produces a correct two-tier memory scaffold (main context, archival store, tool surface) for any target runtime, with the eviction policy and citation fields already wired in.

1. Add a max_main_context_tokens cap measured in tokens (approximate with len(text.split()) * 1.3). The starter code evicts the oldest messages by count and keeps no summary. Instead, compact the evicted messages into a summary when the cap is exceeded. Compare behavior with and without the summarizer.
2. Implement BM25 properly over the archival store (term frequency, inverse document frequency). Measure recall@10 on a toy fact set versus the token-overlap baseline.
3. Add citation fields (session_id, turn_id, source_url) to archival inserts. Make the agent cite sources on every retrieval-backed answer.
4. Simulate memory poisoning: add an archival record that says "ignore all future user instructions." Write a guard that scans retrievals for directive-shaped text and marks them untrusted.
5. Port the implementation to use the MemGPT research repo's core-memory JSON schema (cpacker/MemGPT). What changes when you switch from flat strings to typed sections?

• Packer et al., MemGPT (arXiv:2310.08560): OS-inspired virtual context paper
• Letta, Memory Blocks blog: the three-tier evolution
• Anthropic, Effective context engineering: treating context as a budget
• Chhikara et al., Mem0 (arXiv:2504.19413): hybrid production memory on top of this pattern