Agent Memory

Origin: The concept of Agent Memory traces back to cognitive architecture models like CoALA (Cognitive Architectures for Language Agents) published by Princeton and CMU in 2023. By early 2026, memory has become the central problem of enterprise AI. As teams shift from simple chat interfaces to autonomous agents executing multi-step workflows, statelessness has gone from a minor inconvenience to a catastrophic failure point.

Problem: Standard LLMs are stateless. Every prompt is a fresh start. RAG can retrieve static documents, but it cannot track how an agent’s knowledge evolves. If your agent decides on an architectural path on Tuesday, it will forget that decision on Wednesday unless you re-explain the entire history. This leads to context drift, token rot, and memory hallucinations—costing companies millions in bloated inference fees and broken workflows.

Solution: Agent Memory Cascade—a multi-tiered state persistence system that allows agents to read, write, update, and resolve conflicts in their own context over time. By combining volatile working memory with long-term episodic, semantic, and procedural stores, agents gain persistent identities and reliable task continuity.

RAG vs. Agent Memory

The most common architectural error in production AI is treating RAG as a substitute for memory. They are fundamentally different subsystems:

Memory Pipelines

A production-grade memory system should never read or write directly to database storage. Instead, it relies on structured validation and retrieval pipelines:

The Memory Write Pipeline

Before a new observation is committed to long-term memory, it runs through an extraction and safety filter:

┌─────────────────┐     ┌─────────────────┐     ┌─────────────────┐     ┌─────────────────┐
│ 1. Extract Fact │ ──> │  2. Classify    │ ──> │  3. Resolve     │ ──> │  4. Mask PII &  │
│   From Context  │     │   Memory Type   │     │    Conflicts    │     │   Persist Fact  │
└─────────────────┘     └─────────────────┘     └─────────────────┘     └─────────────────┘

1. Extraction: The system isolates candidate facts, preferences, or events from the active thread.
2. Classification: The memory is routed based on taxonomy (e.g. Episodic vs. Semantic vs. Procedural).
3. Conflict Resolution: The write pipeline checks for contradictions with existing records. If the user downgraded from “Pro” to “Free”, the old fact is updated or deprecated rather than duplicated.
4. Safety & Storage: PII is redacted, and the cleaned memory is persisted to the appropriate storage backend with an audit trail.

The Memory Retrieval Pipeline

When the agent receives a new query, memories are surfaced dynamically to fit the context window:

1. Scope Identification: Detect if the query requires user-scoped, session-scoped, or organization-scoped knowledge.
2. Multi-Signal Querying: Query vector databases for similarity, graph stores for relations, and SQL databases for exact facts.
3. Relevance & Recency Ranking: Score retrieved candidates based on semantic match, recency, and importance.
4. Context Injection: Format and pack the top-K memories into the active context window, removing stale records.

LLM Wiki Pattern (Karpathy Design)

A concrete implementation pattern for agent memory systems based on Andrej Karpathy’s LLM Wiki design. Instead of treating memory as simple RAG retrieval, this pattern structures agent memory as a persistent, compounding wiki that the LLM actively maintains and updates.

The Problem with RAG-Based Memory

Most agent memory systems work like traditional RAG: retrieve relevant chunks at query time and generate answers. This approach has fundamental limitations:

The Wiki Solution

The LLM Wiki pattern fundamentally changes this equation. Instead of just retrieving from raw documents at query time, the agent incrementally builds and maintains a persistent wiki — a structured, interlinked collection of markdown files that sits between you and the raw sources.

The key difference: When you add a new source, the agent doesn’t just index it. It:

1. Reads and comprehends the content
2. Extracts key information
3. Integrates it into the existing wiki — updating entity pages, revising summaries, flagging contradictions
4. Strengthens or challenges the evolving synthesis

The knowledge is compiled once and kept current, not re-derived on every query.

Three-Layer Architecture

┌─────────────────────────────────────────────────────────┐
│                    RAW SOURCES                           │
│  Articles • Papers • Images • Data Files                │
│  (Immutable — source of truth)                           │
└─────────────────────────────────────────────────────────┘
                            ↓
┌─────────────────────────────────────────────────────────┐
│                      THE WIKI                            │
│  Summaries • Entity Pages • Concepts • Comparisons      │
│  (Agent-generated markdown — you read, agent writes)    │
└─────────────────────────────────────────────────────────┘
                            ↓
┌─────────────────────────────────────────────────────────┐
│                      THE SCHEMA                          │
│  Structure • Conventions • Workflows                    │
│  (Configuration file — makes agent disciplined)         │
└─────────────────────────────────────────────────────────┘

Layer 1: Raw Sources

Your curated collection of source documents — articles, papers, images, data files. These are immutable. The agent reads from them but never modifies them. This is your source of truth.

Layer 2: The Wiki

A directory of agent-generated markdown files containing:

• Summaries and entity pages
• Concept pages and comparisons
• Overview and synthesis documents

The agent owns this layer entirely. It creates pages, updates them when new sources arrive, maintains cross-references, and keeps everything consistent. You read it; the agent writes it.

Layer 3: The Schema

A configuration document (e.g., CLAUDE.md for Claude Code or AGENTS.md for other agents) that defines:

• Wiki structure and organization
• Naming conventions and formatting standards
• Workflows for ingesting sources, answering questions, maintaining the wiki

This is what makes the agent a disciplined wiki maintainer rather than a generic chatbot.

Core Operations

1. Ingest

When you add a new source, the agent processes it through a structured flow:

New Source → Read & Discuss → Write Summary → Update Index →
Update Entity Pages → Update Concept Pages → Append to Log

A single source might touch 10-15 wiki pages as the agent integrates the new information comprehensively.

2. Query

When you ask questions, the agent:

Key insight: Good answers can be filed back into the wiki as new pages. Your explorations compound in the knowledge base just like ingested sources do.

3. Lint

Periodically, the agent health-checks the wiki for:

Navigation Infrastructure

Two special files help navigate the growing wiki:

index.md (Content Catalog)

A content-oriented catalog listing:

The agent updates this on every ingest. When answering queries, it reads the index first to find relevant pages, then drills down.

log.md (Chronological Record)

An append-only timeline of wiki activity:

Pro tip: Use consistent prefixes (e.g., ## [2026-04-02] ingest | Article Title) to make the log parseable with Unix tools:

grep "^## \[" log.md | tail -5  # Last 5 entries

Why This Pattern Works

The tedious part of maintaining a knowledge base isn’t reading or thinking — it’s bookkeeping:

Humans abandon wikis because the maintenance burden grows faster than the value.

Agents excel at this because they:

Division of Labor

The human focuses on high-level thinking and curation. The agent handles everything else.

Context Engineering Strategies

Context engineering determines how retrieved memories are presented within the model’s token limit. Standard implementations rely on three patterns:

SLIDING WINDOW
┌─────────────────────────┬─────────────────────────┐
│   Summarized History    │   Verbatim Last Turns   │
└─────────────────────────┴─────────────────────────┘

HIERARCHICAL SUMMARIZATION
┌─────────────────┬───────────────────┬─────────────┐
│  Yearly (1 sentence) │ Monthly (1 paragraph) │ Verbatim    │
└─────────────────┴───────────────────┴─────────────┘

MEMORY OFFLOADING (ACE)
┌───────────────┬───────────────────┬───────────────┐
│ Static Prompt │ Retrieved Memory  │ Active Task   │
└───────────────┴───────────────────┴───────────────┘

1. Sliding Window

Keeps only the most recent $N$ turns verbatim. Older turns are discarded or summarized. Excellent for simple chatbots but poor for long-horizon planning.

2. Hierarchical Summarization

Compresses history at varying levels of abstraction. The last turn is fully preserved, the last session is summarized into a paragraph, and the older history is distilled into single-sentence semantic facts.

3. Memory Offloading (ACE)

Treats the context window like CPU RAM and external databases like a hard drive. Active task context is kept light, and all other facts are fetched via tools or dynamically injected based on semantic triggers.

Governance & Safety Checklist

When deploying memory systems in enterprise environments, safety and compliance are paramount:

• Enforce PII Masking: Never allow agents to commit raw SSNs, passwords, or emails to long-term memory. Use deterministic regex masking pre-write.
• Define Retention Policies: Establish expiration dates for volatile memory blocks (e.g., cart items or temporary session keys).
• Isolate User Scopes: Ensure memories from User A never leak into the retrieval path of User B.
• Audit Memory Writes: Log all automated memory edits and updates, allowing administrators to roll back corrupted states.
• EU AI Act Attestation: Ensure your memory retrieval logic can output explanation logs detailing why a specific memory was surfaced to influence a decision.