Much has been said, published and iterated on the topic of memory systems over the last few years. Most of that work, and most of the benchmarks, is about finding a specific piece of information in a large amount of data. I think that part is mostly solved. The bigger problem in daily work is that the agent often doesn't even know there is something to search for.
Update, Oct 2026: Since writing this I found a benchmark that tests exactly this blind spot, InMind. pd-memory gets 70.4% of its tasks right; the best memory system in the paper gets 14.4%. The InMind team verified the result, and pd-memory is now first on their leaderboard. Details are in the README.
Where I'm coming from
I have been playing with memory systems on the side for years, and for about a year professionally, too. I first got interested in this through Notion and the whole second brain idea, where you capture everything, link it and end up with a system that helps you organize. I soon realised that the manual labour to do this well is just too high, at least for my taste.
LLMs remove most of that labour, since reading, sorting, summarising and linking are all cheap now. But they bring their own problem of relevance realization, of deciding what matters, and that is exactly the kind of judgment where humans currently still have an advantage. So we have to give the system a seed of that judgment it can build on, and ideally extract a lot of it implicitly from what gets ingested.
I think the platonic ideal is a system you give a rough overview of your goals. It takes in arbitrary streams of information and assembles them into whatever shape is most navigable. The reader of that shape may be me or another agent. Agents and humans need somewhat different things from an information store, but there are many similarities as well, for instance not being overloaded with information that isn't relevant to the task.
The question guiding my design thinking is whether there is a fundamental solution here that works across people and organisations, or whether this is inherently specific.
Search is mostly solved
Two simple approaches already get you surprisingly far. One is Karpathy-style Markdown wikis, where an agent maintains a folder of plain notes and reads them back in. The other is to embed everything and let agents search, since retrieval over large amounts of data works well now.
Fundamentally, memory is a problem of compression and search.
There are many benchmarks that measure whether a system can find a specific piece of information in a large amount of data. I tested my own system on one of them, LongMemEval, and it does about as well as plain retrieval. At that point I noticed that these benchmarks weren't really measuring the right thing in the first place.
Benchmarks like this don't separate the approaches much anymore, because they assume the question is already known.
What I would want to measure instead is whether an agent knows what to look for, rather than how well the search works for a known needle. When I wrote this I didn't know of any benchmark that captures it; see the update at the top.
Why the agent needs a map
If I ask what to gift my girlfriend, that's easy, because the agent knows what it is missing and can write a query for it. But if I ask what I should read this weekend, there is no good query. The agent could search for my interests and would get back a few entries that happen to match, but it can't tell what it isn't seeing, which of these are current and which are central. The answer depends less on any single fact than on the shape of everything that is there.
The more open-ended a request becomes, the more this matters. A request like "tell me something interesting" depends almost entirely on that shape.
Importantly, I also think this will matter more over time. AI work seems to be moving towards higher levels of abstraction, and the higher the level, the more an answer depends on context that was never mentioned in the request.
How do we design the map?
This is what my recent project, pd-memory, tries to answer. It is my attempt at a general solution, though what it looks like is itself shaped by how context works in current AI systems.
The core constraint for an agent is token bloat, since everything in the standing context is read again on every turn and competes for the model's attention.
There are two failure modes to avoid: by trying to achieve completeness, one may be tempted to increase the token budget and bloat the context, and on the other end there is the search-only case, where the agent doesn't know what it can search for.
So the map has to be compressed strongly, and at first that gives you a long list of one-liners. What I landed on is a dictionary.
The central piece is this dictionary-style overview, which is assembled on demand and in variable length. It can cover the whole system, a specific area or subtopic, or whatever a specific conversation is about, as a starting point for the agent. From there the agent can notice where it lacks detail, open the relevant entry, follow links to related entries and come back with what it needs.
Here is a heavily cut slice of the whole-system overview my own agents get at the start of a session.
## daimon (artifact) — Daimon – graphical interface and backend for coding conversations, sessions, context handoff, presented artifacts, and probability-benchmark work. (125)
- 9dbcbe [2+] Daimon should create the conversation thread and durable submission together on the first Send, so Pi starts from recorded user intent rather than an empty pre-created session. (3)
...
## pd-memory (project) — pd-memory – experimental project for progressively disclosed, navigable, editable, and uncertainty-aware durable memory for agents. (45)
- 0d1c84 [2+] pd-memory should retain only knowledge likely to remain useful several weeks later, prioritizing lasting importance over mere truth. (5)
- 403282 [2+] pd-memory must attribute a decision to Mark only when Mark explicitly confirms it; an assistant proposal is never Mark's decision without that confirmation. (1)
...
### local-conversation-archive-ingestion (project) — Local conversation archive ingestion – planned local processing of Mark's Codex, Claude, and related conversation archives for durable memory. (7)
...
## pi (artifact) — Pi – authoritative coding-session runtime providing session history, native tools, extensions, and runtime semantics. (25)
...
## backup-and-recovery (topic) — Backup and recovery – durable backup coverage, restoration practices, and retention decisions for Mark's Macs and project data. (6)
- 7f14eb [2+] Restic provides deduplicated, versioned snapshots, so deleting a source file does not remove it from earlier snapshots until retention removes the last snapshot containing it. (1)
...
## mark (actor) — Mark – Person whose durable views, decisions, and working preferences are recorded here. (7)
...
## forecasting-and-calibration (topic) — Forecasting and calibration – prediction under uncertainty, prospective evaluation, and the reliability of reported confidence. (5)
- 52722a [2+] Mini-Jev may initially use logits as a confidence signal, but they must be calibrated and validated rather than treated as trustworthy probabilities by default. (1)
...
## battlesim (project) — BattleSim – project for experimenting with and evaluating combat simulation models and implementations. (8)
### battlesim-renderer (artifact) — BattleSim renderer – code-first renderer work for inspecting replayed battles with rich terrain, free-camera exploration, and individually represented soldiers. (3)
...How to read this
Headings are pages, which are durable subjects and can be nested. The lines below them are observations. The id is used to open an entry, the number in brackets shows how prominent it is, a + means there is more detail behind it, and the count at the end is the number of sources that back it.
This slice already shows that I am building a memory system and an agent interface, and working on forecasting and a combat simulation on the side. For the question of what to read this weekend, that says more than any search for my interests would.
Getting this format right is mostly a question of cognitive ergonomics for the model. Raw UUIDs waste tokens, and if opening the next layer takes several awkward tool calls, the agent will guess from the summary instead.
This slice, like my memory in general, is currently strongly biased towards engineering, because coding sessions are what I have ingested so far. The mechanism itself is meant to be general. It should work the same way for a company knowledge base or a personal memory built from chat logs, and for almost any topic, because the dynamics of what an agent needs to know and when it needs it are the same.
It loses out when the big picture itself is the core information, because extracted observations tend to drop it. An earlier system I built worked the other way around, by summarising sources and progressively compressing older summaries into higher-level ones. It kept the big picture well and works well for clearly scoped projects. But as soon as a claim is relevant to several projects, it gets tricky, because the claim sits somewhere inside a mass of summary text that has to be combed through. Smaller units of information are much handier here, especially for assembling an overview on demand. That is what pushed me towards individual entries.
I haven't figured out yet how to get the big picture back into them. The direction I want to go is to make the basic entry wide enough to also hold observations that cut across many sources, so that statements about the big picture live in memory next to the individual facts and can be opened and corrected the same way.
Is text the right level?
Not really. Ultimately much stronger compression will be possible that isn't based on text at all. The most interesting work I've seen here is neural KV cache compaction by Baseten, which they also presented at Cursor's Compile conference. Instead of summarising into text, a small learned network compresses the model's own internal cache, and the model reads the compressed version as if it were the original context. At 8x compression it still answers about 85 to 90% of fact questions about the document correctly, and it does this within milliseconds, where earlier methods like Cartridges and Attention Matching need seconds to hours per document. They describe the same gap I am working on: memory today is either the full context, which is lossless but expensive, or Markdown files and retrieval, which are cheap but lossy. All of this needs direct access to the model's internal state, so it only works if you run the model yourself.
Cursor's own approach stays in text. They trained Composer to write its own summaries, and these came out at around a fifth of the length of a carefully prompted baseline, while the errors caused by compaction dropped by half. So relevance realization can be learned when there is a reward signal. In a coding task that signal arrives quickly, because you soon see whether the summary kept the right things, while with long-term memory it may take weeks to find out.
For now I see text as an intermediate stage. Its big advantage is that it is readable, which means humans and agents consume the same representation, and that makes it easy to think about and iterate on. It's also quite interesting to read what your agents think they know about you, I can recommend it.
The code is here.