What agents should remember

1What the exam actually measures

Before reading any of the designs I spent time on the benchmark they are scored on, because a number is meaningless without knowing the test. The most quoted comparison in this area is against full context, and the full-context row is the one worth staring at: roughly 26,000 tokens a question, about 10 seconds at the median and 17 at the 95th percentile.

That row is what every memory system is really competing with, and it sets the honest framing. A memory layer is not competing on recall alone. It is competing on recall per token, and the baseline it has to beat has near-perfect recall and a terrible bill.

2Taking one system apart

I wrote a full read of one widely used system, from the ground up, as eleven chapters meant to be read in order. The interesting mechanics are not the storage. They are the decisions: an extractor that sees a running summary plus the last handful of messages, and then, per extracted fact, a single call that looks at the ten most similar existing memories and decides add, update, delete or do nothing.

The delete branch is the one people skip when they build this themselves, and it is why home-made memory drifts. Without it, 'Maya is vegetarian' and 'Maya eats fish now' both sit in the store forever, and retrieval returns whichever embeds closer to the question.

3Building the small version

The library I am building runs on SQLite and any OpenAI-compatible endpoint. No Docker, no local model server, no graph database. The graph everyone reaches for is two SQLite tables, and I have not yet found a query that needs more than that.

Retrieval scores on relevance, recency and importance together. Episodic memories are forgotten after ninety days unless something promotes them. The user identifier is a hard filter rather than a ranking feature, because that is the difference between a privacy boundary and a privacy preference.

The targets I am measuring against: a read budget around 75 tokens, under 50 ms of read overhead, and beating the open-source numbers that people actually reproduce rather than the headline ones, at a smaller token budget than the reference system's roughly 7,000 per question.

4What is still open

No benchmark numbers yet. The explainer is written and the library runs end to end; the scores on the three public benchmarks are the point of the exercise and they do not exist, so there is nothing here to claim. The scale arithmetic — a million users, two hundred million memories, around 664 GB raw — is a projection from the notes, not a measurement, and I have marked it that way everywhere it appears.

5Evidence

This thread has no lab notebook yet. What exists is the working folder below and the measurements in Table 1; until the runs are written up as claims with verdicts, treat the numbers as this page states them and no further.

Where the work lives:

• ~/apps/memory — the study and the explainer site.
• ~/research/memory_agents/mem0-from-the-ground-up.md — a full read of one system.
• ~/company/wsz-research/memory-class — notes.