Skip to the document
Madhuopen lab
← All the work, summarised

proprietaryAIMemoryEvaluation

hanumemAI

Remember everything, inject almost nothing.

Long-term memory for an assistant that reads a fifth of the context and has the highest published score on LoCoMo.

The idea

A language model is a brilliant doctor with amnesia: it forgets every patient the moment they leave. hanumemAI is the patient file. It keeps everything an assistant was ever told, and for each question puts only the few lines that matter in front of the model.

Read the full story →

Why it matters

Any memory system can score well by handing the model everything. The bill and the wait grow with every session. What counts is how little it can hand over and still be right, so every score here sits beside the tokens it read.

Long-term memoryContext costBenchmarksAssistants

How it works

In
Everything an assistant was ever told, and the question in front of it.
Out
The few lines that matter, small enough to read at a fraction of the usual context.

Closed source · results onlyHow it does this is not published. What it scored, and what it had to read to score it, is.

Explore deeper Each path is a different view of the same work.

Why it exists

A language model keeps nothing between one call and the next. What it knows about you is what is in front of it. Paste the whole history into every message and the bill grows with every session; summarise it and the detail you ask for next month is the one that was dropped.

A memory layer is the patient file: it decides, for each question, which few lines of everything it was told go in front of the model.

What it is judged by

Two numbers, always together: how often the answer is right, and how much the model had to read to get there. Any system can score well by handing the model everything.

So each score below sits beside its question count and its context tokens per question. A score without those cannot be compared with anything.

What it measured

On LoCoMo, all 1,540 questions: 96.1, the highest published score we can find, at 1,533 tokens per question. The systems nearest to it report 92.5 to 93.6 while reading 6,700 to 45,800.

On BEAM at one million tokens, all 700 questions: 68.5 against a published 64.1 for Mem0, at 40% of the tokens. On LongMemEval-S, all 500 questions: 94.0, in the top group and level with Mem0's 94.4, with a small open model answering.

What it does not claim

It is not first on LongMemEval-S, where 96.2 is the highest published figure, or on BEAM, where Hindsight reports 73.9. No two vendors share a judge, and none of these numbers, ours included, has been reproduced by a third party.

The same answers under a second judge score 91.6 on LoCoMo and 91.8 on LongMemEval-S. Both are published.

What was measured

Each number is read from the file named under it. Nothing here is rounded up.

96.1LoCoMo, all 1,540 questions, at 1,533 context tokens per questionmemory research log (private): STATUS.md §1
68.5BEAM-1M mean rubric, all 700 questions, at 2,716 tokensmemory research log (private): STATUS.md §1
94.0LongMemEval-S, all 500 questions, at 2,428 tokens (91.8 under a second judge)memory research log (private): STATUS.md §1
1,533context tokens per LoCoMo question; the nearest published systems read 6,700 to 45,800memory research log (private): STATUS.md §1-2
162logged runsmemory research log (private): results.tsv
23changes kept; 40 were discardedmemory research log (private): results.tsv

Common questions

Can I use it?

It is proprietary. Ask through the request form and say what you are building.

Is the design published?

No. The scores, the comparison and their limits are public. The book AI Agent Memory: Novice to Ninja teaches the subject from zero; how this library is built is shared on request.

Are the comparisons like for like?

No. Every system used its own answering model and judge. Other systems' figures are as their vendors published them.

Ask a question

Ask about what it does and what it measured. Answers come from what is written on this site.

Carry thisA memory score is a pair: how often it is right, and how much it had to read.

proprietary · results public · the library on request