Chapter 4

How does an agent find the right memory?

About the read path: how a question turns into the three or ten memories that go into the prompt. We will also see why one search signal is never enough, and how three signals are fused into one ranking.

4 min read886 words3 recall cards

Before you read, guess

How does an agent determine which memories to retrieve?

Take ten seconds and guess — even a wrong guess makes the answer stick. Tap to see where the chapter lands, or just read on.

Then they remember what is recent and what is important.

In this blog, we will learn about the read path: how a question turns into the three or ten memories that go into the prompt. We will also see why one search signal is never enough, and how three signals are fused into one ranking.

The librarian

You walk up to a librarian and ask "any ideas for a weekend activity with the family?" A good librarian does not hand you every book. They think about what you mean, remember that you have two kids, and pull three books.

The read path is that librarian, and it has to be fast. The notes budget under 50 ms, invisible next to the 200 to 500 ms the model takes to say its first word.

question --> embed (~20 ms) --> search user's memories (~5 ms) --> top 3 --> ~75 tokens

Signal 1: meaning (dense vectors)

An embedding model turns text into a list of numbers, a point in space. Texts with similar meaning land close together. "What should I cook tonight?" lands near "Maya is vegetarian" even though they share no words.

0.435  Maya has two kids, aged 6 and 9.            <- in
0.382  Maya is training for the Berlin marathon.   <- in
0.373  Maya's partner Sam has a birthday June 18.  <- in
0.293  Maya has a serious peanut allergy.          <- stays behind

That is cosine similarity, and it is the backbone of every memory system. It has one blind spot: names and exact terms. "Osteria Francescana" and "a nice restaurant" embed close together, and that is exactly when you needed the name.

Signal 2: words (BM25)

BM25 is the classic keyword score behind search engines. It rewards memories that contain the rare words of the query. Ask about "Poppy" and the memory that says "Poppy" wins, no matter what the embedding thinks.

Mem0 v3 runs BM25 next to the vector search and normalizes the raw score with a sigmoid so it lands in [0, 1] like the cosine does. In our build, SQLite's FTS5 gives us BM25 for free, no extra service.

Signal 3: names (entities)

Mem0 v3 adds a third signal. At write time it pulls proper nouns out of each memory ("Poppy", "Shopify", "Paris") and keeps a table of entity -> memories that mention it. At read time it pulls entities out of the query, looks them up, and boosts the linked memories.

This is a tiny graph. Not Neo4j, just a table. And it is the cheapest form of "multi-hop": a question about Sam's gift reaches "Sam loves vinyl" through the shared entity Sam even if the embedding of "gift" is far away.

Fusing the three

Each signal is normalized to [0, 1], then added with weights and divided by the maximum possible, so the final score is in [0, 1] too.

score = ( w_dense * cosine + w_bm25 * bm25 + w_entity * boost ) / (w_dense + w_bm25 + w_entity)

Mem0 uses 1.0, 1.0, 0.5. The alternative is Reciprocal Rank Fusion, which adds 1/(60 + rank) across lists and ignores raw scores. Both work; the weighted sum is easier to explain with explain=True, so we start there and measure RRF later.

Beyond similarity: recency and importance

The Generative Agents paper added two more terms, and the class scoring experiment shows why. Query: "gift for Sam?"

                                   rel    rec    imp
"printer is out of ink" (1h old)   low    1.00   1      recency's pick
"Sam loves vinyl + old jazz"       high   0.00   6      the right answer
(83 days old)                                           rescued by rel + imp

recency    = 0.995 ^ hours_since_last_access      (halves every ~6 days)
importance = LLM-rated 1..10 at write time
score      = relevance + recency + importance      (each min-max normalized)

Relevance alone cannot tell a critical fact from chit-chat. Recency alone retrieves trivia. Importance rescues old-but-critical facts. Production systems tune the three weights; we will tune them on a dev split.

Time-aware queries

"What did I do last summer?" has a date range hidden in it. The LongMemEval paper found that turning that phrase into a range and filtering on the memory's event date improved temporal recall by 7 to 11 points. So the read path has a small parser: "last week", "in 2023", "yesterday" become (start, end) relative to the question's date, and memories whose event_date falls inside get a boost.

What goes into the prompt

Known facts about the user (newest first, with dates):
- Maya has two kids, aged 6 and 9.              (since 2026-01)
- Maya is training for the Berlin marathon.     (since 2026-03)
- Maya's partner Sam has a birthday on June 18. (since 2026-02)
Personalize the answer using these facts where relevant.
Do not recite these facts back unless asked.

Dates ride along so the model can tell last week from last year. The block is capped in tokens, so cost stays flat whether the store holds 8 memories or 8,000.

Closing

A good librarian uses three senses: what you mean, the words you said, and the names you dropped. Then they remember what is recent and what is important. Fuse those, keep the top few, and stamp a date on each. That is the read path.