Why do AI agents forget?

In this blog, we will learn why a Large Language Model forgets everything between two conversations. We will also see why the obvious fixes fail, and what a memory layer costs compared to them.

What is a Large Language Model

A Large Language Model is a program that reads text and predicts the next word. It has no notebook. It has no hard drive of its own. Everything it "knows" about you has to be inside the text you send it right now.

Think of it as a brilliant consultant with amnesia. Every meeting, you walk in, and they have no idea who you are. If you want them to remember that you are vegetarian, you have to say it again. Every single time.

The problem, in one conversation

Monday
  Maya: "I'm vegetarian and I have a serious peanut allergy."
  Bot:  "Noted!"

Thursday, new chat window
  Maya: "Suggest a snack for my kids' school trip."
  Bot:  "How about peanut butter energy balls?"

Nothing was "noted". The Monday text is gone. The Thursday chat starts from zero.

The naive fix: resend everything

The first idea everyone has: just paste the whole history into every new chat. It works for a week. Then the math catches up.

one session          ~1,500 tokens
sessions per year    2 x 365 = 730
one year of chat     730 x 1,500 = 1,095,000 tokens

One year of one user's chat already overflows a one-million-token context window, the biggest windows we have today. And even before it overflows, you pay for all of it on every message.

at session 10   resend 10 sessions = 15,000 tokens  -> $0.045 per message
at session 100  resend all         = 150,000 tokens -> $0.45 per message

That is the consultant reading your entire life story aloud before every question. Slow, expensive, and it gets worse every day.

Why a bigger window does not save you

A bigger context window is a bigger table in the consultant's office. You can spread more papers on it. But you still have to carry every paper in every time, and the consultant still has to read every one of them before answering. The cost grows with the pile, not with the question.

Why fine-tuning does not save you either

Fine-tuning means retraining the model on your data. It is slow, it costs a lot, it cannot be edited or deleted easily, and one model per user is not a thing anyone runs. It is the wrong tool for "Maya moved to Paris last week".

The real fix: a memory layer

Instead of carrying every paper, keep a small card file. After each conversation, an assistant writes down the two or three things worth keeping. Before the next conversation, you pull the three cards that match the question and hand only those to the consultant.

        write path (after the chat, nobody waits)
  transcript ----> extractor ----> "Maya is vegetarian."
  1,500 tokens                    "Maya has a peanut allergy."
                                  "Maya has two kids, 6 and 9."

        read path (before the next answer)
  question ----> pick top 3 cards ----> ~75 tokens into the prompt

Now the read cost is flat. Three cards at session 10, three cards at session 1,000. The class notes put it at 200x cheaper at session 10 and 2,000x cheaper at session 100, against resending history.

What a memory layer must do

• Remember durable things across sessions, not small talk.
• Use them at the right moment without being asked.
• Update when life changes: "moved to Paris" must beat "lives in London".
• Never leak one user's memories to another.
• Let the user see, edit, and delete everything.

Each of those is one of the following blogs.

Closing

An LLM is a genius with amnesia. You cannot fix amnesia by shouting your whole life story louder each time. You fix it with a small, well-kept card file and a rule for which cards to pull. That card file is the memory layer, and the rest of this series is about how to build one that stays true, fast, and cheap.