by Amit Shekhar · 5 September 2026
How do Attention Sinks work?
How attention sinks work in Large Language Models. We will also see why streaming long conversations becomes a problem, why the first tokens become a sink that the naive fix breaks, and how StreamingLLM uses attention sinks in the real world to keep models running.
Read on Outcome School ↗then come back to lock it in
Before you read, guessHow does the model treat previous words when predicting the next token?
Ten seconds, a guess, then read — a wrong guess still makes the answer stick.
What this article covers
- What is a Large Language Model
- What is attention
- The problem of streaming with long conversations
- The naive fix and why it fails
- What is an attention sink
- Why the first tokens become a sink
- A step-by-step numeric walkthrough
- The fix in code
- StreamingLLM and modern attention sinks
- Importance of attention sinks
The article lives on outcomeschool.com. Read it there, then come back: the tutor in the margin has read it and will answer questions, and the questions below check what stayed.