Skip to the document
Madhuopen lab
Outcome School · Inference and serving22 min read

by Amit Shekhar · 11 April 2026

Decoding Flash Attention in LLMs

Flash Attention by decoding it piece by piece - understanding why standard attention is slow, what makes Flash Attention fast, how it uses GPU memory cleverly, and why it is used in almost every modern Large Language Model (LLM).

4,382 words#llm#ai#machine-learning4 recall cards

Decoding Flash Attention in LLMs
Read on Outcome School ↗then come back to lock it in
Before you read, guess

Why is standard attention slow?

Ten seconds, a guess, then read — a wrong guess still makes the answer stick.

What this article covers

  1. The Big Picture
  2. A Quick Recap of Standard Attention
  3. Why Is Standard Attention Slow?
  4. How GPU Memory Actually Works
  5. The Core Idea Behind Flash Attention
  6. Tiling
  7. The Softmax Problem
  8. Online Softmax
  9. Putting It All Together
  10. Recomputation in the Backward Pass
  11. Flash Attention 2
  12. Flash Attention 3
  13. Advantages of Flash Attention

The article lives on outcomeschool.com. Read it there, then come back: the tutor in the margin has read it and will answer questions, and the questions below check what stayed.

Before you go

In one sentence, what was this chapter about?

From memory, without scrolling up. Writing it is what makes it yours; the grade is only to show you what you had.

How sure?