Skip to the document
Madhuopen lab
Outcome School · Transformers and architecture13 min read

by Amit Shekhar · 5 September 2026

How does Sliding Window Attention work?

How sliding window attention works. We will also see why normal attention becomes slow and expensive for long text, and how sliding window attention comes to the rescue.

2,480 words#llm#ai#machine-learning4 recall cards

How does Sliding Window Attention work?
Read on Outcome School ↗then come back to lock it in
Before you read, guess

How does normal attention process word interactions?

Ten seconds, a guess, then read — a wrong guess still makes the answer stick.

What this article covers

  1. What is attention?
  2. The problem with normal attention
  3. What is sliding window attention?
  4. A simple step-by-step walkthrough
  5. How information still travels far away
  6. Comparing normal attention and sliding window attention
  7. Where sliding window attention is used
  8. Advantages and trade-offs

The article lives on outcomeschool.com. Read it there, then come back: the tutor in the margin has read it and will answer questions, and the questions below check what stayed.

Before you go

In one sentence, what was this chapter about?

From memory, without scrolling up. Writing it is what makes it yours; the grade is only to show you what you had.

How sure?