Skip to the document
Madhuopen lab
Outcome School · Transformers and architecture16 min read

by Amit Shekhar · 5 April 2026

Math behind √dₖ Scaling Factor in Attention

Why we scale the dot product attention by √dₖ in the Transformer architecture with a step-by-step numeric example.

3,059 words#math#llm#ai#machine-learning4 recall cards

Math behind √dₖ Scaling Factor in Attention
Read on Outcome School ↗then come back to lock it in
Before you read, guess

What characteristic do the outputs of the softmax function exhibit?

Ten seconds, a guess, then read — a wrong guess still makes the answer stick.

What this article covers

  1. The Attention Formula (Quick Recap)
  2. What Happens Without Scaling?
  3. Why Do Dot Products Grow with dₖ?
  4. Understanding Variance of the Dot Product
  5. Proving It Step by Step: Variance of the Dot Product is dₖ
  6. What Large Dot Products Do to Softmax
  7. Why √dₖ is the Right Scaling Factor
  8. Seeing It with Real Numbers
  9. Putting It All Together

The article lives on outcomeschool.com. Read it there, then come back: the tutor in the margin has read it and will answer questions, and the questions below check what stayed.

Before you go

In one sentence, what was this chapter about?

From memory, without scrolling up. Writing it is what makes it yours; the grade is only to show you what you had.

How sure?