Skip to the document
Madhuopen lab
Outcome School · Transformers and architecture17 min read

by Amit Shekhar · 22 April 2026

Grouped Query Attention

Grouped-Query Attention (GQA) and how it differs from Multi-Head Attention (MHA).

3,292 words#llm#ai#machine-learning4 recall cards

Grouped Query Attention
Read on Outcome School ↗then come back to lock it in
Before you read, guess

What memory issue arises from storing Key and Value vectors for every head during text generation?

Ten seconds, a guess, then read — a wrong guess still makes the answer stick.

What this article covers

  1. The Big Picture
  2. Quick Recap: Multi-Head Attention (MHA)
  3. The Problem with Multi-Head Attention
  4. What is Multi-Query Attention (MQA)?
  5. What is Grouped-Query Attention (GQA)?
  6. How Grouped-Query Attention Works
  7. GQA is a Generalization of MHA and MQA
  8. GQA vs MHA vs MQA
  9. Real-World Use Cases
  10. A Note on Terminology
  11. Uptraining: Converting MHA to GQA

The article lives on outcomeschool.com. Read it there, then come back: the tutor in the margin has read it and will answer questions, and the questions below check what stayed.

Before you go

In one sentence, what was this chapter about?

From memory, without scrolling up. Writing it is what makes it yours; the grade is only to show you what you had.

How sure?