Skip to the document
Madhuopen lab
Outcome School · Transformers and architecture16 min read

by Amit Shekhar · 13 August 2026

Decoding InstructGPT

In this blog, we are going to learn about InstructGPT, the model that taught GPT-3 to actually follow our instructions, and the work that led directly to ChatGPT.

3,122 words#ai#machine-learning4 recall cards

Decoding InstructGPT
Read on Outcome School ↗then come back to lock it in
Before you read, guess

What is the core mismatch in InstructGPT's alignment problem?

Ten seconds, a guess, then read — a wrong guess still makes the answer stick.

What this article covers

  1. What is the InstructGPT paper?
  2. The building blocks we must know first
  3. The big picture: what InstructGPT does
  4. Why GPT-3 was not enough
  5. Helpful, Honest, and Harmless
  6. The three-step method
  7. Step 1: Supervised Fine-Tuning
  8. Step 2: The Reward Model
  9. Step 3: Reinforcement Learning with PPO
  10. The alignment tax
  11. The results
  12. What alignment looks like today

The article lives on outcomeschool.com. Read it there, then come back: the tutor in the margin has read it and will answer questions, and the questions below check what stayed.

Before you go

In one sentence, what was this chapter about?

From memory, without scrolling up. Writing it is what makes it yours; the grade is only to show you what you had.

How sure?