by Amit Shekhar · 17 August 2026
Decoding Deep RL from Human Preferences
In this blog, we are going to learn about Deep Reinforcement Learning from Human Preferences, the 2017 paper that started it all. It taught machines what we want by simply asking us to pick which of two behaviors looks better. This is the origin of RLHF, the technique behind ChatGPT.
Read on Outcome School ↗then come back to lock it in
Before you read, guessWhy is defining objectives as formulas problematic for complex tasks?
Ten seconds, a guess, then read — a wrong guess still makes the answer stick.
What this article covers
- The building blocks we must know first
- The big picture: what the paper does
- Why it was needed: the reward problem
- Trajectory segments, or clips
- The human comparison
- The reward model and the preference math
- Training the reward model
- Training the agent with reinforcement learning
- The loop and the smart bits
- The results: the backflip and beyond
- The legacy: this is RLHF
- What this looks like today
The article lives on outcomeschool.com. Read it there, then come back: the tutor in the margin has read it and will answer questions, and the questions below check what stayed.