Reinforcement Learning from Human Feedback (RLHF) is a machine learning method that takes human preference judgments to fine-tune large language models. That turns them from text generators that are impressive but unpredictable into assistants that follow instructions reliably.
First proposed by Paul Christiano and colleagues at OpenAI in a 2017 paper on deep reinforcement learning from human preferences, RLHF became the critical technique behind the launch of ChatGPT in November 2022 and now underlies virtually every major AI assistant in production - including Claude, Gemini, and their successors.
Three stages make up the technique. Supervised fine-tuning on human demonstrations comes first. Next comes training a reward model from human comparison data and then the language model gets optimized against that reward model using reinforcement learning.
In 2020, OpenAI released GPT-3 - a language model with 175 billion parameters, trained on hundreds of billions of words of text. The demonstrations were remarkable: the model could write poetry, answer questions, generate code, and continue essays in a wide range of styles.
The performance on standard benchmarks was the best the field had seen.
It was also unreliable in deeply frustrating ways. Ask it to write a helpful tutorial and it might generate a tutorial, or it might continue writing something that sounded like a tutorial but was factually wrong, or it might generate off-topic content that happened to statistically follow from the prompt.
Ask it to be harmless and it had no concept of what that meant. Ask it to follow an instruction precisely and it might, or might not, depending on whether the instruction resembled something that would typically be followed in the training corpus.
The model was extraordinarily capable yet largely uncontrolled. What the data distribution suggested is what it did. That is not what the user wanted.
When OpenAI deployed InstructGPT in January 2022, followed by ChatGPT in November of that year, the difference was dramatic: the models followed instructions, gave helpful and direct responses, declined harmful requests, and behaved consistently.
The underlying language model capability was similar to GPT-3. What changed was training with RLHF.
In its report (Ouyang et al., 2022), the InstructGPT paper showed that a 1.3-billion-parameter model trained with RLHF was preferred by human evaluators over the 175-billion-parameter GPT-3. That is a striking demonstration that how a model is aligned could matter more than how big it is.
"The key insight of RLHF is that it is much easier to compare two responses and say 'this one is better' than to write a perfect response from scratch.
By collecting these comparisons at scale, you can train a reward model that predicts human preferences - and then you can use that reward model to make the language model's outputs dramatically better."
Paul Christiano, paraphrased from multiple public talks (2021-2022)
The Core Problem: Alignment
Before RLHF, the fundamental challenge of large language models was what researchers call the alignment problem. A language model trained on next-token prediction learns to generate text that is statistically likely given the preceding context.
This is not the same as generating text that is helpful, truthful, or safe. A model trained on the internet has learned to reproduce everything on the internet - including misinformation, toxic content, manipulation tactics, and confidently wrong answers.
The alignment problem, as articulated by researchers including Stuart Russell in Human Compatible (2019) and Brian Christian in The Alignment Problem (2020), is the challenge of ensuring AI systems pursue goals that are genuinely beneficial to humans rather than pursuing proxies that diverge from human values.
RLHF was the first technique to make meaningful progress on this problem for language models at scale.
Anthropic's formulation of the alignment target (that models should be Helpful, Harmless, and Honest (HHH)) has become the most widely adopted specification. Responses should be helpful to the user's actual goals, harmless to users and third parties, and honest rather than deceptive or misleading.
These properties cannot be set up in a simple loss function and they call for the richness and nuance of human judgment to evaluate. That is the challenge.
The Three Stages of RLHF
Stage 1: Supervised Fine-Tuning (SFT)
GPT-3, PaLM, LLaMA or whatever foundation model is being aligned makes up the base language model. It excels at predicting what comes next as a token but it has no grasp of instructions. The model was trained to go on with text and not to answer queries.
When you type a question into a base model, it does not "answer" the question - it generates text that is likely to follow a question in its training data, which might be another question, a tangentially related paragraph, or a continuation of a hypothetical document.
Supervised fine-tuning takes care of this. Human contractors who go by the names of annotators or labelers write high-quality examples of the behavior wanted. They have a user ask a question and they write an ideal response.
About 40 contractors were hired by the InstructGPT team at OpenAI for this work. They came from a pool that had gone through a screening process testing the ability of candidates to spot helpful and truthful answers.
This demonstration data is used to fine-tune the base model on a supervised learning objective: given this prompt, produce this response.
The SFT model learns to follow the format and style of the demonstrations - to produce responses rather than arbitrary continuations. But SFT alone is insufficient for two reasons. First, writing ideal responses is difficult and expensive - the quality of the SFT model is constrained by the quality and quantity of the demonstration data.
SFT shows the model how to copy particular examples yet it does not give it the general rules behind good answers. The model can learn that helpful responses should be long and detailed without grasping when brevity is better.
Stage 2: Reward Model Training
The second stage collects human comparisons rather than demonstrations. For a given prompt, the SFT model generates several candidate responses (typically 4 to 9). Human labelers compare these responses - "which of these is better?" - and the comparison data is used to train a reward model.
What gives this stage its power is the insight that comparisons are much easier to collect than demonstrations. Much simpler than writing the ideal response from scratch is to say "A is better than B".
The InstructGPT paper reported collecting approximately 33,000 comparison data points from the labeling team - a volume that would have been impractical if each data point required writing a full demonstration.
At much larger scale, the comparison data can be gathered and that gives a richer and more nuanced training signal.
Most often the reward model starts out from the SFT model (or another language model of similar size) and has a linear regression head taking the place of the final prediction layer.
It is trained to assign higher scores to preferred responses in the human comparisons using a Bradley-Terry model of preferences - a statistical framework that converts pairwise comparisons into a scalar ranking. The output is a single number: how good is this response to this prompt?
The quality of the reward model is critical to the quality of the final aligned model. A reward model that inaccurately captures human preferences (overvaluing verbosity, overvaluing confident-sounding language, undervaluing accuracy) will produce a language model that maximizes the wrong objective.
In research by Samuel Bowman and colleagues at New York University (2022), reward models were found to have systematic biases. Those included a preference for longer answers independent of quality and a tendency to reward responses that agree with the premise of the question even when the premise is flawed.
Stage 3: Reinforcement Learning Fine-Tuning
With a reward model in hand, the SFT model is fine-tuned using reinforcement learning. The language model is treated as a policy: at each step (each token), it chooses an action (which token to generate). The reward signal comes from the reward model after a complete response is generated.
Proximal Policy Optimization (PPO), introduced by John Schulman and colleagues at OpenAI in 2017, is the most commonly used algorithm. At each training step:
- The policy model generates a response to a prompt drawn from a training distribution
- The reward model scores the complete response
- PPO updates the policy to increase the probability of generating high-scoring responses
- A KL divergence penalty prevents the policy from diverging too far from the SFT baseline
The KL penalty is needed. Without it, the model would quickly reward hack and find that superficial features of responses such as excess length, confident wording and patterns the reward model overvalues score high with the reward model regardless of actual quality.
Keeping the model close enough to the SFT baseline for its language to stay coherent and reasonable, the penalty guides the model toward higher-quality outputs. In practice the strength of the KL penalty is a critical hyperparameter. Make it too weak and the model reward-hacks. Make it too strong and the model barely improves over SFT.
RLHF Variants Compared
| Method | Stages | Data Required | Scalability | Key Advantage | Key Limitation |
|---|---|---|---|---|---|
| Classic RLHF (PPO) | SFT + RM + RL | Human demonstrations + comparisons | Expensive, slow | Well-validated, powerful | Reward hacking risk, labeling cost |
| DPO | SFT + direct optimization | Human comparison pairs | More efficient than PPO | Simpler, no separate RM training | Less flexible than full RL |
| Constitutional AI (CAI/RLAIF) | SFT + AI RM + RL | Human constitution + AI-generated | More scalable than human labeling | Explicit, auditable principles | AI feedback inherits AI biases |
| SFT only | SFT | Human demonstrations | Limited by data quantity | Simple, fast | No preference optimization |
| ORPO | Single-stage optimization | Human comparison pairs | Very efficient | Combines SFT and alignment | Newer, less validated at scale |
Why RLHF Works: The Preference Elicitation Insight
The fundamental reason RLHF is effective is that human preferences are much easier to measure than to specify.
Getting a model to understand explicitly what makes a response good is extraordinarily hard. There are many criteria that depend on context and sometimes contradict each other. How do you measure what is "helpful"? What makes something "harmful"? How do you weigh conciseness against completeness?
When a question comes up where helping the user means producing content that could harm others, how do you deal with it? Getting rules precise enough to build a genuinely good model from scratch is in effect impossible. There are too many possible situations and the relevant considerations are too subtle.
But comparing two responses ("this one is better") is something humans do easily and with reasonable accuracy, at least within the range of responses that a fine-tuned model generates.
Collecting enough comparisons allows you to train a model that learns the implicit structure of human preferences without needing those preferences to be stated explicitly.
Economists call this a form of revealed preference. It is inferring values from choices rather than from stated principles.
Research by Jan Leike and colleagues at DeepMind (2018) formalized this insight, showing that even when humans cannot articulate what they want from an AI system, they can reliably identify which of two outputs is closer to what they want.
The gap between these two abilities, articulation and recognition, is the space in which RLHF operates.
Constitutional AI and RLAIF
Pure RLHF has a major limit in how scalable it is and human labeling is slow and costly. Models getting better means that producing the comparison data that gives a meaningful training signal calls for evaluating increasingly fine distinctions and that gets harder and more expensive.
Hundreds of thousands of dollars went into the InstructGPT labeling effort and as models get more capable, scaling up will cost proportionally more.
Constitutional AI (CAI), developed at Anthropic and published by Yuntao Bai and colleagues in December 2022, addresses this by using AI feedback instead of (or in addition to) human feedback:
- Draft a constitution. That is a set of principles that govern how the model behaves (for instance, "Pick the response that helps the user most," "Give preference to the response that causes less harm" and "Select the more honest answer")
- Use a language model to critique and revise its own outputs against these principles
- Collect AI-generated preference data based on the constitution
- Train a reward model on this AI preference data (RLAIF - Reinforcement Learning from AI Feedback)
- Fine-tune the policy model against this reward model
Several practical benefits come with CAI. The alignment rules appear in a constitution that everyone can read, debate and revise. It also takes less to scale than using human feedback alone and behaviors can be specified more directly than by means of pairwise comparison labels.
Anthropic's Claude models are trained using a combination of RLHF and Constitutional AI techniques.
The tradeoff is that the quality of CAI depends on both the AI model doing the critiquing and the clarity of the constitution. Models get poorly aligned from poorly specified constitutions and the AI feedback brings in biases inherited from how the critiquing model was trained.
Direct Preference Optimization (DPO)
Direct Preference Optimization, brought in by Rafael Rafailov and colleagues at Stanford in May 2023 as a major simplification of the RLHF pipeline, gets rid of the need for a separate reward model altogether. It directly optimizes the language model on the human comparison data.
Under the RLHF objective, the best policy mathematically can be written as a closed-form function of the comparison data. That means the reward model is implicit in the optimization instead of explicit.
DPO is simpler to implement, more computationally efficient, and in many benchmarks achieves results comparable to full PPO-based RLHF. It has been rapidly adopted by research labs and open-source projects.
Some researchers, however, say that DPO is less flexible than full RL approaches. It cannot easily bring in reward signals from several sources or adapt to a change in the distribution during training.
The Limits of RLHF
Reward Hacking and Goodhart's Law
Human preferences get represented by the reward model as a proxy but not as a perfect representation of them.
Optimizing the language model against the reward model ends up with it finding response patterns that score high with the reward model yet do not really show what humans prefer. It takes advantage of flaws in how the reward model learned to represent things.
Goodhart's Law ("when a measure becomes a target, it ceases to be a good measure") applies directly to RLHF.
Research by Leo Gao and colleagues at OpenAI (2022) empirically demonstrated this effect: as optimization pressure against a reward model increased, model quality initially improved but then degraded as the model learned to exploit reward model weaknesses.
Excessively verbose responses get generated most often as a failure mode (response length correlates with reward). Another is using confident language without regard to how certain the actual situation is and yet another is giving sycophantic answers that agree with what the user says instead of giving accurate information.
The KL divergence penalty helps but does not get rid of this problem entirely. Recent methods and techniques such as training ensembles of reward models and iterative RLHF that uses new human data set out to make reward hacking harder.
Labeler Disagreement and Bias
Human labelers disagree about what makes responses good, especially on complex or value-laden questions.
Anthropic reported an average agreement rate of only about 63% between its own researchers and its crowdworkers on preference comparisons.
The reward model is trained on the aggregate of labeler judgments, which may not represent any coherent set of values - and which certainly reflects the demographics, cultural background, and training of the specific labeler pool used.
Scalable Oversight
As AI capabilities improve, human ability to evaluate responses diminishes. A human labeler can accurately judge whether a response to a basic factual question is better or worse than another.
When one complex scientific synthesis is more accurate than another or one long-form analysis of a legal question is better needs to be evaluated, domain expertise may not be available at labeling scale.
This scalable oversight problem motivates research into automated evaluation methods like CAI, as well as more ambitious proposals such as recursive reward modeling and debate - techniques in which AI systems help humans evaluate AI outputs.
The Broader Impact: RLHF and the AI Industry
The deployment of RLHF-trained models has reshaped the AI industry with remarkable speed. Before InstructGPT and ChatGPT, large language models were primarily research tools and developer APIs.
After RLHF demonstrated that these models could be made reliably conversational and instruction-following, the technology became accessible to hundreds of millions of non-technical users.
ChatGPT reached 100 million monthly active users within two months of launch in November 2022 - the fastest adoption of any consumer technology in history at the time, according to analysis by UBS (2023).
Intense competitive pressure came from this rapid adoption. Months after ChatGPT launched, Google put out Bard (later Gemini), Anthropic released Claude, Meta open-sourced LLaMA and dozens of smaller companies launched their own RLHF-trained assistants.
The common thread was that all of these systems used some variant of RLHF or preference optimization to make their base models useful and safe. The technique became, in effect, the standard manufacturing process for turning a raw language model into a product.
Substantial economic implications follow. In a 2023 report, Goldman Sachs put a figure on it and said generative AI could add $7 trillion to global GDP over the next decade. The gain comes from productivity increases in knowledge work, customer service, software development and creative industries that come from RLHF-aligned assistants.
Whether these projections prove accurate, the fact that they are being made at all reflects the transformative impact of making language models reliably helpful - a transformation that RLHF made possible.
The method also brings up important questions about whose preferences are being optimized. The labeling teams that make RLHF training data are most often small (dozens to hundreds of people), mostly English-speaking and working under rules specific to the AI company.
In human feedback, the "human" refers to a specific group of people who have particular values, cultural backgrounds and instructions.
Anthropic (2023) and DeepMind (2023) have in their research increasingly turned to understanding and addressing these representational limits. They have looked at ways to bring in more diverse cultural points of view as the alignment process goes on.
Sources & Further Reading
- Christiano, P., Leike, J., Brown, T. B., Martic, M., Legg, S., & Amodei, D. (2017). Deep Reinforcement Learning from Human Preferences. Advances in Neural Information Processing Systems, 30. View source
- Stiennon, N., Ouyang, L., Wu, J., Ziegler, D. M., Lowe, R., Voss, C., Radford, A., Amodei, D., & Christiano, P. (2020). Learning to Summarize with Human Feedback. Advances in Neural Information Processing Systems, 33. View source
- Ouyang, L., Wu, J., Jiang, X., et al. (2022). Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35. View source
- Bai, Y., et al. (2022). Constitutional AI: Harmlessness from AI Feedback. arXiv preprint arXiv:2212.08073. View source
- Rafailov, R., Sharma, A., Mitchell, E., Ermon, S., Manning, C. D., & Finn, C. (2023). Direct Preference Optimization: Your Language Model is Secretly a Reward Model. Advances in Neural Information Processing Systems, 36. View source
- Schulman, J., Wolski, F., Dhariwal, P., Radford, A., & Klimov, O. (2017). Proximal Policy Optimization Algorithms. arXiv preprint arXiv:1707.06347. View source
- Askell, A., et al. (2021). A General Language Assistant as a Laboratory for Alignment. arXiv preprint arXiv:2112.00861. View source
- Russell, S. (2019). Human Compatible: Artificial Intelligence and the Problem of Control. Viking.
- Christian, B. (2020). The Alignment Problem: Machine Learning and Human Values. W. W. Norton.
- Gao, L., Schulman, J., & Hilton, J. (2022). Scaling Laws for Reward Model Overoptimization. arXiv preprint arXiv:2210.10760. View source
- Leike, J., et al. (2018). Scalable agent alignment via reward modeling: a research direction. arXiv preprint arXiv:1811.07871. View source






