Does a person rate every generated answer?

Not necessarily; a reward model can provide training scores.

Does RLHF eliminate hallucinations?

No. Preference improvement is not a guarantee of factual accuracy.

Is the reward model the assistant users interact with?

Usually the generating policy and reward model are separate components in the described pipeline.

AI Glossary / Reinforcement learning and robotics / Intermediate

Reinforcement Learning from Human Feedback

Definition

Reinforcement Learning from Human Feedback, commonly abbreviated RLHF, is a family of methods that uses human evaluations to guide reinforcement learning. In a common approach, people compare generated outputs or behavior, a reward model learns from those judgments, and a policy is optimized using that model's scores. The aim is to improve behavior according to the feedback criteria, not to certify universal correctness.

What it means

Some desired qualities are difficult to express as a simple automatic score. People may find it easier to compare two summaries, explanations, or movement demonstrations than to write a complete scoring function. RLHF turns such evaluations into training information.
The human feedback is not necessarily supplied during every interaction. A learned reward model can score many new examples after training on a smaller set of comparisons. Its predictions extend beyond the examples people actually evaluated, so errors in that generalization matter.

How it works

A common language-model workflow begins with supervised examples of desired responses. People then compare alternative responses to prompts. A reward model learns scores that help predict those preferences, and reinforcement learning adjusts the response-generating policy to obtain higher scores.
The policy and reward model have different jobs. One generates behavior; the other evaluates it. A reference-policy penalty can discourage the trained policy from moving too far from a starting distribution. Such a penalty changes the optimization objective but does not make the reward model infallible.
Human-preference learning also applies beyond text. People can compare short segments of simulated behavior, allowing a reward predictor to guide an agent without a fully hand-written task reward. The feedback collection process, comparison format, and learning algorithm vary by application.
A preference dataset records judgments made by particular people under particular instructions. Disagreement may reflect ambiguity, expertise, or different priorities. Evaluation should therefore test held-out tasks and inspect whether improved predicted scores correspond to improved outcomes.

Three practical examples

Summaries: use human comparisons to favor relevant, readable summaries under a defined evaluation rubric.
Instruction following: train a response policy using judgments about whether answers meet the requested task.
Simulated behavior: use preferences between movement clips to communicate qualities that are awkward to encode directly.

Think of it this way

Think of training a judging assistant from reviewed examples, then using its judgments to coach a performer. The assistant makes feedback more scalable, but the performer may discover ways to impress the assistant that the original reviewers would not approve. Independent review remains useful because the judge is a learned approximation.

Worked example

Consider a fictional training exercise with two possible responses, A and B. A reward model assigns scores 2.0 and 0.5. Under a simple pairwise logistic preference model, the predicted probability that A is preferred is 1/(1 + exp(-(2.0 - 0.5))), approximately 0.8176.
If a reviewer selects A, this comparison contributes a negative log-likelihood of -log(0.8176), approximately 0.2014, to the reward-model training loss. This is a predicted preference probability for this comparison model. It is not an 81.76% probability that every factual claim in A is correct.
Now hold the reward scores fixed. Suppose a reference response policy chooses A and B with probabilities 0.50 and 0.50. Its expected reward-model score is 0.50 times 2.0 + 0.50 times 0.5 = 1.25. A candidate policy choosing A with probability 0.80 and B with 0.20 has expected score 1.70.
The candidate has moved away from the reference. For this two-response distribution, its KL divergence from the reference is 0.80 log(0.80/0.50) + 0.20 log(0.20/0.50), approximately 0.1927 nats. With penalty coefficient 1, its simplified regularized objective is 1.70 - 0.1927 = 1.5073, compared with the reference's 1.25.
With coefficient 3, the candidate's objective becomes approximately 1.1218, below the reference. Thus the preferred tradeoff depends on the stated penalty strength. These calculations illustrate a small distribution-level objective; they are not a complete PPO update or a claim about an actual language model.
Finally, suppose reviewers discover that A contains an unsupported claim. The reward score's numerical improvement does not resolve that problem. The feedback data, evaluation criteria, or reward model may need revision before the policy is considered better.

Illustrative case study

Fictional case: CedarAnswers collects comparisons of customer-help drafts. Reviewers initially favor polished, detailed replies. Later evaluation finds that some highly rated drafts include details absent from the source material. The team adds explicit source-grounding checks to its review rubric and examines disagreements between stylistic preference and factual support.
It keeps a separate evaluation set and compares candidate outputs with the existing assistant before making a deployment decision. This invented case illustrates a feedback-design problem; it does not report measured benefits from RLHF.

Common misconceptions and mistakes

Common mistakes include treating human preference as objective truth, assuming a reward model directly stores human values, and believing every feedback-based training method uses the same reinforcement learning pipeline. High predicted reward can reflect weaknesses in the evaluator. More favorable tone, longer answers, or agreement with a user may receive good scores without improving task correctness.

Business use

Businesses can use RLHF when useful behavior is easier for reviewers to evaluate than for engineers to specify completely. The practical questions include who evaluates outputs, what rubric they follow, how disagreements are handled, and whether gains persist on independent work. Costs include collecting judgments, training and evaluating models, and investigating behavior that exploits an imperfect scoring process.

Check your understanding

What does 0.8176 describe?

Answer: The comparison model's predicted preference for A over B.

Why include a reference-policy penalty?

Answer: To trade predicted reward against departure from a reference distribution.

Why can a higher score still be undesirable?

Answer: The learned evaluator may miss a factual or task-related defect.

Related terms

Reinforcement learning
Supervised fine-tuning
Proximal policy optimization
Human preferences
reward model
KL divergence
preference optimization
reward hacking
evaluation rubric

Learn next

Next, examine how pairwise judgments train a reward model. Then compare reward-model evaluation with independent checks of factual support, task completion, and performance across different user groups.

Sources

Paul Christiano and colleagues, Deep Reinforcement Learning from Human Preferences: https://arxiv.org/abs/1706.03741 Long Ouyang and colleagues, Training Language Models to Follow Instructions with Human Feedback: https://arxiv.org/abs/2203.02155

Previous: Reinforcement learning Next: Reinforcement learning with verifiable rewards