Jargonautica
Training and Post-training

Reward Model

A separate model trained to predict which of two answers a person would prefer. Once you have one, you can score any output automatically, which is what makes preference-based training possible at scale. Its quality caps everything downstream, because the main model will learn to satisfy this judge rather than the humans it stands in for.

Technical view

Trained on pairwise human preference data, usually initialised from the policy model with a scalar output head, optimised so preferred responses score higher. Known failure modes include length bias, style over substance, and poor generalisation off-distribution. Over-optimising against it produces reward hacking, which is why divergence from the reference policy is penalised and why reward model quality is monitored throughout.

Part of AI Terminology Index, section: Training and Post-training.

How it connects

Reward Model links to 11 other terms. The faint lines are links between those terms. See it in the full graph.

Reinforcement Learning from Human FeedbackReinforcement Learn…Preference DataPreference DataData Labelling and AnnotationData Labelling and …Reward Hacking and Specification GamingReward Hacking and …SycophancySycophancyDirect Preference OptimisationDirect Preference O…Proximal Policy OptimisationProximal Policy Opt…Reinforcement Learning from AI FeedbackReinforcement Learn…Reinforcement Learning with Verifiable RewardsReinforcement Learn…Group Relative Policy OptimisationGroup Relative Poli…Process Reward ModelProcess Reward ModelReward Model