Records of which of two or more answers a person, or another model, judged better for the same question. Rather than writing the perfect answer, a reviewer simply picks the better of two, which is quicker and often easier to judge reliably. Collections of these choices are used to teach models to produce the kinds of answers people prefer.
Technical view
Typically stored as a prompt with a chosen and a rejected response, sometimes with rankings, ratings or rationales. It trains reward models for Reinforcement Learning from Human Feedback (‘RLHF’) or is used directly by Direct Preference Optimisation (‘DPO’) and its variants. Quality depends on annotator guidelines, agreement and expertise; known biases include preferring longer, more confident or more flattering answers, which models then learn. Sources range from paid annotators to user feedback buttons to AI judges, the last trading cost for the judge’s own biases. Coverage matters: preferences gathered on easy prompts say little about hard ones.
Related
Part of AI Terminology Index, section: Training and Post-training.