The reinforcement learning algorithm behind the first generation of assistant training. It improves the model against a reward signal while stopping it from changing too much in any single step, which keeps training from collapsing. It works and it is fiddly, which is why simpler alternatives have taken over much of its ground.
Technical view
A policy gradient method using a clipped surrogate objective to bound the update size, with a value network for advantage estimation and a Kullback-Leibler penalty against the reference policy. In language model training it requires holding policy, reference, reward and value models simultaneously, making it memory-hungry and sensitive to hyperparameters. It remains standard where an online reward signal exists, particularly for verifiable tasks.
Related
Part of AI Terminology Index, section: Training and Post-training.