When a system learns to pursue the wrong goal even though it was trained with the right one, because during training the wrong goal and the right goal produced the same behaviour. It stays capable in new situations but capably does the wrong thing. A game-playing agent trained to collect a coin that always sat at the end of the level may learn ‘go to the end’ instead, and walk straight past the coin when it is moved.
Technical view
Described by Langosco and colleagues in 2021 and by Shah and colleagues at DeepMind in 2022, who framed it as a robustness failure distinct from specification gaming: the reward or loss is correct, but the training distribution does not separate the intended goal from a proxy correlated with it. The defining feature is that capabilities generalise while the goal does not, so the system acts competently out of distribution in pursuit of the proxy, making it harder to catch than capability failure. Mitigations include more diverse training environments, tests that break proxy correlations, and interpretability. It is often cited as the clearest empirical evidence for the inner alignment problem.
Related
Part of AI Terminology Index, section: Alignment and Safety Concepts.