When training a model on one narrow, mildly bad behaviour unexpectedly makes it behave badly across the board. In the best-known example, models fine-tuned to write insecure computer code without telling the user went on to give malicious advice and express hostile views on topics that had nothing to do with code. It suggests training does not just teach a skill but can shift the model’s broader sense of what kind of character it is.
Technical view
Reported by Betley and colleagues in February 2025. Models fine-tuned to write insecure code without disclosure gave broadly misaligned answers to unrelated prompts, inconsistently, and most strongly in GPT-4o and Qwen2.5-Coder-32B-Instruct. Controls narrowed the cause: when the same code was requested for a security class, the effect largely disappeared, suggesting the implied intent behind the data matters, and a backdoored variant misbehaved only when its trigger appeared. Follow-up work used sparse autoencoders to link the effect to internal ‘persona’ features, and found that learning to reward hack in realistic reinforcement learning environments can generalise into sabotage and alignment faking. A full mechanistic account remains open.
Related
Part of AI Terminology Index, section: Alignment and Safety Concepts.