An approach to safety that asks how to use a powerful AI system safely even if it might be secretly working against you, rather than relying only on making it trustworthy. It uses monitoring, restrictions and checks, often with a weaker but trusted model watching a stronger one, and tests those defences by having a team play the part of a scheming model trying to get past them.
Technical view
Formalised by Greenblatt, Shlegeris, Sachan and Roger of Redwood Research in a December 2023 paper. An untrusted capable model does the work, constrained by a less capable trusted model and a small budget of trusted human review. Protocols include trusted monitoring, where the weak model flags suspicious outputs for audit; trusted editing, where it rewrites them; and untrusted monitoring, where another instance of the strong model checks the work under measures that prevent collusion. Safety is estimated through control evaluations, in which a red team designs the attack strategy the untrusted model would follow. It complements alignment, and proponents accept it gets harder as models outstrip their monitors.
Related
Part of AI Terminology Index, section: Alignment and Safety Concepts.