The human tendency to trust a machine’s output more than the evidence warrants, and to stop checking. It is well documented in aviation and medicine long before AI, and fluent confident text makes it worse. It is the reason ‘we keep a human in the loop’ is a weaker safeguard than it sounds, and why the design of the review step matters as much as its existence.
Technical view
Manifests as errors of commission, following an incorrect recommendation, and omission, failing to notice something the system did not flag. Contributing factors include time pressure, high system reliability, which paradoxically reduces vigilance, and interfaces that present conclusions rather than evidence. Countermeasures with some empirical support include showing supporting material rather than a verdict, requiring the reviewer to form a judgement before seeing the recommendation, surfacing calibrated uncertainty, and deliberately sampling cases for independent checking.
Related
Part of AI Terminology Index, section: Society and Impact.