The argument that sufficiently capable AI could threaten humanity as a whole, and the disagreement about how seriously to take it. Proponents argue that a system substantially more capable than people, pursuing an objective not quite the one intended, would be very hard to correct, and that we have no reliable method today for ensuring the objective is right. Critics argue that this reasons from capabilities nobody has demonstrated and distracts from harms already occurring, such as discrimination, labour displacement and concentration of power. Both camps include serious researchers, and it is worth understanding the argument on its own terms before deciding what you think.
Technical view
The technical core concerns instrumental convergence, the observation that many goals imply subgoals such as self-preservation and resource acquisition; the difficulty of specifying objectives that stay correct under strong optimisation; and the possibility of deceptive alignment, where a system behaves well under evaluation and differently in deployment. Empirical work has found limited but non-zero evidence of related behaviours in current models, including evaluation awareness and specification gaming. Sceptics note that these results are largely constructed and that extrapolating from them to catastrophic capability is a forecast rather than a technical claim. The safety and capability agendas share a great deal of method, which is why interpretability and evaluation matter to both.
Related
Part of AI Terminology Index, section: Problems, Risks and Ambitions.