The claim that how capable a system is and what it wants are independent. Being extremely good at achieving goals does not make a system’s goals wise or humane. It rules out the comfortable assumption that a sufficiently intelligent system would naturally arrive at good values.
Technical view
Bostrom’s thesis that any level of intelligence is compatible in principle with any final goal. The counterargument is that goals learned from human-generated data are not drawn from an arbitrary space, and that current systems acquire values and capabilities from the same training process rather than independently. The thesis is a claim about possibility, not about what training procedures actually produce.
Related
Part of AI Terminology Index, section: Alignment and Safety Concepts.