Searle’s thought experiment: a person in a room follows a rulebook to answer questions in Chinese without knowing any Chinese, so the argument goes, a system manipulating symbols by rules cannot understand them however convincing the output. It is the standard philosophical challenge to the idea that behaviour is sufficient for mind.
Technical view
Directed at symbolic AI in 1980, and routinely applied to language models today. The main replies are the systems reply, that understanding belongs to the whole system rather than the person inside it, and the robot reply, that grounding through interaction with the world supplies what is missing. Its relevance to modern learned systems is contested, since a learned distributed representation is not obviously the rulebook the argument describes.
Related
Part of AI Terminology Index, section: Alignment and Safety Concepts.