When a model states something false with complete confidence. Invented citations, plausible but wrong figures, non-existent legal cases. This follows directly from how these systems work. They produce text that fits the pattern of a good answer, and a convincing wrong answer fits that pattern just as well as a right one. The model has no internal flag distinguishing what it knows from what it is confabulating, which is why anything consequential needs checking.
Technical view
The training objective rewards plausible continuations, not truthfulness, and there is no separate mechanism for verifying claims. Rates are worse for obscure topics where training data was sparse, and for specifics such as numbers, dates, names and references. Mitigations include retrieval grounding, requiring citations that can be checked, and training models to express uncertainty, but no method eliminates the problem. Some researchers argue it is intrinsic to the approach rather than a bug awaiting a fix.
Related
Part of AI Terminology Index, section: Problems, Risks and Ambitions.