Jargonautica
Testing, Safeguarding and Running Models in Practice

Interpretability

The effort to work out what is actually going on inside a model, rather than treating it as a sealed box that produces answers. This matters because you cannot properly trust, debug or safeguard something whose reasoning you cannot inspect. It is one of the more scientifically interesting parts of the field and remains at an early stage relative to how capable the models have become.

Technical view

Mechanistic interpretability seeks to identify specific circuits and features within networks and explain their function. A central obstacle is superposition, where a network represents more features than it has dimensions by encoding them in overlapping directions, making individual units difficult to interpret. Sparse autoencoders have proved a productive approach for decomposing activations into more interpretable features. Simpler behavioural methods such as attribution and probing are more widely used in practice but tell you far less about mechanism.


Part of AI Terminology Index, section: Testing, Safeguarding and Running Models in Practice.

How it connects

Interpretability links to 30 other terms, and the 12 closest are drawn here. The faint lines are links between those terms. See it in the full graph.

Feature (Interpretability Sense)Feature (Interpreta…CircuitCircuitAlignmentAlignmentDeceptive AlignmentDeceptive AlignmentChain-of-thought FaithfulnessChain-of-thought Fa…Sparse AutoencoderSparse AutoencoderAttribution and SaliencyAttribution and Sal…ProbingProbingSuperpositionSuperpositionEmbeddingsEmbeddingsAttentionAttentionReward Hacking and Specification GamingReward Hacking and …Interpretability