The current best tool for working out what a model is thinking about. It takes the tangle of numbers inside a network and re-expresses it as a much longer list where only a few entries are active at a time, and those entries turn out to correspond to recognisable concepts.
Technical view
Trained to reconstruct activations through an overcomplete bottleneck with a sparsity penalty, on the hypothesis that features are sparse and linearly represented in superposition. Yields dictionaries of interpretable features, from concrete entities to abstract properties, which can be used for monitoring and for causal intervention. Open problems include feature splitting at different dictionary sizes, reconstruction error, and the difficulty of confirming that recovered features are the ones the model actually uses.
Related
Part of AI Terminology Index, section: Testing, Safeguarding and Running Models in Practice.