A system that improves itself, then uses the improved version to improve itself further. The argument is that this could compound quickly. In practice today the loop is partial: models help write code and design experiments, with people in every iteration and physical constraints on compute and data.
Technical view
The classical argument assumes rapidly diminishing effort per capability increment; sceptics point to compounding difficulty, dependence on scarce compute, energy and data, and the empirical observation that capability gains have required exponentially increasing inputs. The concrete near-term version is automation of AI research itself, which is measurable through benchmarks on machine learning engineering tasks and is watched precisely because it would shorten the loop.
Related
Part of AI Terminology Index, section: Alignment and Safety Concepts.