The idea that someone affected by an automated decision should be told why. What that requires in practice is contested, and a model that is not interpretable cannot supply a mechanistic account. In practice explanation usually means describing what information was used, what the decision process was, and how to challenge it.
Technical view
The legal position rests on rights to meaningful information about the logic involved rather than a technical exposition. Practical approaches include documenting inputs and decision criteria, providing counterfactual explanations stating what would need to differ for a different outcome, and local attribution methods, all of which have known limitations regarding faithfulness. Counterfactual explanations are often the most useful and actionable form for the individual.
Related
Part of AI Terminology Index, section: Governance, Law and Regulation.