Actually reading the failures rather than looking at the score. Take fifty wrong answers, sort them into groups, and you will learn more about what to fix than from any aggregate number. It is the highest-value and most frequently skipped activity in applied machine learning.
Technical view
Sample failures stratified across input types, categorise by cause, and quantify each category’s share to prioritise work by expected gain. Distinguish model errors from data errors, label errors and specification ambiguity, since a meaningful fraction of apparent failures are cases where the reference answer is wrong or the task is underspecified. In pipelines, attribute failures to stages, since retrieval and parsing errors are commonly misattributed to the model.
Related
Part of AI Terminology Index, section: Measuring Performance.