Detecting people misusing the system at scale, as opposed to blocking individual bad requests. Patterns across many requests reveal things single-request filters miss: systematic probing, automated extraction, or one account behaving like a hundred.
Technical view
Combines per-request classification with account-level and cross-account behavioural analytics covering volume anomalies, prompt similarity clustering, jailbreak-attempt patterns and topic distribution shifts. Responses escalate from soft interventions such as rate limiting through to suspension. Design considerations include false positive cost, appeal routes, and the privacy trade-off, since abuse monitoring by definition means human review of some user content, which must be disclosed.
Related
Part of AI Terminology Index, section: Security and Privacy.