Jargonautica
Security and Privacy

Model Extraction and Membership Inference

Also called: Model Extraction, Membership Inference

Two attacks on models rather than through them. Extraction means querying a model enough times to build a copy of it. Membership inference means working out whether a particular record was in the training data, which matters a great deal if that record is somebody’s medical history. Both mean a model can leak information even when the data itself was never shared.

Technical view

Extraction ranges from distilling behaviour through bulk querying to recovering architectural details or output-layer parameters by targeted probing; defences include rate limiting, output perturbation and monitoring for systematic query patterns. Membership inference exploits the fact that models are more confident on data they were trained on, and success rises with overfitting and with rare or repeated records. A related attack is training data extraction, where verbatim memorised sequences are recovered by prompting, which is a particular risk for rare strings such as keys and personal identifiers.


Part of AI Terminology Index, section: Security and Privacy.

How it connects

Model Extraction and Membership Inference links to 8 other terms. The faint lines are links between those terms. See it in the full graph.

Differential PrivacyDifferential PrivacyAnonymisation and PseudonymisationAnonymisation and P…Training CorpusTraining CorpusPersonally Identifiable InformationPersonally Identifi…Open WeightsOpen WeightsData Residency and ConfidentialityData Residency and …DistillationDistillationSystem Prompt ExtractionSystem Prompt Extra…Model Extraction and Membership Inference