Data poisoning is a poisoning attack in which an adversary controls part of the training data, as defined in the NIST glossary, which cites the NIST report AI 100-2 E2025.
That report, "Adversarial Machine Learning: A Taxonomy and Terminology of Attacks and Mitigations," places poisoning attacks in the training stage of 📝machine learning. In a data poisoning attack, NIST writes, "an adversary controls a subset of the training data by either inserting or modifying training samples." The report distinguishes this from model poisoning, in which "the adversary controls the model and its parameters," and states that "Data poisoning attacks are applicable to all learning paradigms."
NIST classifies poisoning attacks by the attacker's objective. According to the report, availability poisoning attacks "typically cause indiscriminate degradation of the ML model on all samples," while targeted and backdoor poisoning attacks "induce integrity violations on a small set of target samples." NIST describes backdoor poisoning attacks as requiring a backdoor pattern "which is added to both the poisoned samples and the testing samples to cause misclassification."
For 📝generative AI, NIST notes that foundation model developers commonly scrape data from a wide range of sources, and that "the scale of this data and the diversity of its sources provides a large potential attack surface." Beyond pre-training 📝training data, the report says data poisoning may also affect instruction tuning and reinforcement learning from human feedback. NIST records that the first known poisoning attack "was developed for worm signature generation in 2006."