skip to main content
10.1145/3128572.3140451acmconferencesArticle/Chapter ViewAbstractPublication PagesccsConference Proceedingsconference-collections
research-article

Towards Poisoning of Deep Learning Algorithms with Back-gradient Optimization

Published: 03 November 2017 Publication History

Abstract

A number of online services nowadays rely upon machine learning to extract valuable information from data collected in the wild. This exposes learning algorithms to the threat of data poisoning, i.e., a coordinate attack in which a fraction of the training data is controlled by the attacker and manipulated to subvert the learning process. To date, these attacks have been devised only against a limited class of binary learning algorithms, due to the inherent complexity of the gradient-based procedure used to optimize the poisoning points (a.k.a. adversarial training examples). In this work, we first extend the definition of poisoning attacks to multiclass problems. We then propose a novel poisoning algorithm based on the idea of back-gradient optimization, i.e., to compute the gradient of interest through automatic differentiation, while also reversing the learning procedure to drastically reduce the attack complexity. Compared to current poisoning strategies, our approach is able to target a wider class of learning algorithms, trained with gradient-based procedures, including neural networks and deep learning architectures. We empirically evaluate its effectiveness on several application examples, including spam filtering, malware detection, and handwritten digit recognition. We finally show that, similarly to adversarial test examples, adversarial training examples can also be transferred across different learning algorithms.

References

[1]
Marco Barreno, Blaine Nelson, Anthony Joseph, and J. Tygar. 2010. The security of machine learning. Machine Learning 81 (2010), 121--148. Issue 2.
[2]
Marco Barreno, Blaine Nelson, Russell Sears, Anthony D. Joseph, and J. D. Tygar. 2006. Can machine learning be secure? In Proc. ACM Symp. Information, Computer and Comm. Sec. (ASIACCS '06). ACM, New York, NY, USA, 16--25.
[3]
Y. Bengio. 2000. Gradient-based optimization of hyperparameters. Neural Computation 12, 8 (2000), 1889--1900.
[4]
Battista Biggio, Samuel Rota Bulò, Ignazio Pillai, Michele Mura, Eyasu Zemene Mequanint, Marcello Pelillo, and Fabio Roli. 2014. Poisoning complete-linkage hierarchical clustering. In Joint IAPR Int'l Workshop on Structural, Syntactic, and Statistical Pattern Recognition (Lecture Notes in Computer Science), P. Franti, G. Brown, M. Loog, F. Escolano, and M. Pelillo (Eds.), Vol. 8621. Springer Berlin Heidelberg, Joensuu, Finland, 42--52.
[5]
Battista Biggio, Igino Corona, Giorgio Fumera, Giorgio Giacinto, and Fabio Roli. 2011. Bagging Classifiers for Fighting Poisoning Attacks in Adversarial Classification Tasks. In 10th International Workshop on Multiple Classifier Systems (MCS) (Lecture Notes in Computer Science), Carlo Sansone, Josef Kittler, and Fabio Roli (Eds.), Vol. 6713. Springer-Verlag, 350--359.
[6]
B. Biggio, I. Corona, D. Maiorca, B. Nelson, N. Šrndi, P. Laskov, G. Giacinto, and F. Roli. 2013. Evasion attacks against machine learning at test time. In Machine Learning and Knowledge Discovery in Databases (ECML PKDD), Part III (LNCS), Hendrik Blockeel, Kristian Kersting, Siegfried Nijssen, and Filip Železný (Eds.), Vol. 8190. Springer Berlin Heidelberg, 387--402.
[7]
Battista Biggio, Giorgio Fumera, and Fabio Roli. 2014. Security Evaluation of Pattern Classifiers Under Attack. IEEE Transactions on Knowledge and Data Engineering 26, 4 (April 2014), 984--996.
[8]
Battista Biggio, Blaine Nelson, and Pavel Laskov. 2012. Poisoning attacks against support vector machines, In 29th Int'l Conf. on Machine Learning, John Langford and Joelle Pineau (Eds.). Int'l Conf. on Machine Learning (ICML), 1807--1814.
[9]
Battista Biggio, Ignazio Pillai, Samuel Rota Bulò, Davide Ariu, Marcello Pelillo, and Fabio Roli. 2013. Is Data Clustering in Adversarial Settings Secure? In Proceedings of the 2013 ACM Workshop on Artificial Intelligence and Security (AISec '13). ACM, New York, NY, USA, 87--98.
[10]
Battista Biggio, Konrad Rieck, Davide Ariu, Christian Wressnegger, Igino Corona, Giorgio Giacinto, and Fabio Roli. 2014. Poisoning Behavioral Malware Clustering. In 2014 Workshop on Artificial Intelligent and Security (AISec '14). ACM, New York, NY, USA, 27--36.
[11]
C. Blake and C. J. Merz. 1998. UCI Repository of machine learning databases. http://www.ics.uci.edu/~mlearn/MLRepository.html (1998).
[12]
Nader H. Bshouty, Nadav Eiron, and Eyal Kushilevitz. 1999. PAC Learning with Nasty Noise. In Algorithmic Learning Theory, Osamu Watanabe and Takashi Yokomori (Eds.). Lecture Notes in Computer Science, Vol. 1720. Springer Berlin Heidelberg, 206--218. https://doi.org/10.1007/3-540-46769-6_17
[13]
C. Do, C. S. Foo, and A. Y. Ng. 2008. Efficient multiple hyperparameter learning for log-linear models. In Advances in Neural Information Processing Systems. 377--384.
[14]
Justin Domke. 2012. Generic Methods for Optimization-Based Modeling. In 15th Int'l Conf. Artificial Intelligence and Statistics (Proceedings of Machine Learning Research), Neil D. Lawrence and Mark Girolami (Eds.), Vol. 22. PMLR, La Palma, Canary Islands, 318--326.
[15]
Ian J. Goodfellow, Jonathon Shlens, and Christian Szegedy. 2015. Explaining and Harnessing Adversarial Examples. In International Conference on Learning Representations.
[16]
L. Huang, A. D. Joseph, B. Nelson, B. Rubinstein, and J. D. Tygar. 2011. Adversarial Machine Learning. In 4th ACM Workshop on Artificial Intelligence and Security (AISec 2011). Chicago, IL, USA, 43--57.
[17]
Anthony D. Joseph, Pavel Laskov, Fabio Roli, J. Doug Tygar, and Blaine Nelson. 2013. Machine Learning Methods for Computer Security (Dagstuhl Perspectives Workshop 12371). Dagstuhl Manifestos 3, 1 (2013), 1--30.
[18]
Michael Kearns and Ming Li. 1993. Learning in the presence of malicious errors. SIAM J. Comput. 22, 4 (1993), 807--837. https://doi.org/10.1137/0222052
[19]
Marius Kloft and Pavel Laskov. 2012. Security Analysis of Online Centroid Anomaly Detection. Journal of Machine Learning Research 13 (2012), 3647--3690.
[20]
P. W. Koh and P. Liang. 2017. Understanding Black-box Predictions via Influence Functions. In International Conference on Machine Learning (ICML).
[21]
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. 1998. GradientBased Learning Applied to Document Recognition. In Proceedings of the IEEE, Vol. 86. 2278--2324.
[22]
Dougal Maclaurin, David Duvenaud, and Ryan P. Adams. 2015. Gradient-based Hyperparameter Optimization Through Reversible Learning. In Proceedings of the 32Nd International Conference on International Conference on Machine Learning - Volume 37 (ICML'15). JMLR.org, 2113--2122.
[23]
Shike Mei and Xiaojin Zhu. 2015. Using Machine Teaching to Identify Optimal Training-Set Attacks on Machine Learners. In 29th AAAI Conf. Artificial Intelligence (AAAI '15).
[24]
Seyed-Mohsen, Moosavi-Dezfooli, Alhussein Fawzi, Omar Fawzi, and Pascal Frossard. 2017. Universal adversarial perturbations. In CVPR.
[25]
B. Nelson, M. Barreno, F. J. Chi, A. D. Joseph, B. I. P. Rubinstein, U. Saini, C.A. Sutton, J. D. Tygar, and K. Xia. 2008. Exploiting Machine Learning to Subvert your Spam Filter. LEET 8 (2008), 1--9.
[26]
Blaine Nelson, Marco Barreno, Fuching Jack Chi, Anthony D. Joseph, Benjamin I. P. Rubinstein, Udam Saini, Charles Sutton, J. D. Tygar, and Kai Xia. 2008. Exploiting machine learning to subvert your spam filter. In LEET'08: Proceedings of the 1st Usenix Workshop on Large-Scale Exploits and Emergent Threats. USENIX Association, Berkeley, CA, USA, 1--9.
[27]
Nicolas Papernot, Patrick McDaniel, Ian Goodfellow, Somesh Jha, Z. Berkay Celik, and Ananthram Swami. 2017. Practical Black-Box Attacks Against Machine Learning. In Proceedings of the 2017 ACM on Asia Conference on Computer and Communications Security (ASIA CCS '17). ACM, New York, NY, USA, 506--519.
[28]
Nicolas Papernot, Patrick McDaniel, Somesh Jha, Matt Fredrikson, Z. Berkay Celik, and Ananthram Swami. 2016. The Limitations of Deep Learning in Adversarial Settings. In Proc. 1st IEEE European Symposium on Security and Privacy. IEEE, 372--387.
[29]
K. R. Patil, X. Zhu, L. Kope?, and B. C. Love. 2014. Optimal teaching for limitedcapacity human learners. In Advances in Neural Information Processing Systems. 2465--2473.
[30]
B. A. Pearlmutter. 1994. Fast Exact Multiplication by the Hessian. Neural Computation 6, 1 (1994), 147--160.
[31]
F. Pedregosa. 2016. Hyperparameter optimization with approximate gradient. In 33rd International Conference on Machine Learning (Proceedings of Machine Learning Research), Maria Florina Balcan and Kilian Q. Weinberger (Eds.), Vol. 48. PMLR, New York, New York, USA, 737--746.
[32]
Benjamin I. P. Rubinstein, Blaine Nelson, Ling Huang, Anthony D. Joseph, Shinghon Lau, Satish Rao, Nina Taft, and J. D. Tygar. 2009. ANTIDOTE: understanding and defending against poisoning of anomaly detectors. In Proceedings of the 9th ACM SIGCOMM Internet Measurement Conference (IMC '09). ACM, New York, NY, USA, 1--14.
[33]
D. Sgandurra, L. Muñoz-González, R. Mohsen, and E. C. Lupu. 2016. Automated Dynamic Analysis of Ransomware: Benefits, Limitations and use for Detection. arXiv preprint arXiv:1609.03020 (2016).
[34]
Charles Smutz and Angelos Stavrou. 2012. Malicious PDF Detection Using Metadata and Structural Features. In Proceedings of the 28th Annual Computer Security Applications Conference (ACSAC '12). ACM, New York, NY, USA, 239--248.
[35]
J. Steinhardt, P. W. Koh, and P. Liang. 2017. Certified Defenses for Data Poisoning Attacks. arXiv preprint arXiv:1706.03691 (2017).
[36]
Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. 2014. Intriguing properties of neural networks. In International Conference on Learning Representations. http://arxiv.org/abs/1312.6199
[37]
Nedim Šrndic and Pavel Laskov. 2014. Practical Evasion of a Learning-Based Classifier: A Case Study. In Proc. 2014 IEEE Symp. Security and Privacy (SP '14). IEEE CS, Washington, DC, USA, 197--211.
[38]
Gang Wang, Tianyi Wang, Haitao Zheng, and Ben Y. Zhao. 2014. Man vs. Machine: Practical Adversarial Detection of Malicious Crowdsourcing Workers. In 23rd USENIX Security Symposium (USENIX Security 14). USENIX Association, San Diego, CA.
[39]
Huang Xiao, Battista Biggio, Gavin Brown, Giorgio Fumera, Claudia Eckert, and Fabio Roli. 2015. Is Feature Selection Secure against Training Data Poisoning? In JMLR W&CP - Proc. 32nd Int'l Conf. Mach. Learning (ICML), Francis Bach and David Blei (Eds.), Vol. 37. 1689--1698.
[40]
X. Zhu. 2013. Machine Teaching for Bayesian Learners in the Exponential Family. In Advances in Neural Information Processing Systems. 1905--1913.

Cited By

View all
  • (2025)Trustworthy Distributed AI Systems: Robustness, Privacy, and GovernanceACM Computing Surveys10.1145/364510257:6(1-42)Online publication date: 10-Feb-2025
  • (2025)A survey on Deep Learning in Edge-Cloud Collaboration: Model partitioning, privacy preservation, and prospectsKnowledge-Based Systems10.1016/j.knosys.2025.112965(112965)Online publication date: Jan-2025
  • (2025)Adversarial machine learning threat analysis and remediation in Open Radio Access Network (O-RAN)Journal of Network and Computer Applications10.1016/j.jnca.2024.104090236(104090)Online publication date: Apr-2025
  • Show More Cited By

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences
AISec '17: Proceedings of the 10th ACM Workshop on Artificial Intelligence and Security
November 2017
140 pages
ISBN:9781450352024
DOI:10.1145/3128572
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 03 November 2017

Permissions

Request permissions for this article.

Check for updates

Author Tags

  1. adversarial examples
  2. adversarial machine learning
  3. deep learning
  4. training data poisoning

Qualifiers

  • Research-article

Funding Sources

  • UK EPSRC

Conference

CCS '17
Sponsor:

Acceptance Rates

AISec '17 Paper Acceptance Rate 11 of 36 submissions, 31%;
Overall Acceptance Rate 94 of 231 submissions, 41%

Upcoming Conference

CCS '25

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)237
  • Downloads (Last 6 weeks)17
Reflects downloads up to 16 Feb 2025

Other Metrics

Citations

Cited By

View all
  • (2025)Trustworthy Distributed AI Systems: Robustness, Privacy, and GovernanceACM Computing Surveys10.1145/364510257:6(1-42)Online publication date: 10-Feb-2025
  • (2025)A survey on Deep Learning in Edge-Cloud Collaboration: Model partitioning, privacy preservation, and prospectsKnowledge-Based Systems10.1016/j.knosys.2025.112965(112965)Online publication date: Jan-2025
  • (2025)Adversarial machine learning threat analysis and remediation in Open Radio Access Network (O-RAN)Journal of Network and Computer Applications10.1016/j.jnca.2024.104090236(104090)Online publication date: Apr-2025
  • (2025)Workplace security and privacy implications in the GenAI age: A surveyJournal of Information Security and Applications10.1016/j.jisa.2024.10396089(103960)Online publication date: Mar-2025
  • (2025)Practical clean-label backdoor attack against static malware detectionComputers & Security10.1016/j.cose.2024.104280150(104280)Online publication date: Mar-2025
  • (2024)Improving SAM requires rethinking its optimization formulationProceedings of the 41st International Conference on Machine Learning10.5555/3692070.3694310(54475-54487)Online publication date: 21-Jul-2024
  • (2024)Disguised copyright infringement of latent diffusion modelsProceedings of the 41st International Conference on Machine Learning10.5555/3692070.3693418(33196-33218)Online publication date: 21-Jul-2024
  • (2024)Nonsmooth implicit differentiationProceedings of the 41st International Conference on Machine Learning10.5555/3692070.3692718(16250-16274)Online publication date: 21-Jul-2024
  • (2024)A Holistic Review of Machine Learning Adversarial Attacks in IoT NetworksFuture Internet10.3390/fi1601003216:1(32)Online publication date: 19-Jan-2024
  • (2024)Federated Learning in Dynamic and Heterogeneous Environments: Advantages, Performances, and Privacy ProblemsApplied Sciences10.3390/app1418849014:18(8490)Online publication date: 20-Sep-2024
  • Show More Cited By

View Options

Login options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Figures

Tables

Media

Share

Share

Share this Publication link

Share on social media