Cited By
View all- Swacha JGracel M(2023)Machine Learning in Gamification and Gamification in Machine Learning: A Systematic Literature MappingApplied Sciences10.3390/app13201142713:20(11427)Online publication date: 18-Oct-2023
The Design and Development of a Game to Study Backdoor Poisoning Attacks: The Backdoor Game
Authors: 
Zahra Ashktorab, Casey Dugan, James Johnson, Aabhas Sharma, Dustin Ramsey Torres, Ingrid Lange, Benjamin Hoover, Heiko Ludwig, Bryant Chen, Nathalie Baracaldo, 
Werner Geyer, Qian PanAuthors Info & Claims
Zahra Ashktorab, Casey Dugan, James Johnson, Aabhas Sharma, Dustin Ramsey Torres, Ingrid Lange, Benjamin Hoover, Heiko Ludwig, Bryant Chen, Nathalie Baracaldo, Werner Geyer, Qian PanAuthors Info & ClaimsPages 423 - 433
Abstract
AI Security researchers have identified a new way crowdsourced data can be intentionally compromised. Backdoor attacks are a process through which an adversary creates a vulnerability in a machine learning model by ?poisoning?’ the training set by selectively mislabelling images containing a backdoor object. The model continues to perform well on standard testing data but misclassifies on the inputs that contain the backdoor chosen by the adversary. In this paper, we present the design and development of the Backdoor Game, the first game in which users can interact with different poisoned classifiers and upload their own images containing backdoor objects in an engaging way. We conduct semi-structured interviews with eight different participants who interacted with a first version of the Backdoor Game and deploy the game to Mechanical Turk users (N=68) to demonstrate how users interacted with the backdoor objects. We present results including novel types of interactions that emerged as a result of game play and design recommendations for the improvement of the system. The combined design, development and deployment of our system can help AI Security researchers to study this emerging concept, from determining the effectiveness of different backdoor objects to help compiling a collection of diverse and unique backdoor objects from the public, increasing the safety of future AI systems.
References
[1]
Martín Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Geoffrey Irving, Michael Isard, 2016. Tensorflow: a system for large-scale machine learning. In OSDI, Vol. 16. 265–283.
[2]
Josh M Attenberg, Pagagiotis G Ipeirotis, and Foster Provost. 2011. Beat the machine: Challenging workers to find the unknown unknowns. In Workshops at the Twenty-Fifth AAAI Conference on Artificial Intelligence.
[3]
Nathalie Baracaldo, Bryant Chen, Heiko Ludwig, and Jaehoon Amir Safavi. 2017. Mitigating poisoning attacks on machine learning models: A data provenance based approach. In Proceedings of the 10th ACM Workshop on Artificial Intelligence and Security. 103–110.
[4]
Michael S Bernstein. 2010. Crowd-powered interfaces. In Adjunct proceedings of the 23nd annual ACM symposium on User interface software and technology. 347–350.
[5]
Jonathan Bragg and Daniel S Weld. 2016. Optimal testing for crowd workers. In Proceedings of the 2016 International Conference on Autonomous Agents & Multiagent Systems. 966–974.
[6]
LJ Cao, Kok Seng Chua, WK Chong, HP Lee, and QM Gu. 2003. A comparison of PCA, KPCA and ICA for dimensionality reduction in support vector machine. Neurocomputing 55, 1-2 (2003), 321–336.
[7]
Dennis Chao. 2001. Doom as an interface for process management. In Proceedings of the SIGCHI conference on Human factors in computing systems. ACM, 152–157.
[8]
Bryant Chen, Wilka Carvalho, Nathalie Baracaldo, Heiko Ludwig, Benjamin Edwards, Taesung Lee, Ian Molloy, and Biplav Srivastava. 2018. Detecting Backdoor Attacks on Deep Neural Networks by Activation Clustering. arXiv preprint arXiv:1811.03728(2018).
[9]
Xinyun Chen, Chang Liu, Bo Li, Kimberly Lu, and Dawn Song. 2017. Targeted backdoor attacks on deep learning systems using data poisoning. arXiv preprint arXiv:1712.05526(2017).
[10]
Lydia B Chilton, Greg Little, Darren Edge, Daniel S Weld, and James A Landay. 2013. Cascade: Crowdsourcing taxonomy creation. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. 1999–2008.
[11]
Otto Chrons and Sami Sundell. 2011. Digitalkoot: Making old archives accessible using crowdsourcing. In Workshops at the Twenty-Fifth AAAI Conference on Artificial Intelligence.
[12]
Seth Cooper, Firas Khatib, Adrien Treuille, Janos Barbero, Jeehyung Lee, Michael Beenen, Andrew Leaver-Fay, David Baker, Zoran Popović, 2010. Predicting protein structures with a multiplayer online game. Nature 466, 7307 (2010), 756.
[13]
Ritendra Datta, Weina Ge, Jia Li, and James Z Wang. 2007. Toward bridging the annotation-retrieval gap in image search. IEEE MultiMedia 14, 3 (2007).
[14]
Ritendra Datta, Dhiraj Joshi, Jia Li, and James Z Wang. 2008. Image retrieval: Ideas, influences, and trends of the new age. ACM Computing Surveys (Csur) 40, 2 (2008), 5.
[15]
Sebastian Deterding. 2012. Gamification: designing for motivation. interactions 19, 4 (2012), 14–17.
[16]
Steven Dow, Anand Kulkarni, Brie Bunge, Truc Nguyen, Scott Klemmer, and Björn Hartmann. 2011. Shepherding the crowd: managing and providing feedback to crowd workers. In CHI’11 Extended Abstracts on Human Factors in Computing Systems. 1669–1674.
[17]
Kevin Eykholt, Ivan Evtimov, Earlence Fernandes, Bo Li, Amir Rahmati, Chaowei Xiao, Atul Prakash, Tadayoshi Kohno, and Dawn Song. 2018. Robust physical-world attacks on deep learning visual classification. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 1625–1634.
[18]
Andrej Gisbrecht, Alexander Schulz, and Barbara Hammer. 2015. Parametric nonlinear dimensionality reduction using kernel t-SNE. Neurocomputing 147(2015), 71–82.
[19]
Leo A Goodman. 1961. Snowball sampling. The annals of mathematical statistics(1961), 148–170.
[20]
Mary L Gray and Siddharth Suri. 2019. Ghost Work: How to Stop Silicon Valley from Building a New Global Underclass. Eamon Dolan Books.
[21]
Andrew G Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam. 2017. Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv:1704.04861(2017).
[22]
Panagiotis G Ipeirotis, Foster Provost, and Jing Wang. 2010. Quality management on amazon mechanical turk. In Proceedings of the ACM SIGKDD workshop on human computation. 64–67.
[23]
Marius Kloft and Pavel Laskov. 2010. Online anomaly detection under adversarial impact. In Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics. 405–412.
[24]
Ivan Krasin, Tom Duerig, Neil Alldrin, Vittorio Ferrari, Sami Abu-El-Haija, Alina Kuznetsova, Hassan Rom, Jasper Uijlings, Stefan Popov, Shahab Kamali, Matteo Malloci, Jordi Pont-Tuset, Andreas Veit, Serge Belongie, Victor Gomes, Abhinav Gupta, Chen Sun, Gal Chechik, David Cai, Zheyun Feng, Dhyanesh Narayanan, and Kevin Murphy. 2017. OpenImages: A public dataset for large-scale multi-label and multi-class image classification.Dataset available from https://storage.googleapis.com/openimages/web/index.html(2017).
[25]
Walter S Lasecki, Jaime Teevan, and Ece Kamar. 2014. Information extraction and manipulation threats in crowd-powered systems. In Proceedings of the 17th ACM conference on Computer supported cooperative work & social computing. ACM, 248–256.
[26]
Thomas W Malone. 1981. Toward a theory of intrinsically motivating instruction. Cognitive science 5, 4 (1981), 333–369.
[27]
Winter Mason and Duncan J Watts. 2009. Financial incentives and the performance of crowds. In Proceedings of the ACM SIGKDD workshop on human computation. ACM, 77–85.
[28]
Cristina Ioana Muntean. 2011. Raising engagement in e-learning through gamification. In Proc. 6th International Conference on Virtual Learning ICVL, Vol. 1. 323–329.
[29]
Blaine Nelson, Marco Barreno, Fuching Jack Chi, Anthony D Joseph, Benjamin IP Rubinstein, Udam Saini, Charles Sutton, JD Tygar, and Kai Xia. 2009. Misleading learners: Co-opting your spam filter. In Machine learning in cyber trust. Springer, 17–51.
[30]
Nikolaos Pitropakis, Emmanouil Panaousis, Thanassis Giannetsos, Eleftherios Anastasiadis, and George Loukas. 2019. A taxonomy and survey of attacks against machine learning. Computer Science Review 34 (2019), 100199.
[31]
Marko Puljic and Robert Kozma. 2005. Activation clustering in neural and social networks. Complexity 10, 4 (2005), 42–50.
[32]
Ali Shafahi, W Ronny Huang, Mahyar Najibi, Octavian Suciu, Christoph Studer, Tudor Dumitras, and Tom Goldstein. 2018. Poison frogs! targeted clean-label poisoning attacks on neural networks. In Advances in Neural Information Processing Systems. 6103–6113.
[33]
Jacob Steinhardt, Pang Wei W Koh, and Percy S Liang. 2017. Certified defenses for data poisoning attacks. In Advances in neural information processing systems. 3517–3529.
[34]
Christian Szegedy, Sergey Ioffe, Vincent Vanhoucke, and Alexander A Alemi. 2017. Inception-v4, inception-resnet and the impact of residual connections on learning. In Thirty-first AAAI conference on artificial intelligence.
[35]
Brandon Tran, Jerry Li, and Aleksander Madry. 2018. Spectral signatures in backdoor attacks. In Advances in Neural Information Processing Systems. 8000–8010.
[36]
MW Van Someren, YF Barnard, and JAC Sandberg. 1994. The think aloud method: a practical approach to modelling cognitive. (1994).
[37]
Fabio Viola. 2011. Gamification-I Videogiochi nella Vita Quotidiana. Fabio Viola.
[38]
Luis Von Ahn. 2006. Games with a purpose. Computer 39, 6 (2006), 92–94.
[39]
Luis Von Ahn and Laura Dabbish. 2004. Labeling images with a computer game. In Proceedings of the SIGCHI conference on Human factors in computing systems. ACM, 319–326.
[40]
Luis Von Ahn and Laura Dabbish. 2008. Designing games with a purpose. Commun. ACM 51, 8 (2008), 58–67.
[41]
Gang Wang, Tianyi Wang, Haitao Zheng, and Ben Y Zhao. 2014. Man vs. machine: Practical adversarial detection of malicious crowdsourcing workers. In 23rd {USENIX} Security Symposium ({USENIX} Security 14). 239–254.
[42]
Vanessa Williamson. 2016. On the ethics of crowdsourced research. PS: Political Science & Politics 49, 1 (2016), 77–81.
[43]
Haoqi Zhang, Edith Law, Rob Miller, Krzysztof Gajos, David Parkes, and Eric Horvitz. 2012. Human computation tasks with global constraints. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. 217–226.
Index Terms
- The Design and Development of a Game to Study Backdoor Poisoning Attacks: The Backdoor Game
Index terms have been assigned to the content through auto-classification.
Recommendations
Detecting and Mitigating Backdoor Attacks with Dynamic and Invisible Triggers
Neural Information ProcessingAbstractWhen a deep learning-based model is attacked by backdoor attacks, it behaves normally for clean inputs, whereas outputs unexpected results for inputs with specific triggers. This causes serious threats to deep learning-based applications. Many ...
BadDet: Backdoor Attacks on Object Detection
Computer Vision – ECCV 2022 WorkshopsAbstractBackdoor attack is a severe security threat which injects a backdoor trigger into a small portion of training data such that the trained model gives incorrect predictions when the specific trigger appears. While most research in backdoor attacks ...
Backdoor attack detection via prediction trustworthiness assessment
AbstractBackdoor attack aims to compromise clean models without arousing suspicion, in which poisoned models behave normally for clean inputs yet return adversary-desired results when triggers appear. Due to the great insidiousness and hazard of backdoor ...
Comments
Information & Contributors
Information
Published In
April 2021
618 pages
ISBN:9781450380171
DOI:10.1145/3397481
- General Chairs:
- Tracy Hammond,
- Katrien Verbert,
- Dennis Parra,
- Program Chairs:
- Bart Knijnenburg,
- John O'Donovan,
- Paul Teale
Copyright © 2021 ACM.
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]
Sponsors
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 14 April 2021
Check for updates
Author Tags
Qualifiers
- Research-article
- Research
- Refereed limited
Conference
IUI '21: 26th International Conference on Intelligent User Interfaces
April 14 - 17, 2021
TX, College Station, USA
Upcoming Conference
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- View Citations1Total Citations
- 200Total Downloads
- Downloads (Last 12 months)18
- Downloads (Last 6 weeks)3
Reflects downloads up to 28 Feb 2025
Other Metrics
Citations
Cited By
View all- Swacha JGracel M(2023)Machine Learning in Gamification and Gamification in Machine Learning: A Systematic Literature MappingApplied Sciences10.3390/app13201142713:20(11427)Online publication date: 18-Oct-2023
View Options
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign inFull Access
View options
View or Download as a PDF file.
PDFeReader
View online with eReader.
eReaderHTML Format
View this article in HTML Format.
HTML Format