research-article

Black box fairness testing of machine learning models

Authors:

Aniya Aggarwal,

Diptikalyan SahaAuthors Info & Claims

ESEC/FSE 2019: Proceedings of the 2019 27th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering

Pages 625 - 635

https://doi.org/10.1145/3338906.3338937

Published: 12 August 2019 Publication History

Abstract

Any given AI system cannot be accepted unless its trustworthiness is proven. An important characteristic of a trustworthy AI system is the absence of algorithmic bias. 'Individual discrimination' exists when a given individual different from another only in 'protected attributes' (e.g., age, gender, race, etc.) receives a different decision outcome from a given machine learning (ML) model as compared to the other individual. The current work addresses the problem of detecting the presence of individual discrimination in given ML models. Detection of individual discrimination is test-intensive in a black-box setting, which is not feasible for non-trivial systems. We propose a methodology for auto-generation of test inputs, for the task of detecting individual discrimination. Our approach combines two well-established techniques - symbolic execution and local explainability for effective test case generation. We empirically show that our approach to generate test cases is very effective as compared to the best-known benchmark systems that we examine.

References

[1]

Julius Adebayo and Lalana Kagal. 2016. Iterative Orthogonal Feature Projection for Diagnosing Bias in Black-Box Models. arXiv: arXiv:1611.04967

[2]

Cristian Cadar, Vijay Ganesh, Peter M. Pawlowski, David L. Dill, and Dawson R. Engler. 2006. EXE: Automatically Generating Inputs of Death. In Proceedings of the 13th ACM Conference on Computer and Communications Security (CCS ’06). ACM, New York, NY, USA, 322–335.

Digital Library

[3]

Mark William Craven. 1996. Extracting Comprehensible Models from Trained Neural Networks. Ph.D. Dissertation. AAI9700774.

[4]

Leonardo de Moura and Nikolaj Bjørner. 2008. Z3: An Efficient SMT Solver. In Tools and Algorithms for the Construction and Analysis of Systems, C. R. Ramakrishnan and Jakob Rehof (Eds.). Springer Berlin Heidelberg, Berlin, Heidelberg, 337–340.

Digital Library

[5]

Cynthia Dwork, Moritz Hardt, Toniann Pitassi, Omer Reingold, and Richard Zemel. 2012. Fairness Through Awareness. In Proceedings of the 3rd Innovations in Theoretical Computer Science Conference (ITCS ’12). ACM, New York, NY, USA, 214–226.

Digital Library

[6]

Michael Feldman, Sorelle A Friedler, John Moeller, Carlos Scheidegger, and Suresh Venkatasubramanian. 2015. Certifying and removing disparate impact. In Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. ACM, 259–268.

Digital Library

[7]

Sainyam Galhotra, Yuriy Brun, and Alexandra Meliou. 2017. Fairness Testing: Testing Software for Discrimination. In Proceedings of the 2017 11th Joint Meeting on Foundations of Software Engineering (ESEC/FSE 2017). ACM, New York, NY, USA, 498–510.

Digital Library

[8]

Patrice Godefroid. 2007. Compositional Dynamic Test Generation. In Proceedings of the 34th Annual ACM SIGPLAN-SIGACT Symposium on Principles of Programming Languages (POPL ’07). ACM, New York, NY, USA, 47–54.

Digital Library

[9]

Patrice Godefroid, Nils Klarlund, and Koushik Sen. 2005. DART: Directed Automated Random Testing. In Proceedings of the 2005 ACM SIGPLAN Conference on Programming Language Design and Implementation (PLDI ’05). ACM, New York, NY, USA, 213–223.

Digital Library

[10]

Divya Gopinath, Kaiyuan Wang, Mengshi Zhang, Corina S. Pasareanu, and Sarfraz Khurshid. 2018.

[11]

Symbolic Execution for Deep Neural Networks. arXiv: arXiv:1807.10439

[12]

Pang Wei Koh and Percy Liang. 2017. Understanding Black-box Predictions via Influence Functions. In Proceedings of the 34th International Conference on Machine Learning (Proceedings of Machine Learning Research), Doina Precup and Yee Whye Teh (Eds.), Vol. 70. PMLR, International Convention Centre, Sydney, Australia, 1885–1894. http://proceedings.mlr.press/v70/koh17a.html

Digital Library

[13]

Matt Kusner, Joshua Loftus, Chris Russell, and Ricardo Silva. 2017. Counterfactual Fairness. In Proceedings of the 31st International Conference on Neural Information Processing Systems (NIPS’17). Curran Associates Inc., USA, 4069–4079. http: //dl.acm.org/citation.cfm?id=3294996.3295162

Digital Library

[14]

Lei Ma, Felix Juefei-Xu, Fuyuan Zhang, Jiyuan Sun, Minhui Xue, Bo Li, Chunyang Chen, Ting Su, Li Li, Yang Liu, Jianjun Zhao, and Yadong Wang. 2018. DeepGauge: multi-granularity testing criteria for deep learning systems. In Proceedings of the 33rd ACM/IEEE International Conference on Automated Software Engineering, ASE 2018, Montpellier, France, September 3-7, 2018. 120–131. 3238147.3238202

Digital Library

[15]

Kexin Pei, Yinzhi Cao, Junfeng Yang, and Suman Jana. 2017. DeepXplore: Automated Whitebox Testing of Deep Learning Systems. In Proceedings of the 26th Symposium on Operating Systems Principles (SOSP ’17). ACM, New York, NY, USA, 1–18.

Digital Library

[16]

Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. 2016. Why should i trust you?: Explaining the predictions of any classifier. In Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining. ACM, 1135–1144.

Digital Library

[17]

R. R. Selvaraju, M. Cogswell, A. Das, R. Vedantam, D. Parikh, and D. Batra. 2017. Grad-CAM: Visual Explanations from Deep Networks via Gradient-Based Localization. In 2017 IEEE International Conference on Computer Vision (ICCV). 618–626.

[18]

Koushik Sen, Darko Marinov, and Gul Agha. 2005. CUTE: A Concolic Unit Testing Engine for C. In Proceedings of the 10th European Software Engineering Conference Held Jointly with 13th ACM SIGSOFT International Symposium on Foundations of Software Engineering (ESEC/FSE-13). ACM, New York, NY, USA, 263–272.

Digital Library

[19]

Youcheng Sun, Xiaowei Huang, and Daniel Kroening. 2018. Testing Deep Neural Networks. arXiv: arXiv:1803.04792

[20]

Youcheng Sun, Min Wu, Wenjie Ruan, Xiaowei Huang, Marta Kwiatkowska, and Daniel Kroening. 2018.

[21]

Concolic Testing for Deep Neural Networks. arXiv: arXiv:1805.00089

[22]

Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian J. Goodfellow, and Rob Fergus. 2013. Intriguing properties of neural networks. CoRR abs/1312.6199 (2013). arXiv: 1312.6199 http://arxiv.org/abs/1312.6199

[23]

Yuchi Tian, Kexin Pei, Suman Jana, and Baishakhi Ray. 2018. DeepTest: Automated Testing of Deep-neural-network-driven Autonomous Cars. In Proceedings of the 40th International Conference on Software Engineering (ICSE ’18). ACM, New York, NY, USA, 303–314.

Digital Library

[24]

F. TramÃĺr, V. Atlidakis, R. Geambasu, D. Hsu, J. Hubaux, M. Humbert, A. Juels, and H. Lin. 2017. FairTest: Discovering Unwarranted Associations in Data-Driven Applications. In 2017 IEEE European Symposium on Security and Privacy (EuroS P). 401–416.

[25]

Sakshi Udeshi, Pryanshu Arora, and Sudipta Chattopadhyay. 2018. Automated Directed Fairness Testing. Automated Directed Fairness Testing. In Proceedings of the 2018 33rd ACM/IEEE International Conference on Automated Software Engineering (ASE 18), September 3-7, 2018, Montpellier, France. (2018).

Digital Library

[26]

arXiv: arXiv:1807.00468

Cited By

Chen JZhang YZhang LZhang MWan CSu TPu GFilkov VRay BZhou M(2024)FIPSER: Improving Fairness Testing of DNN by Seed PrioritizationProceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering10.1145/3691620.3695486(1069-1081)Online publication date: 27-Oct-2024
https://dl.acm.org/doi/10.1145/3691620.3695486
Zhao ZToda TKitamura TFilkov VRay BZhou M(2024)Approximation-guided Fairness Testing through Discriminatory Space AnalysisProceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering10.1145/3691620.3695481(1007-1018)Online publication date: 27-Oct-2024
https://dl.acm.org/doi/10.1145/3691620.3695481
Kitamura TAmasaki SInoue JIsobe YToda TFilkov VRay BZhou M(2024)Toward Individual Fairness Testing with Data ValidityProceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering10.1145/3691620.3695300(2284-2288)Online publication date: 27-Oct-2024
https://dl.acm.org/doi/10.1145/3691620.3695300
Show More Cited By

Index Terms

Black box fairness testing of machine learning models
1. Software and its engineering
  1. Software creation and management
    1. Software verification and validation
      1. Software defect analysis
        Software testing and debugging

Recommendations

Fairness testing: testing software for discrimination
ESEC/FSE 2017: Proceedings of the 2017 11th Joint Meeting on Foundations of Software Engineering

This paper defines software fairness and discrimination and develops a testing-based method for measuring if and how much software discriminates, focusing on causality in discriminatory behavior. Evidence of software discrimination has been found in ...
Latent Imitator: Generating Natural Individual Discriminatory Instances for Black-Box Fairness Testing
ISSTA 2023: Proceedings of the 32nd ACM SIGSOFT International Symposium on Software Testing and Analysis

Machine learning (ML) systems have achieved remarkable performance across a wide area of applications. However, they frequently exhibit unfair behaviors in sensitive application domains (e.g., employment and loan), raising severe fairness concerns. To ...
Achieving both model and code coverage with automated gray-box testing
A-MOST '07: Proceedings of the 3rd international workshop on Advances in model-based testing

We have devised a novel technique to automatically generate test cases for a software system, combining black-box model-based testing with white-box parameterized unit testing. The former provides general guidance for the structure of the tests in the ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

ESEC/FSE 2019: Proceedings of the 2019 27th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering

August 2019

1264 pages

ISBN:9781450355728

DOI:10.1145/3338906

General Chairs:
Marlon Dumas
University of Tartu, Estonia
,
Dietmar Pfahl
University of Tartu, Estonia
,
Program Chairs:
Sven Apel
Saarland University, Germany
,
Alessandra Russo
Imperial College, UK

Copyright © 2019 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGSOFT: ACM Special Interest Group on Software Engineering

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 12 August 2019

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

ESEC/FSE '19

Sponsor:

SIGSOFT

ESEC/FSE '19: 27th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering

August 26 - 30, 2019

Tallinn, Estonia

Acceptance Rates

Overall Acceptance Rate 112 of 543 submissions, 21%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

109
Total Citations
View Citations
2,655
Total Downloads

Downloads (Last 12 months)378
Downloads (Last 6 weeks)31

Reflects downloads up to 03 Mar 2025

Other Metrics

View Author Metrics

Citations

Cited By

Chen JZhang YZhang LZhang MWan CSu TPu GFilkov VRay BZhou M(2024)FIPSER: Improving Fairness Testing of DNN by Seed PrioritizationProceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering10.1145/3691620.3695486(1069-1081)Online publication date: 27-Oct-2024
https://dl.acm.org/doi/10.1145/3691620.3695486
Zhao ZToda TKitamura TFilkov VRay BZhou M(2024)Approximation-guided Fairness Testing through Discriminatory Space AnalysisProceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering10.1145/3691620.3695481(1007-1018)Online publication date: 27-Oct-2024
https://dl.acm.org/doi/10.1145/3691620.3695481
Kitamura TAmasaki SInoue JIsobe YToda TFilkov VRay BZhou M(2024)Toward Individual Fairness Testing with Data ValidityProceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering10.1145/3691620.3695300(2284-2288)Online publication date: 27-Oct-2024
https://dl.acm.org/doi/10.1145/3691620.3695300
Openja MKhomh FFoundjem AJiang ZAbidi MHassan A(2024)An Empirical Study of Testing Machine Learning in the WildACM Transactions on Software Engineering and Methodology10.1145/368046334:1(1-63)Online publication date: 24-Jul-2024
https://dl.acm.org/doi/10.1145/3680463
Du CChen T(2024)Contexts Matter: An Empirical Study on Contextual Influence in Fairness Testing for Deep Learning SystemsProceedings of the 18th ACM/IEEE International Symposium on Empirical Software Engineering and Measurement10.1145/3674805.3686673(107-118)Online publication date: 24-Oct-2024
https://dl.acm.org/doi/10.1145/3674805.3686673
Wang ZSun HZhao ZCai JKankanhalli MPrabhakaran BBoll SSubramanian RZheng LSingh VCesar PXie LXu D(2024)FedEvalFair: A Privacy-Preserving and Statistically Grounded Federated Fairness Evaluation FrameworkProceedings of the 32nd ACM International Conference on Multimedia10.1145/3664647.3681545(7191-7199)Online publication date: 28-Oct-2024
https://dl.acm.org/doi/10.1145/3664647.3681545
Sun ZChen ZZhang JHao D(2024)Fairness Testing of Machine Translation SystemsACM Transactions on Software Engineering and Methodology10.1145/366460833:6(1-27)Online publication date: 27-Jun-2024
https://dl.acm.org/doi/10.1145/3664608
Robles Herrera SMonjezi VKreinovich VTrivedi ATizpaz-Niari SShang WLamothe MWan Z(2024)Predicting Fairness of ML Software ConfigurationsProceedings of the 20th International Conference on Predictive Models and Data Analytics in Software Engineering10.1145/3663533.3664040(56-65)Online publication date: 10-Jul-2024
https://dl.acm.org/doi/10.1145/3663533.3664040
Chen ZZhang JHort MHarman MSarro F(2024)Fairness Testing: A Comprehensive Survey and Analysis of TrendsACM Transactions on Software Engineering and Methodology10.1145/365215533:5(1-59)Online publication date: 4-Jun-2024
https://dl.acm.org/doi/10.1145/3652155
Dasu VKumar ATizpaz-Niari STan GChristakis MPradel M(2024)NeuFair: Neural Network Fairness Repair with DropoutProceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis10.1145/3650212.3680380(1541-1553)Online publication date: 11-Sep-2024
https://dl.acm.org/doi/10.1145/3650212.3680380
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten