Abstract
Fairness-aware techniques are designed to remove socially sensitive information, such as gender or race. Many types of fairness-aware predictors have been developed, but they were designed essentially to improve the accuracy or fairness of the prediction results. We focus herein on another aspect of fairness-aware predictors, i.e., the stability. We define that fairness-aware techniques are stable if the same models are learned when a training dataset contains the same information except for the sensitive information. We sought to collect benchmark datasets to investigate such stability. We collected preference data in a manner ensuring that the users’ responses were influenced by cognitive biases. If the same models are learned for a dataset influenced by different types of cognitive biases, the learner of the models can be considered stable. We performed preliminary experiments using this dataset, but we failed to fully remove the influence of cognitive biases. We discuss the necessary next steps to solve this problem.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Calders, T., Verwer, S.: Three naive bayes approaches for discrimination-free classification. Data Min. Knowl. Disc. 21, 277–292 (2010). https://doi.org/10.1007/s10618-010-0190-x
Chandler, D., Horton, J.: Labor allocation in paid crowdsourcing: experimental evidence on positioning, nudges and prices. In: AAAI Workshop: Human Computation, pp. 14–19 (2011). https://www.aaai.org/ocs/index.php/WS/AAAIW11/paper/view/3983
Dwork, C., Hardt, M., Pitassi, T., Reingold, O., Zemel, R.: Fairness through awareness. In: Proceedings of the 3rd Innovations in Theoretical Computer Science Conference, pp. 214–226 (2012). https://doi.org/10.1145/2090236.2090255
Eickhoff, C.: Cognitive biases in crowdsourcing. In: Proceedings of the 11th ACM International Conference on Web Search and Data Mining, pp. 162–170 (2018). https://doi.org/10.1145/3159652.3159654
Hardt, M., Price, E., Srebro, N.: Equality of opportunity in supervised learning. In: Advances in Neural Information Processing Systems, vol. 29 (2016). https://papers.neurips.cc/paper/6374-equality-of-opportunity-in-supervised-learning
Kamiran, F., Karim, A., Zhang, X.: Decision theory for discrimination-aware classification. In: Proceedings of the 12th IEEE International Conference on Data Mining, pp. 924–929 (2012). https://doi.org/10.1109/ICDM.2012.45. https://doi.ieeecomputersociety.org/10.1109/ICDM.2012.45
Kamishima, T.: Nantonac collaborative filtering: recommendation based on order responses. In: Proceedings of l9th International Conference on Knowledge Discovery and Data Mining, pp. 583–588 (2003). https://doi.org/10.1145/956750.956823
Zhang, L., Wu, Y., Wu, X.: Anti-discrimination learning: from association to causation. In: The 24th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, Tutorial (2018). http://csce.uark.edu/~xintaowu/kdd18-tutorial/
Žliobaitė, I.: Measuring discrimination in algorithmic decision making. Data Min. Knowl. Disc. 31(4), 1060–1089 (2017). https://doi.org/10.1007/s10618-017-0506-1
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2021 Springer Nature Switzerland AG
About this paper
Cite this paper
Kamishima, T., Akaho, S., Baba, Y., Kashima, H. (2021). Preliminary Experiments to Examine the Stability of Bias-Aware Techniques. In: Boratto, L., Faralli, S., Marras, M., Stilo, G. (eds) Advances in Bias and Fairness in Information Retrieval. BIAS 2021. Communications in Computer and Information Science, vol 1418. Springer, Cham. https://doi.org/10.1007/978-3-030-78818-6_4
Download citation
DOI: https://doi.org/10.1007/978-3-030-78818-6_4
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-78817-9
Online ISBN: 978-3-030-78818-6
eBook Packages: Computer ScienceComputer Science (R0)