skip to main content
10.1145/3607865.3616175acmconferencesArticle/Chapter ViewAbstractPublication PagesmmConference Proceedingsconference-collections
research-article

Generalised Bias Mitigation for Personality Computing

Published: 29 October 2023 Publication History

Abstract

Building systems with the capability of predicting the socio-emotional states of humans has many promising applications. However, if not properly designed, such systems might lead to biased decisions if biased data was used for training. Bias mitigation remains an open problem, which tackles the correction of a model's disparate performance over different groups defined by particular sensitive attributes (e.g., gender, age, race). Most existing methods are designed and tested in simple settings, limiting their general applicability to more complex real-world scenarios. In this work, we design a novel fairness loss function named Multi-Group Parity (MGP) to provide a generalised approach for bias mitigation in personality computing. In contrast to existing works in the literature, MGP is generalised as it features four 'multiple' properties (4Mul): multiple tasks, multiple modalities, multiple sensitive attributes, and multi-valued attributes. Extensive experiments on two large multi-modal benchmark personality computing datasets demonstrate that the MGP sets new state-of-the-art performance both in the traditional and in the proposed 4Mul settings.

References

[1]
Alekh Agarwal, Alina Beygelzimer, Miroslav Dudík, John Langford, and Hanna Wallach. 2018. A reductions approach to fair classification. In International Conference on Machine Learning. PMLR, 60--69.
[2]
Alekh Agarwal, Miroslav Dudík, and Zhiwei Steven Wu. 2019. Fair regression: Quantitative definitions and reduction-based algorithms. In International Conference on Machine Learning. PMLR, 120--129.
[3]
Rachel KE Bellamy, Kuntal Dey, Michael Hind, Samuel C Hoffman, Stephanie Houde, Kalapriya Kannan, Pranay Lohia, Jacquelyn Martino, Sameep Mehta, Aleksandra Mojsilovi", et al. 2019. AI Fairness 360: An extensible toolkit for detecting and mitigating algorithmic bias. IBM Journal of Research and Development 63, 4/5 (2019), 4--1.
[4]
Yoshua Bengio, Nicholas Léonard, and Aaron Courville. 2013. Estimating or propagating gradients through stochastic neurons for conditional computation. arXiv preprint arXiv:1308.3432 (2013).
[5]
Hugo Berg, Siobhan Mackenzie Hall, Yash Bhalgat, Wonsuk Yang, Hannah Rose Kirk, Aleksandar Shtedritski, and Max Bain. 2022. A prompt array keeps the bias away: Debiasing vision-language models with adversarial learning. arXiv preprint arXiv:2203.11933 (2022).
[6]
Joy Buolamwini and Timnit Gebru. 2018. Gender shades: Intersectional accuracy disparities in commercial gender classification. In Proceedings of the 1st Conference on Fairness, Accountability and Transparency. 77--91.
[7]
Aylin Caliskan, Joanna J Bryson, and Arvind Narayanan. 2017. Semantics derived automatically from language corpora contain human-like biases. Science 356, 6334 (2017), 183--186.
[8]
Simon Caton and Christian Haas. 2020. Fairness in machine learning: A survey. arXiv preprint arXiv:2010.04053 (2020).
[9]
Oya Celiktutan and Hatice Gunes. 2015. Automatic prediction of impressions in time and across varying context: Personality, attractiveness and likeability. IEEE transactions on affective computing 8, 1 (2015), 29--42.
[10]
L Elisa Celis, Lingxiao Huang, Vijay Keswani, and Nisheeth K Vishnoi. 2019. Classification with fairness constraints: A meta-algorithm with provable guarantees. In Proceedings of the conference on fairness, accountability, and transparency. 319--328.
[11]
Lieyang Chen, Anthony Cruz, Steven Ramsey, Callum J. Dickson, Jose S. Duca, Viktor Hornak, David R. Koes, and Tom Kurtzman. 2019. Hidden bias in the DUD-E dataset leads to misleading performance of deep learning in structurebased virtual screening. PLOS ONE 14, 8 (08 2019), 1--22. https://doi.org/10.1371/ journal.pone.0220113
[12]
Jiaee Cheong, Sinan Kalkan, and Hatice Gunes. 2021. The Hitchhiker's Guide to Bias and Fairness in Facial Affective Signal Processing: Overview and techniques. IEEE Signal Processing Magazine 38, 6 (2021), 39--49. https://doi.org/10.1109/MSP. 2021.3106619
[13]
Vetha Vikashini Chithrra Raghuram, Hanan Salam, Jauwairia Nasir, Barbara Bruno, and Oya Celiktutan. 2022. Personalized productive engagement recognition in robot-mediated collaborative learning. In Proceedings of the 2022 International Conference on Multimodal Interaction. 632--641.
[14]
Nikhil Churamani, Ozgur Kara, and Hatice Gunes. 2022. Domain-incremental continual learning for mitigating bias in facial expression and action unit recognition. IEEE Transactions on Affective Computing (2022).
[15]
Sahraoui Dhelim, Nyothiri Aung, Mohammed Amine Bouras, Huansheng Ning, and Erik Cambria. 2022. A survey on personality-aware recommendation systems. Artificial Intelligence Review (2022), 1--46.
[16]
Cynthia Dwork, Vitaly Feldman, Moritz Hardt, Toniann Pitassi, Omer Reingold, and Aaron Roth. 2012. Fairness through awareness. In Innovations in Theoretical Computer Science Conference. ACM, 214--226.
[17]
Cynthia Dwork, Nicole Immorlica, Adam Tauman Kalai, and Max Leiserson. 2018. Decoupled classifiers for group-fair and efficient machine learning. In Conference on fairness, accountability and transparency. PMLR, 119--133.
[18]
William Falcon et al. 2019. PyTorch Lightning. GitHub. Note: https://github.com/PyTorchLightning/pytorch-lightning 3 (2019).
[19]
Michael Feldman, Sorelle A Friedler, John Moeller, Carlos Scheidegger, and Suresh Venkatasubramanian. 2015. Certifying and removing disparate impact. In proceedings of the 21th ACM SIGKDD international conference on knowledge discovery and data mining. 259--268.
[20]
Oguzhan Gencoglu. 2020. Cyberbullying detection with fairness constraints. IEEE Internet Computing 25, 1 (2020), 20--29.
[21]
Leilani H. Gilpin, Danielle M. Olson, and Tarfah Alrashed. 2018. Perception of Speaker Personality Traits Using Speech Signals (CHI EA '18). Association for Computing Machinery, New York, NY, USA, 1--6. https://doi.org/10.1145/ 3170427.3188557
[22]
Jianzhu Guo, Xiangyu Zhu, and Zhen Lei. 2018. 3DDFA. https://github.com/ cleardusk/3DDFA.
[23]
Jianzhu Guo, Xiangyu Zhu, Yang Yang, Fan Yang, Zhen Lei, and Stan Z Li. 2020. Towards fast, accurate and stable 3d dense face alignment. In Computer Vision--ECCV 2020: 16th European Conference, Glasgow, UK, August 23--28, 2020, Proceedings, Part XIX. Springer, 152--168.
[24]
Wei Han, Hui Chen, and Soujanya Poria. 2021. Improving multimodal fusion with hierarchical mutual information maximization for multimodal sentiment analysis. arXiv preprint arXiv:2109.00412 (2021).
[25]
Andrew Jaegle, Felix Gimeno, Andy Brock, Oriol Vinyals, Andrew Zisserman, and Joao Carreira. 2021. Perceiver: General perception with iterative attention. In International conference on machine learning. PMLR, 4651--4664.
[26]
Robert Jenke, Angelika Peer, and Martin Buss. 2014. Feature Extraction and Selection for Emotion Recognition from EEG. IEEE Transactions on Affective Computing 5, 3 (2014), 327--339. https://doi.org/10.1109/TAFFC.2014.2339834
[27]
Jian Jiang and Oya Celiktutan. 2023. Neural Weight Search for Scalable Task Incremental Learning. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision. 1390--1399.
[28]
Faisal Kamiran and Toon Calders. 2012. Data preprocessing techniques for classification without discrimination. Knowledge and Information Systems 33, 1 (2012), 1--33.
[29]
Faisal Kamiran, Sameen Mansha, Asim Karim, and Xiangliang Zhang. 2018. Exploiting reject option in classification for social discrimination control. Information Sciences 425 (2018), 18--33.
[30]
Heysem Kaya and Albert Ali Salah. 2018. Multimodal personality trait analysis for explainable modeling of job interview decisions. Explainable and Interpretable Models in Computer Vision and Machine Learning (2018), 255--275.
[31]
Isaac E Kim Jr and Indra Neil Sarkar. 2019. Racial representation disparity of population-level genomic sequencing efforts. In MEDINFO 2019: Health and Wellbeing e-Networks for All. IOS Press, 974--978.
[32]
Hemank Lamba, Kit T Rodolfa, and Rayid Ghani. 2021. An empirical comparison of bias reduction methods on real-world problems in high-stakes policy settings. ACM SIGKDD Explorations Newsletter 23, 1 (2021), 69--85.
[33]
Jialin Li, Alia Waleed, and Hanan Salam. 2023. A survey on personalized affective computing in human-machine interaction. arXiv preprint arXiv:2304.00377 (2023).
[34]
Shan Li and Weihong Deng. 2022. A Deeper Look at Facial Expression Dataset Bias. IEEE Transactions on Affective Computing 13, 2 (2022), 881--893. https: //doi.org/10.1109/TAFFC.2020.2973158
[35]
Yang Li, Amirmohammad Kazameini, Yash Mehta, and Erik Cambria. 2021. Multitask learning for emotion and personality detection. arXiv preprint arXiv:2101.02346 (2021).
[36]
Rongfan Liao, Siyang Song, and Hatice Gunes. 2022. An open-source benchmark of deep learning models for audio-visual apparent and self-reported personality recognition. arXiv preprint arXiv:2210.09138 (2022).
[37]
Daniel Lowd and Christopher Meek. 2005. Adversarial Learning. In Proceedings of the Eleventh ACM SIGKDD International Conference on Knowledge Discovery in Data Mining (Chicago, Illinois, USA) (KDD '05). Association for Computing Machinery, New York, NY, USA, 641--647. https://doi.org/10.1145/1081870.1081950
[38]
Andrew Lowy, Rakesh Pavan, Sina Baharlouei, Meisam Razaviyayn, and Ahmad Beirami. 2021. FERMI: Fair Empirical Risk Minimization via Exponential R\'enyi Mutual Information. arXiv preprint arXiv:2102.12586 (2021).
[39]
Padala Manisha and Sujit Gujar. 2020. FNNC: Achieving Fairness through Neural Networks. arXiv:1811.00247 [cs.LG]
[40]
Ninareh Mehrabi, Fred Morstatter, Nanyun Peng, and Aram Galstyan. 2019. Debiasing community detection: the importance of lowly connected nodes. In Proceedings of the 2019 IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining. 509--512.
[41]
Ali Mollahosseini, Behzad Hasani, and Mohammad H Mahoor. 2019. Deep Learning for Emotion Recognition on Small Datasets Using Transfer Learning and Data Augmentation. In 2019 14th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2019). IEEE, 1--8.
[42]
Dena F Mujtaba and Nihar R Mahapatra. 2021. Multi-Task Deep Neural Networks for Multimodal Personality Trait Prediction. In 2021 International Conference on Computational Science and Computational Intelligence (CSCI). IEEE, 85--91.
[43]
Razieh Nabi, Daniel Malinsky, and Ilya Shpitser. 2019. Learning Optimal Fair Policies. In Proceedings of the 36th International Conference on Machine Learning (Proceedings of Machine Learning Research, Vol. 97), Kamalika Chaudhuri and Ruslan Salakhutdinov (Eds.). PMLR, 4674--4682. https://proceedings.mlr.press/ v97/nabi19a.html
[44]
Razieh Nabi and Ilya Shpitser. 2018. Fair inference on outcomes. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 32.
[45]
Cristina Palmero, Germán Barquero, Julio Junior, Albert Clapés, Johnny Núñez Cano, David Janó, Javier Selva, Zejian Zhang, David Saeteros Pérez, David Gallardo-Pujol, Georgina Guilera, David Leiva, Universitat Barcelona, Spain Feng, Xiaoxue Feng, Jennifer He, Wei-Wei Tu, Thomas Moeslund, and Sergio Escalera. 2022. ChaLearn LAP Challenges on Self-Reported Personality Recognition and Non-Verbal Behavior Forecasting During Social Dyadic Interactions: Dataset, Design, and Results. (04 2022).
[46]
Le Vy Phan and John F Rauthmann. 2021. Personality computing: New frontiers in personality assessment. Social and personality psychology compass 15, 7 (2021), e12624.
[47]
Geoff Pleiss, Manish Raghavan, Felix Wu, Jon Kleinberg, and Kilian Q Weinberger. 2017. On fairness and calibration. Advances in neural information processing systems 30 (2017).
[48]
Víctor Ponce-López, Baiyu Chen, Marc Oliu, Ciprian Corneanu, Albert Clapés, Isabelle Guyon, Xavier Baró, Hugo Jair Escalante, and Sergio Escalera. 2016. ChaLearn LAP 2016: First Round Challenge on First Impressions - Dataset and Results. In Computer Vision -- ECCV 2016 Workshops, Gang Hua and Hervé Jégou (Eds.). Springer International Publishing, Cham, 400--418.
[49]
Kalpana Rangra, Virender Kadyan, and Monit Kapoor. 2023. Emotional speechbased personality prediction using NPSO architecture in deep learning. Measurement: Sensors 25 (2023), 100655. https://doi.org/10.1016/j.measen.2022.100655
[50]
Nils Reimers and Iryna Gurevych. 2019. Sentence-bert: Sentence embeddings using siamese bert-networks. arXiv preprint arXiv:1908.10084 (2019).
[51]
Arjun Roy and Eirini Ntoutsi. 2023. Learning to Teach Fairness-Aware Deep Multi-task Learning. In Machine Learning and Knowledge Discovery in Databases: European Conference, ECML PKDD 2022, Grenoble, France, September 19--23, 2022, Proceedings, Part I. Springer, 710--726.
[52]
Hanan Salam, Oya Celiktutan, Hatice Gunes, and Mohamed Chetouani. 2023. Automatic Context-Aware Inference of Engagement in HMI: A Survey. IEEE Transactions on Affective Computing (2023).
[53]
Hanan Salam, Oya Celiktutan, Isabelle Hupont, Hatice Gunes, and Mohamed Chetouani. 2016. Fully automatic analysis of engagement and its relationship to personality in human-robot interactions. IEEE Access 5 (2016), 705--721.
[54]
Hanan Salam, Viswonathan Manoranjan, Jian Jiang, and Oya Celiktutan. 2022. Learning Personalised Models for Automatic Self-Reported Personality Recognition. In Understanding Social Behavior in Dyadic and Small Group Interactions (Proceedings of Machine Learning Research, Vol. 173), Cristina Palmero, Julio C. S. Jacques Junior, Albert Clapés, Isabelle Guyon, Wei-Wei Tu, Thomas B. Moeslund, and Sergio Escalera (Eds.). PMLR, 53--73.
[55]
Arashdeep Singh, Jashandeep Singh, Ariba Khan, and Amar Gupta. 2022. Developing a Novel Fair-Loan Classifier through a Multi-Sensitive Debiasing Pipeline: DualFair. Machine Learning and Knowledge Extraction 4, 1 (2022), 240--253.
[56]
Jiaming Song, Pratyusha Kalluri, Aditya Grover, Shengjia Zhao, and Stefano Ermon. 2019. Learning Controllable Fair Representations. In Proceedings of the Twenty-Second International Conference on Artificial Intelligence and Statistics (Proceedings of Machine Learning Research, Vol. 89), Kamalika Chaudhuri and Masashi Sugiyama (Eds.). PMLR, 2164--2173. https://proceedings.mlr.press/v89/ song19a.html
[57]
Chris Sweeney and Maryam Najafian. 2020. Reducing Sentiment Polarity for Demographic Attributes in Word Embeddings Using Adversarial Learning. In Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency (Barcelona, Spain) (FAT* '20). Association for Computing Machinery, New York, NY, USA, 359--368. https://doi.org/10.1145/3351095.3372837
[58]
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, "ukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Advances in neural information processing systems 30 (2017).
[59]
Alessandro Vinciarelli and Gelareh Mohammadi. 2014. A survey of personality computing. IEEE Transactions on Affective Computing 5, 3 (2014), 273--291.
[60]
Christina Wadsworth, Francesca Vera, and Chris Piech. 2018. Achieving fairness through adversarial learning: an application to recidivism prediction. arXiv preprint arXiv:1807.00199 (2018).
[61]
Tianlu Wang, Jieyu Zhao, Mark Yatskar, Kai-Wei Chang, and Vicente Ordonez. 2019. Balanced Datasets Are Not Enough: Estimating and Mitigating Gender Bias in Deep Image Representations. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV).
[62]
Yuyan Wang, Xuezhi Wang, Alex Beutel, Flavien Prost, Jilin Chen, and Ed H Chi. 2021. Understanding and improving fairness-accuracy trade-offs in multitask learning. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining. 1748--1757.
[63]
Yongkai Wu, Lu Zhang, and Xintao Wu. 2018. Fairness-aware classification: Criterion, convexity, and bounds. arXiv preprint arXiv:1809.04737 (2018).
[64]
Shen Yan, Di Huang, and Mohammad Soleymani. 2020. Mitigating biases in multimodal personality assessment. In Proceedings of the 2020 International Conference on Multimodal Interaction. 361--369.
[65]
Shen Yan, Hsien-Te Kao, Kristina Lerman, Shrikanth Narayanan, and Emilio Ferrara. 2021. Mitigating the Bias of Heterogeneous Human Behavior in Affective Computing. In 2021 9th International Conference on Affective Computing and Intelligent Interaction (ACII). 1--8. https://doi.org/10.1109/ACII52823.2021.9597439
[66]
Yang You, Jing Li, Sashank Reddi, Jonathan Hseu, Sanjiv Kumar, Srinadh Bhojanapalli, Xiaodan Song, James Demmel, Kurt Keutzer, and Cho-Jui Hsieh. 2019. Large batch optimization for deep learning: Training bert in 76 minutes. arXiv preprint arXiv:1904.00962 (2019).
[67]
Zhe Yu, Joymallya Chakraborty, and Tim Menzies. 2021. Fairer Machine Learning Software on Multiple Sensitive Attributes With Data Preprocessing. arXiv preprint arXiv:2107.08310 (2021).
[68]
Brian Hu Zhang, Blake Lemoine, and Margaret Mitchell. 2018. Mitigating unwanted biases with adversarial learning. In Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. 335--340.
[69]
Oya Çeliktutan and Hatice Gunes. 2017. Automatic Prediction of Impressions in Time and across Varying Context: Personality, Attractiveness and Likeability. IEEE Transactions on Affective Computing 8, 1 (2017), 29--42. https://doi.org/10. 1109/TAFFC.2015.2513401

Cited By

View all
  • (2024)MRAC Track 1: 2nd Workshop on Multimodal, Generative and Responsible Affective ComputingProceedings of the 2nd International Workshop on Multimodal and Responsible Affective Computing10.1145/3689092.3690042(1-6)Online publication date: 28-Oct-2024

Index Terms

  1. Generalised Bias Mitigation for Personality Computing

    Recommendations

    Comments

    Information & Contributors

    Information

    Published In

    cover image ACM Conferences
    MRAC '23: Proceedings of the 1st International Workshop on Multimodal and Responsible Affective Computing
    October 2023
    88 pages
    ISBN:9798400702884
    DOI:10.1145/3607865
    Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

    Sponsors

    Publisher

    Association for Computing Machinery

    New York, NY, United States

    Publication History

    Published: 29 October 2023

    Permissions

    Request permissions for this article.

    Check for updates

    Author Tags

    1. bias mitigation
    2. multi-modality learning
    3. multi-task learning

    Qualifiers

    • Research-article

    Funding Sources

    • King's China Scholarship Council (K-CSC) PhD Scholarship programme
    • NVIDIA Academic Hardware Grant Program
    • EPSRC

    Conference

    MM '23
    Sponsor:

    Contributors

    Other Metrics

    Bibliometrics & Citations

    Bibliometrics

    Article Metrics

    • Downloads (Last 12 months)67
    • Downloads (Last 6 weeks)2
    Reflects downloads up to 21 Jan 2025

    Other Metrics

    Citations

    Cited By

    View all
    • (2024)MRAC Track 1: 2nd Workshop on Multimodal, Generative and Responsible Affective ComputingProceedings of the 2nd International Workshop on Multimodal and Responsible Affective Computing10.1145/3689092.3690042(1-6)Online publication date: 28-Oct-2024

    View Options

    Login options

    View options

    PDF

    View or Download as a PDF file.

    PDF

    eReader

    View online with eReader.

    eReader

    Media

    Figures

    Other

    Tables

    Share

    Share

    Share this Publication link

    Share on social media