Skip to main content
SpringerLink
Log in

Diversity in issue assignment: humans vs bots

  • Published:
Empirical Software Engineering Aims and scope Submit manuscript

Abstract

Issue assignment process is a common practice in open source projects for managing incoming and existing issues. While traditionally performed by humans, the adoption of software bots for automating this process has become prevalent in recent years. The objective of this paper is to examine the diversity in issue assignments between bots and humans in open source projects, with the aim of understanding how open source communities can foster diversity and inclusivity. To achieve this, we conducted a quantitative analysis on three major open source projects hosted on GitHub, focusing on the most likely racial and ethnic diversity of both human and bot assignors during the issue assignment process. We analyze how issues are assigned by humans and bots, as well as the distribution of issue types among White and Non-White open source collaborators. Additionally, we explore how the diversity in issue assignments evolves over time for human and bot assignors. Our results reveal that both human and bot assignors majorly assign issues to developers of the same most likely race and ethnicity. Notably, we find bots assign more issues to perceived White developers than Non-White developers. In conclusion, our findings suggest that bots display higher levels of bias than humans in most cases, although humans also demonstrate significant bias in certain instances. Thus, open source communities must actively address these potential biases in their GitHub issue assignment process to promote diversity and inclusivity.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4

Similar content being viewed by others

Data Availability

The authors hereby confirm that all data generated or analyzed during the course of this study has been made publicly available and can be accessed through the following repository: https://github.com/Demfier/diversity-in-issue-assignment-humans-vs-bots. It is important to note that the data pertaining to GitHub users has undergone a thorough anonymization process to safeguard individual privacy.

Notes

  1. We acknowledge that code review and issue assignment serve different purposes, but in both cases, there is an assignee and an assignor.

  2. https://github.com/Demfier/diversity-in-issue-assignment-humans-vs-bots

  3. https://www.deepcode.ai/

  4. http://oss.x-lab.info/github-insight-report-2020-en.pdf

  5. https://www.census.gov/topics/population/race/about.html

  6. NameSor outputs a fifth category called “Other” in addition to the four categories.

  7. Full probability distribution for this example: \(p_1=48\%, p_2=2\%, p_3=25\%, p_4=25\%\)

  8. More details about vscode-triage-bot: https://github.com/microsoft/vscode-github-triage-actions

  9. https://www.census.gov/topics/population/race/about.html

References

  • Adams B, Khomh F (2020) The diversity crisis of software engineering for artificial intelligence. IEEE Softw 37(5):104–108

    Article  Google Scholar 

  • Ali M, Kulik CT, Metz I (2011) The gender diversity-performance relationship in services and manufacturing organizations. Int J Hum Resour Manag 22(07):1464–1485

    Article  Google Scholar 

  • Balachandran V (2013) Reducing human effort and improving quality in peer code reviews using automatic static analysis and reviewer recommendation. In: 2013 35th international conference on software engineering (ICSE). IEEE, pp 931–940

  • Bellamy RKE, Dey K, Hind M, Hoffman SC, Houde S, Kannan K, Lohia P, Martino J, Mehta S, Mojsilovic A et al (2018) Ai fairness 360: an extensible toolkit for detecting, understanding, and mitigating unwanted algorithmic bias. arXiv:1810.01943

  • Blau PM (1977) Inequality and heterogeneity: a primitive theory of social structure, vol 7. Free Press, New York

    Google Scholar 

  • Bosu A, Sultana KZ (2019) Diversity and inclusion in open source software (oss) projects: where do we stand? In: 2019 ACM/IEEE international symposium on empirical software engineering and measurement (ESEM). IEEE, pp 1–11

  • Brokmeier P (2017) Project level effects of gender on contribution evaluation on github. PeerJ PrePrints 5:e2989v1

  • Brown C, Parnin C (2019) Sorry to bother you: designing bots for effective recommendations. In: 2019 IEEE/ACM 1st international workshop on bots in software engineering (BotSE). IEEE, pp 54–58

  • Buolamwini J, Gebru T (2018) Gender shades: intersectional accuracy disparities in commercial gender classification. In: Conference on fairness, accountability and transparency. PMLR, pp 77–91

  • Caldarelli G, De Nicola R, Del Vigna F, Petrocchi M, Saracco F (2020) The role of bot squads in the political propaganda on twitter. Commun Phys 3(1):1–15

    Article  Google Scholar 

  • Carleton AD, Harper E, Menzies T, Xie T, Eldh S, Lyu MR (2020) The ai effect: working at the intersection of ai and se. IEEE Softw 37(4):26–35

    Article  Google Scholar 

  • Chakraborty J, Majumder S, Menzies T (2021) Bias in machine learning software: why? How? What to do? arXiv:2105.12195

  • Chakraborty J, Majumder S, Yu Z, Menzies T (2020) Fairway: a way to build fair ml software. In: Proceedings of the 28th ACM joint meeting on European software engineering conference and symposium on the foundations of software engineering, pp 654–665

  • Chen J, Ren Y, Riedl J (2010) The effects of diversity on group productivity and member withdrawal in online volunteer groups. In: Proceedings of the SIGCHI conference on human factors in computing systems, pp 821–830

  • Chouldechova A, Roth A (2018) The frontiers of fairness in machine learning. arXiv:1810.08810

  • Corbett-Davies S, Goel S (2018) The measure and mismeasure of fairness: a critical review of fair machine learning. arXiv:1808.00023

  • Dabbish L, Stuart C, Tsay J, Herbsleb J (2012) Social coding in github: transparency and collaboration in an open software repository. In: Proceedings of the ACM 2012 conference on computer supported cooperative work, pp 1277–1286

  • Dastin J (2023) Amazon scraps secret AI recruiting tool that showed bias against women. In: Ethics of data and analytics. Auerbach Publications, pp 296–299

  • Devlin J, Chang M-W, Lee K, Toutanova K (2019) BERT: pre-training of deep bidirectional transformers for language understanding. In: Proceedings of the 2019 conference of the north American chapter of the association for computational linguistics: human language technologies, vol 1 (Long and Short Papers). Association for Computational Linguistics, Minneapolis, pp 4171–4186

  • Erlenhov L, de Oliveira Neto FG, Scandariato R, Leitner P (2019) Current and future bots in software development. In: 2019 IEEE/ACM 1st international workshop on bots in software engineering (BotSE). IEEE, pp 7–11

  • Feng Z, Chatterjee A, Sarma A, Ahmed I (2022) Implicit mentoring: the unacknowledged developer efforts in open source. arXiv:2202.11300

  • Github rest api documentation. https://docs.github.com/en/rest. Accessed 19 Aug 2022

  • Harrison DA, Klein KJ (2007) What’s the difference? Diversity constructs as separation, variety, or disparity in organizations. Acad Manag Rev 32(4):1199–1228

    Article  Google Scholar 

  • Hern A. Twitter apologises for ’racist’ image-cropping algorithm

  • Hill K (2020) Wrongfully accused by an algorithm. In: Ethics of data and analytics. Auerbach Publications, pp 138–142

  • Huang Y, Leach K, Sharafi Z, McKay N, Santander T, Weimer W (2020) Biases and differences in code review using medical imaging and eye-tracking: genders, humans, and machines. In: Proceedings of the 28th ACM joint meeting on European software engineering conference and symposium on the foundations of software engineering, pp 456–468

  • Jackson SE, Joshi A (2004) Diversity in social context: a multi-attribute, multilevel analysis of team diversity and sales performance. J Org Behav Int J Indust Occup Org Psychol Behav 25(6):675–702

    Google Scholar 

  • Jiang J, He J-H, Chen X-Y (2015) Coredevrec: automatic core member recommendation for contribution evaluation. J Comput Sci Technol 30:998–1016

    Article  Google Scholar 

  • Johnson B, Bartola J, Angell R, Keith K, Witty S, Giguere SJ, Brun Y (2020) Fairkit, fairkit, on the wall, who’s the fairest of them all? Supporting data scientists in training fair models. arXiv:2012.09951

  • Klare BF, Burge MJ, Klontz JC, Vorder Bruegge RW, Jain AK (2012) Face recognition performance: role of demographic information. IEEE Trans Info Forensic Sec 7(6):1789–1801

    Article  Google Scholar 

  • Kubernetes ci bot. https://github.com/k8s-ci-robot. Accessed 19 Aug 2022

  • Lebeuf C, Storey M-A, Zagalsky A (2017) Software bots. IEEE Softw 35(1):18–23

    Article  Google Scholar 

  • Lebeuf C, Zagalsky A, Foucault M, Storey M-A (2019) Defining and classifying software bots: a faceted taxonomy. In: 2019 IEEE/ACM 1st international workshop on bots in software engineering (BotSE). IEEE, pp 1–6

  • Ledford H (2019) Millions of black people affected by racial bias in health-care algorithms. Nature 574(7780):608–610

    Article  Google Scholar 

  • Lin T-Y, Goyal P, Girshick R, He K, Dollár P (2017) Focal loss for dense object detection. In: Proceedings of the IEEE international conference on computer vision, pp 2980–2988

  • Martin N (2018) Are AI hiring programs eliminating bias or making it worse? Forbes. https://www.forbes.com/sites/nicolemartin1/2018/12/13/are-ai-hiring-programs-eliminating-bias-or-making-it-worse/. Accessed 13 Dec 2018

  • Monperrus M, Urli S, Durieux T, Martinez M, Baudry B, Seinturier L (2019) Repairnator patches programs automatically. Ubiquity 2019(July):1–12

    Article  Google Scholar 

  • Murphy-Hill E, Dicker J, Horvath A, Hodges MM, Egelman CD, Weingart LR, Jaspan C, Green C, Chen N (2023) Systemic gender inequities in who reviews code. Proc ACM Human-Comput Inter 7(CSCW1):1–59

    Article  Google Scholar 

  • Murphy-Hill E, Jaspan C, Egelman C, Cheng L (2022) The pushback effects of race, ethnicity, gender, and age in code review. Commun ACM 65(3):52–57

    Article  Google Scholar 

  • Nadri R, Rodriguez-Perez G, Nagappan M (2021) On the relationship between the developer’s perceptible race and ethnicity and the evaluation of contributions in OSS. IEEE Trans Softw Eng

  • Nadri R, Rodriguez-Perez G, Nagappan M (2021) Insights into nonmerged pull requests in github: is there evidence of bias based on perceptible race? IEEE Softw 38(2):51–57

    Article  Google Scholar 

  • Namsor tool. https://www.namsor.com/. Accessed 19 Aug 2022

  • Obermeyer Z, Powers B, Vogeli C, Mullainathan S (2019) Dissecting racial bias in an algorithm used to manage the health of populations. Science 366(6464):447–453

    Article  Google Scholar 

  • Peng Z, Yoo J, Xia M, Kim S, Ma X (2018) Exploring how software developers work with mention bot in Github. In: Proceedings of the sixth international symposium of Chinese CHI, pp 152–155

  • Rodríguez-Pérez G, Nadri R, Nagappan M (2021) Perceived diversity in software engineering: a systematic literature review. Empir Softw Eng 26(5):1–38

    Article  Google Scholar 

  • Sebo P (2021) Performance of gender detection tools: a comparative study of name-to-gender inference services. J Med Library Assoc JMLA 109(3):414

    Google Scholar 

  • Steinmacher I, Pinto G, Wiese IS, Gerosa MA (2018) Almost there: a study on quasi-contributors in open source software projects. In: Proceedings of the 40th international conference on software engineering, pp 256–266

  • Storey M-A, Zagalsky A (2016) Disrupting developer productivity one bot at a time. In: Proceedings of the 2016 24th ACM SIGSOFT international symposium on foundations of software engineering, pp 928–931

  • Teachman JD (1980) Analysis of population diversity: measures of qualitative variation. Sociol Methods Res 8(3):341–362

    Article  Google Scholar 

  • Tensorflowbutler. https://github.com/tensorflowbutler. Accessed 19 Aug 2022

  • Terrell J, Kofink A, Middleton J, Rainear C, Murphy-Hill ER, Parnin C (2016) Gender bias in open source: pull request acceptance of women versus men. PeerJ Prepr 4:e1733

  • Vasilescu B, Posnett D, Ray B, van den Brand MGJ, Serebrenik A, Devanbu P, Filkov V (2015) Gender and tenure diversity in github teams. In: Proceedings of the 33rd annual ACM conference on human factors in computing systems. ACM, New York, pp 3789–3798

  • Vaswani A, Shazeer N, Parmar N, Uszkoreit J, Jones L, Gomez AN, Kaiser Ł, Polosukhin I (2017) Attention is all you need. Adv Neural Info Process Syst 30

  • Vscode triage bot. https://github.com/vscode-triage-bot. Accessed 19 Aug 2022

  • Web scraper - the #1 web scraping tool. https://webscraper.io/. Accessed 19 Aug 2022

  • Wessel M, De Souza BM, Steinmacher I, Wiese IS, Polato I, Chaves AP, Gerosa MA (2018) The power of bots: characterizing and understanding bots in OSS projects. Proc ACM Human-Comput Inter 2(CSCW):1–19

  • Wessel M, Serebrenik A, Wiese I, Steinmacher I, Gerosa MA (2020) Effects of adopting code review bots on pull requests to OSS projects. In: 2020 IEEE international conference on software maintenance and evolution (ICSME). IEEE, pp 1–11

  • Wessel M, Serebrenik A, Wiese I, Steinmacher I, Gerosa MA (2020) What to expect from code review bots on Github? A survey with OSS maintainers. In: Proceedings of the 34th Brazilian symposium on software engineering, pp 457–462

  • Wessel M, Steinmacher I, Wiese I, Gerosa MA (2019) Should i stale or should i close? An analysis of a bot that closes abandoned issues and pull requests. In: 2019 IEEE/ACM 1st international workshop on bots in software engineering (BotSE). IEEE, pp 38–42

  • Wessel M, Wiese I, Steinmacher I, Gerosa MA (2021) Don’t disturb me: challenges of interacting with software bots on open source software projects. Proc ACM Human-Comput Inter 5(CSCW2):1–21

    Article  Google Scholar 

  • Wohlin C, Runeson P, Höst M, Ohlsson MC, Regnell B, Wesslén A (2012) Experimentation in software engineering. Springer Science & Business Media, Berlin

    Book  Google Scholar 

  • Wolf MJ, Miller K, Grodzinsky FS (2017) Why we should have seen that coming: comments on microsoft’s tay" experiment," and wider implications. Acm Sigcas Comput Soc 47(3):54–64

    Article  Google Scholar 

  • Wyrich M, Bogner J (2019) Towards an autonomous bot for automatic source code refactoring. In: 2019 IEEE/ACM 1st international workshop on bots in software engineering (BotSE). IEEE, pp 24–28

  • Zhang BH, Lemoine B, Mitchell M (2018) Mitigating unwanted biases with adversarial learning. In: Proceedings of the 2018 AAAI/ACM conference on AI, ethics, and society, pp 335–340

  • Zhang JM, Harman M (2021) “ignorance and prejudice” in software fairness. In: 2021 IEEE/ACM 43rd international conference on software engineering (ICSE). IEEE, pp 1436–1447

Download references

Author information

Authors and Affiliations

  1. University of Waterloo, Waterloo, ON, Canada

    Aniruddhan Murali, Gaurav Sahu, Kishanthan Thangarajah, Brian Zimmerman & Meiyappan Nagappan

  2. University of British Columbia, Kelowna, BC, Canada

    Gema Rodríguez-Pérez

Authors
  1. Aniruddhan Murali
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Gaurav Sahu
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Kishanthan Thangarajah
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Brian Zimmerman
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Gema Rodríguez-Pérez
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Meiyappan Nagappan
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding authors

Correspondence to Gema Rodríguez-Pérez or Meiyappan Nagappan.

Ethics declarations

Conflicts of interests

The authors declared that they have no conflict of interest.

Additional information

Communicated by: Brittany Johnson and Justin Smith

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

This article belongs to the Topical Collection: Special Issue on Equitable Data and Technology.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Murali, A., Sahu, G., Thangarajah, K. et al. Diversity in issue assignment: humans vs bots. Empir Software Eng 29, 37 (2024). https://doi.org/10.1007/s10664-023-10424-6

Download citation

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s10664-023-10424-6

Keywords

Search

Navigation