Mohamed Amine Batoun
Abstract
The pull request mechanism is commonly used to propose source code modifications and get feedback from the community before merging them into a software repository. On GitHub, practitioners can provide feedback on a pull request by either commenting on the pull request or simply reacting to it using a set of pre-defined GitHub reactions, i.e., “Thumbs-up”, “Laugh”, “Hooray”, “Heart”, “Rocket”, “Thumbs-down”, “Confused”, and “Eyes”. While a large number of prior studies investigated how to improve different software engineering activities (e.g., code review and integration) by investigating the feedback on pull requests, they focused only on pull requests’ comments as a source of feedback. However, the GitHub reactions, according to our preliminary study, contain feedback that is not manifested within the comments of pull requests. In fact, our preliminary analysis of six popular projects shows that a median of 100% of the practitioners who reacted to a pull request did not leave any comment suggesting that reactions can be a unique source of feedback to further improve the code review and integration process.
To help future studies better leverage reactions as a feedback mechanism, we conduct an empirical study to understand the usage of GitHub reactions and understand their promises and limitations. We investigate in this article how reactions are used, when and who use them on what types of pull requests, and for what purposes. Our study considers a quantitative analysis on a set of 380 k reactions on 63 k pull requests of six popular open-source projects on GitHub and three qualitative analyses on a total number of 989 reactions from the same six projects. We find that the most common used GitHub reactions are the positive ones (i.e., “Thumbs-up”, “Hooray”, “Heart”, “Rocket”, and “Laugh”). We observe that reactors use positive reactions to express positive attitude (e.g., approval, appreciation, and excitement) on the proposed changes in pull requests. A median of just 1.95% of the used reactions are negative ones, which are used by reactors who disagree with the proposed changes for six reasons, such as feature modifications that might have more downsides than upsides or the use of the wrong approach to address certain problems. Most (a median of 78.40%) reactions on a pull request come before the closing of the corresponding pull requests. Interestingly, we observe that non-contributors (i.e., outsiders who potentially are the “end-users” of the software) are also active on reacting to pull requests. On top of that, we observe that core contributors, peripheral contributors, casual contributors and outsiders have different behaviors when reacting to pull requests. For instance, most core contributors react in the early stages of a pull request, while peripheral contributors, casual contributors and outsiders react around the closing time or, in some cases, after a pull request is merged. Contributors tend to react to the pull request’s source code, while outsiders are more concerned about the impact of the pull request on the end-user experience. Our findings shed light on common patterns of GitHub reactions usage on pull requests and provide taxonomies about the intention of reactors, which can inspire future studies better leverage pull requests’ reactions.
- [1] . 2018. Sentiment analysis for software engineering: How far can we go? In Proceedings of the 2018 International Conference on Software Engineering. 94–104.Google Scholar
- [2] . 2017. Using millions of emoji occurrences to learn any-domain representations for detecting sentiment, emotion and sarcasm. Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing.Google Scholar
- [3] . 2016. Add Reactions to Pull Requests, Issues, and Comments. Retrieved from https://github.blog/2016-03-10-add-reactions-to-pull-requests-issues-and-comments/. Accessed 30 May 2023.Google Scholar
- [4] . 2014. Security and emotion: Sentiment analysis of security discussions on GitHub. In Proceedings of the 2014 Working Conference on Mining Software Repositories. 348–351.Google Scholar
- [5] . 2017. Anger and its direction in collaborative software development. In Proceedings of the 2017 IEEE/ACM International Conference on Software Engineering: New Ideas and Emerging Technologies Results Track. 11–14.Google Scholar
- [6] . 2010. Enhanced sentiment learning using Twitter hashtags and smileys. In Proceedings of the Coling 2010: Posters. 241–249.Google Scholar
- [7] . 2020. Achieving reliable sentiment analysis in the software engineering domain using BERT. In Proceedings of the 2020 IEEE International Conference on Software Maintenance and Evolution. 162–173.Google Scholar
- [8] . 2019. Analysis of newcomers activity in communicative posts on GitHub. In Proceedings of the 2019 International Conference on Digital Transformation and Global Society. 452–460.Google Scholar
- [9] . 2017. An empirical comparison of developer retention in the RubyGems and Npm software ecosystems. Innovations in Systems and Software Engineering 13 (2017), 101–115.Google Scholar
- [10] . 2013. Towards emotional awareness in software development teams. In Proceedings of the 2013 Joint Meeting on Foundations of Software Engineering. 671–674.Google Scholar
- [11] . 2019. Understanding the effect of developer sentiment on fix-inducing changes: An exploratory study on GitHub pull requests. In Proceedings of the 2019 Asia-Pacific Software Engineering Conference. 514–521.Google Scholar
- [12] . 2019. On the abandonment and survival of open source projects: An empirical investigation. In Proceedings of the 2019 ACM/IEEE International Symposium on Empirical Software Engineering and Measurement. 1–12.Google Scholar
- [13] . 2016. A novel approach for estimating truck factors. In Proceedings of the 2016 IEEE 24th International Conference on Program Comprehension. 1–10.Google Scholar
- [14] . 2016. More common than you think: An in-depth study of casual contributors. In Proceedings of the 2016 IEEE 23rd International Conference on Software Analysis, Evolution, and Reengineering. 112–123.Google Scholar
- [15] . 2019. Beyond textual issues: Understanding the usage and impact of GitHub reactions. In Proceedings of the 2019 Brazilian Symposium on Software Engineering. 397–406.Google Scholar
- [16] . 2006. Core and periphery in free/libre and open source software team communications. In Proceedings of the 2006 Annual Hawaii International Conference on System Sciences. 118a–118a.Google Scholar
- [17] . 2021. Exploiting the unique expression for improved sentiment analysis in software engineering text. In Proceedings of the 2021 IEEE/ACM International Conference on Program Comprehension. 149–159.Google Scholar
- [18] . 2004. Reliability in content analysis: Some common misconceptions and recommendations. Human Communication Research 30, 3 (2004), 411–433.Google Scholar
- [19] . 2018. A first look at emoji usage on GitHub: An empirical study. arXiv:1812.04863. Retrieved from https://arxiv.org/abs/1812.04863.Google Scholar
- [20] . 2018. On the use of emoticons in open source software development. In Proceedings of the 2018 ACM/IEEE International Symposium on Empirical Software Engineering and Measurement. 1–4.Google Scholar
- [21] . 2021. StackEmo: Towards enhancing user experience by augmenting stack overflow with emojis. In Proceedings of the 2021 ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering. 1550–1554.Google Scholar
- [22] . 2015. Are bullies more productive? empirical study of affectiveness vs. issue fixing time. In Proceedings of the 2015 IEEE/ACM Working Conference on Mining Software Repositories. 303–313.Google Scholar
- [23] . 2018. Analyzing the impact of feedback in GitHub on the software developer’s mood. In Proceedings of the 2018 International Conference on Software Engineering and Knowledge Engineering. 445–450.Google Scholar
- [24] . 2018. Which contributions predict whether developers are accepted into GitHub teams. In Proceedings of the 2018 IEEE/ACM International Conference on Mining Software Repositories. 403–413.Google ScholarDigital Library
- [25] . 2016. Mining valence, arousal, and dominance: Possibilities for detecting burnout and productivity?. In Proceedings of the 2016 International Conference on Mining Software Repositories. 247–258.Google Scholar
- [26] . 2020. Analysis of Textual and Non-Textual Sources of Sentiment in Github. Master’s thesis. University of Waterloo, Waterloo, Ontario, Canada.Google Scholar
- [27] . 2018. A benchmark study on sentiment analysis for software engineering research. In Proceedings of the 2018 IEEE/ACM International Conference on Mining Software Repositories. 364–375.Google Scholar
- [28] . 2019. A systematic review of emoji: Current research and future perspectives. Frontiers in Psychology 10 (2019), 2221.Google Scholar
- [29] . 2019. Improved the performance of the K-means cluster using the sum of squared error (SSE) optimized by using the elbow method. In Proceedings of the Journal of Physics: Conference Series. 012–015.Google Scholar
- [30] . 2015. Choosing your weapons: On sentiment analysis tools for software engineering research. In Proceedings of the 2015 IEEE International Conference on Software Maintenance and Evolution. 531–535.Google Scholar
- [31] . 2017. Sentiment analysis of Travis CI builds. In Proceedings of the 2017 IEEE/ACM International Conference on Mining Software Repositories. 459–462.Google Scholar
- [32] . 2020. Optimal (Weighted) Univariate Clustering. https://search.r-project.org/CRAN/refmans/Ckmeans.1d.dp/html/Ckmeans.1d.dp.html. Accessed 30 May 2023.Google Scholar
- [33] . 2019. Leveraging change intents for characterizing and identifying large-review-effort changes. In Proceedings of the 2019 International Conference on Predictive Models and Data Analytics in Software Engineering. 46–55.Google Scholar
- [34] . 2019. How do I feel? Identifying emotional expressions on Facebook reactions using clustering mechanism. IEEE Access 7 (2019), 53909–53921.Google Scholar
- [35] . 2021. More than react: Investigating the role of emoji reaction in GitHub pull requests. arXiv:2108.08094. Retrieved from https://arxiv.org/abs/2108.08094.Google Scholar
- [36] . 2020. Sentiment analysis for software engineering: How far can pre-trained transformer models go? In Proceedings of the 2020 IEEE International Conference on Software Maintenance and Evolution. 70–80.Google Scholar
- [37] . 2014. Influence of social and technical factors for evaluating contribution in GitHub. In Proceedings of the 2014 International Conference on Software Engineering. 356–366.Google ScholarDigital Library
- [38] . 2013. Emotions in the software development process. In Proceedings of the 2013 International Conference on Human System Interactions. 518–523.Google ScholarCross Ref
- [39] . 2016. Towards the participant observation of emotions in software development teams. In Proceedings of the 2016 Federated Conference on Computer Science and Information Systems. 1545–1548.Google ScholarCross Ref
- [40] . 2021. RIP emojis and words to contextualize mourning on Twitter. In Proceedings of the 2021 ACM Conference on Hypertext and Social Media. 257–263.Google Scholar
- [41] . 2016. Earec: Leveraging expertise and authority for pull-request reviewer recommendation in GitHub. In Proceedings of the 2016 IEEE/ACM International Workshop on CrowdSourcing in Software Engineering. 29–35.Google ScholarDigital Library
- [42] . 2014. Who should review this pull-request: Reviewer recommendation to expedite crowd collaboration. In Proceedings of the 2014 Asia-Pacific Software Engineering Conference. 335–342.Google ScholarDigital Library
- [43] . 2018. Through a gender lens: Learning usage patterns of emojis from large-scale android users. In Proceedings of the 2018 World Wide Web Conference. 763–772.Google Scholar
- [44] . 2021. Emoji-powered sentiment and emotion detection from software developers’ communication data. ACM Transactions on Software Engineering and Methodology 30, 2 (2021), 1–48.Google Scholar
- [45] . 2019. SEntiMoji: An emoji-powered learning approach for sentiment analysis in software engineering. In Proceedings of the 2019 ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering. 841–852.Google Scholar
- [46] . 2021. Predicting community smells’ occurrence on individual developers by sentiments. In Proceedings of the 2021 IEEE/ACM International Conference on Program Comprehension. 230–241.Google Scholar
Index Terms
- An Empirical Study on GitHub Pull Requests’ Reactions
Recommendations
An insight into the pull requests of GitHub
MSR 2014: Proceedings of the 11th Working Conference on Mining Software RepositoriesGiven the increasing number of unsuccessful pull requests in GitHub projects, insights into the success and failure of these requests are essential for the developers. In this paper, we provide a comparative study between successful and unsuccessful ...
Interpersonal Trust in OSS: Exploring Dimensions of Trust in GitHub Pull Requests
ICSE-NIER '23: Proceedings of the 45th International Conference on Software Engineering: New Ideas and Emerging ResultsInterpersonal trust plays a crucial role in facilitating collaborative tasks, such as software development. While previous research recognizes the significance of trust in an organizational setting, there is a lack of understanding in how trust is ...
Finding Impact Factors for Rejection of Pull Requests on GitHub
ICNCC '18: Proceedings of the 2018 VII International Conference on Network, Communication and ComputingA pull request is an important method for code contributions in GitHub that will be submitted when the developers would like to merge their code changes from their local machine to the main repository on which all source code in the project are stored. ...
Comments