Nasif Imtiaz
ABSTRACT
Many software engineering researchers use sentiment and politeness analysis tools to study the emotional environment within collaborative software development. However, papers that use these tools rarely establish their reliability. In this paper, we evaluate popular existing tools for sentiment and politeness detection over a dataset of 589 manually rated GitHub comments that represent developer discussions. We also develop a coding scheme on how to quantify politeness for conversational texts found on collaborative platforms. We find that not only do the tools have a low agreement with human ratings on sentiment and politeness, human raters also have a low agreement among themselves.
- Toufique Ahmed, Amiangshu Bosu, Anindya Iqbal, and Shahram Rahimi. 2017. SentiCR: a customized sentiment analysis tool for code review interactions. In Proceedings of the 32nd IEEE/ACM International Conference on Automated Software Engineering. IEEE Press, 106--111. Google Scholar
Digital Library
- Steven Bird, Ewan Klein, and Edward Loper. 2009. Natural language processing with Python: analyzing text with the natural language toolkit. "O'Reilly Media, Inc.". Google ScholarDigital Library
- Cássio Castaldi Araujo Blaz and Karin Becker. 2016. Sentiment analysis in tickets for it support. In Mining Software Repositories (MSR), 2016 IEEE/ACM 13th Working Conference on. IEEE, 235--246. Google ScholarDigital Library
- Chris Brown, Justin Middleton, Esha Sharma, and Emerson Murphy-Hill. {n. d.}. How Software Users Recommend Tools to Each Other. ({n. d.}).Google Scholar
- Penelope Brown and Stephen C Levinson. 1987. Politeness: Some universals in language usage. Vol. 4. Cambridge university press.Google ScholarCross Ref
- Fabio Calefato, Filippo Lanubile, Federico Maiorano, and Nicole Novielli. 2017. Sentiment Polarity Detection for Software Development. Empirical Software Engineering (2017), 1--31. Google ScholarDigital Library
- Wikipedia contributors. 2018. Politeness --- Wikipedia, The Free Encyclopedia. (2018). https://en.wikipedia.org/wiki/Politeness {Online; accessed 18 Jan, 2018}.Google Scholar
- Jonathan Culpeper. 1996. Towards an anatomy of impoliteness. Journal of pragmatics 25, 3 (1996), 349--367.Google ScholarCross Ref
- Bill Curtis, Herb Krasner, and Neil Iscoe. 1988. A field study of the software design process for large systems. Commun. ACM 31, 11 (1988), 1268--1287. Google ScholarDigital Library
- Cristian Danescu-Niculescu-Mizil, Moritz Sudhof, Dan Jurafsky, Jure Leskovec, and Christopher Potts. 2013. A computational approach to politeness with application to social factors. arXiv preprint arXiv:1306.6078 (2013).Google Scholar
- Prasun Dewan. 2015. Towards emotion-based collaborative software engineering. In Proceedings of the Eighth International Workshop on Cooperative and Human Aspects of Software Engineering. IEEE Press, 109--112. Google ScholarDigital Library
- Felipe Ebert, Fernando Castor, Nicole Novielli, and Alexander Serebrenik. 2017. Confusion detection in code reviews. In Software Maintenance and Evolution (ICSME), 2017 IEEE International Conference on. IEEE, 549--553.Google ScholarCross Ref
- Daviti Gachechiladze, Filippo Lanubile, Nicole Novielli, and Alexander Serebrenik. 2017. Anger and its direction in collaborative software development. In Proceedings of the 39th International Conference on Software Engineering: New Ideas and Emerging Results Track. IEEE Press, 11--14. Google ScholarDigital Library
- Michael Gamon, Anthony Aue, Simon Corston-Oliver, and Eric Ringger. 2005. Pulse: Mining customer opinions from free text. Lecture notes in computer science 3646 (2005), 121--132. Google ScholarDigital Library
- David Garcia, Marcelo Serrano Zanetti, and Frank Schweitzer. 2013. The role of emotions in contributors activity: A case study on the gentoo community. In Cloud and green computing (CGC), 2013 third international conference on. IEEE, 410--417. Google ScholarDigital Library
- Georgios Gousios, Bogdan Vasilescu, Alexander Serebrenik, and Andy Zaidman. 2014. Lean GHTorrent: GitHub data on demand. In Proceedings of the 11th working conference on mining software repositories. ACM, 384--387. Google ScholarDigital Library
- Daniel Graziotin, Xiaofeng Wang, and Pekka Abrahamsson. 2014. Happy software developers solve problems better: psychological measurements in empirical software engineering. PeerJ 2 (2014), e289.Google ScholarCross Ref
- Emitza Guzman, David Azócar, and Yang Li. 2014. Sentiment analysis of commit comments in GitHub: an empirical study. In Proceedings of the 11th Working Conference on Mining Software Repositories. ACM, 352--355. Google ScholarDigital Library
- Emitza Guzman and Bernd Bruegge. 2013. Towards emotional awareness in software development teams. In Proceedings of the 2013 9th joint meeting on foundations of software engineering. ACM, 671--674. Google ScholarDigital Library
- Md Rakibul Islam and Minhaz F Zibran. 2016. Towards understanding and exploiting developers' emotional variations in software engineering. In Software Engineering Research, Management and Applications (SERA), 2016 IEEE 14th International Conference on. IEEE, 185--192.Google Scholar
- Robbert Jongeling, Proshanta Sarkar, Subhajit Datta, and Alexander Serebrenik. 2017. On negative results when using sentiment analysis tools for software engineering research. Empirical Software Engineering (2017), 1--42. Google ScholarDigital Library
- Iftikhar Ahmed Khan, Willem-Paul Brinkman, and Robert M Hierons. 2011. Do moods affect programmersâÃŹ debug performance? Cognition, Technology & Work 13, 4 (2011), 245--258. Google ScholarDigital Library
- J Richard Landis and Gary G Koch. 1977. The measurement of observer agreement for categorical data. biometrics (1977), 159--174.Google Scholar
- Bin Lin, Fiorella Zampetti, Gabriele Bavota, Massimiliano Di Penta, Michele Lanza, and Rocco Oliveto. 2018. Sentiment Analysis for So ware Engineering: How Far Can We Go? (2018).Google Scholar
- Saif Mohammad. 2016. A Practical Guide to Sentiment Annotation: Challenges and Solutions.. In WASSA@ NAACL-HLT. 174--179.Google Scholar
- Alessandro Murgia, Parastou Tourani, Bram Adams, and Marco Ortu. 2014. Do developers feel emotions? an exploratory analysis of emotions in software artifacts. In Proceedings of the 11th working conference on mining software repositories. ACM, 262--271. Google ScholarDigital Library
- Nicole Novielli, Fabio Calefato, and Filippo Lanubile. 2014. Towards discovering the role of emotions in stack overflow. In Proceedings of the 6th international workshop on social software engineering. ACM, 33--36. Google ScholarDigital Library
- Nicole Novielli, Fabio Calefato, and Filippo Lanubile. 2015. The challenges of sentiment detection in the social programmer ecosystem. In Proceedings of the 7th International Workshop on Social Software Engineering. ACM, 33--40. Google ScholarDigital Library
- Marco Ortu, Bram Adams, Giuseppe Destefanis, Parastou Tourani, Michele Marchesi, and Roberto Tonelli. 2015. Are bullies more productive?: empirical study of affectiveness vs. issue fixing time. In Proceedings of the 12th Working Conference on Mining Software Repositories. IEEE Press, 303--313. Google ScholarDigital Library
- Marco Ortu, Giuseppe Destefanis, Mohamad Kassab, Steve Counsell, Michele Marchesi, and Roberto Tonelli. 2015. Would you mind fixing this issue?. In International Conference on Agile Software Development. Springer, 129--140.Google ScholarCross Ref
- Marco Ortu, Alessandro Murgia, Giuseppe Destefanis, Parastou Tourani, Roberto Tonelli, Michele Marchesi, and Bram Adams. 2016. The emotional side of software developers in JIRA. In 2016 IEEE/ACM 13th Working Conference on Mining Software Repositories (MSR). IEEE, 480--483. Google ScholarDigital Library
- Bo Pang, Lillian Lee, et al. 2008. Opinion mining and sentiment analysis. Foundations and Trends® in Information Retrieval 2, 1--2 (2008), 1--135. Google ScholarDigital Library
- Brian Parkinson, Peter Totterdell, Rob B Briner, and SA Reynolds. 1996. Changing moods: The psychology of mood and mood regulation. Longman.Google Scholar
- Daniel Pletea, Bogdan Vasilescu, and Alexander Serebrenik. 2014. Security and emotion: sentiment analysis of security discussions on GitHub. In Proceedings of the 11th working conference on mining software repositories. ACM, 348--351. Google ScholarDigital Library
- Athanasios-Ilias Rousinopoulos, Gregorio Robles, and Jesús M González-Barahona. 2014. SENTIMENT ANALYSIS OF FREE/OPEN SOURCE DEVELOPERS: PRELIMINARY FINDINGS FROM A CASE STUDY/ANÁLISE DE SENTIMENTOS DE DESENVOLVEDORES DE SOFTWARE LIVRE: ACHADOS PRELIMINARES DE UM ESTUDO DE CASO. Revista Electronica de Sistemas de Informaçao 13, 2 (2014), 1.Google Scholar
- Vinayak Sinha, Alina Lazar, and Bonita Sharif. 2016. Analyzing developer sentiment in commit logs. In Proceedings of the 13th International Conference on Mining Software Repositories. ACM, 520--523. Google ScholarDigital Library
- Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Ng, and Christopher Potts. 2013. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 conference on empirical methods in natural language processing. 1631--1642.Google Scholar
- Igor Steinmacher, Tayana Conte, Marco Aurélio Gerosa, and David Redmiles. 2015. Social barriers faced by newcomers placing their first contribution in open source software projects. In Proceedings of the 18th ACM conference on Computer supported cooperative work & social computing. ACM, 1379--1392. Google ScholarDigital Library
- Margaret-Anne Storey, Christoph Treude, Arie van Deursen, and Li-Te Cheng. 2010. The impact of social media on software engineering practices and tools. In Proceedings of the FSE/SDP workshop on Future of software engineering research. ACM, 359--364. Google ScholarDigital Library
- Mike Thelwall, Kevan Buckley, Georgios Paltoglou, Di Cai, and Arvid Kappas. 2010. Sentiment strength detection in short informal text. Journal of the Association for Information Science and Technology 61, 12 (2010), 2544--2558. Google ScholarDigital Library
- Jason Tsay, Laura Dabbish, and James Herbsleb. 2014. Let's talk about it: evaluating contributions through discussion in GitHub. In Proceedings of the 22nd ACM SIGSOFT international symposium on foundations of software engineering. ACM, 144--154. Google ScholarDigital Library
- Ning Wang, W Lewis Johnson, Richard E Mayer, Paola Rizzo, Erin Shaw, and Heather Collins. 2008. The politeness effect: Pedagogical agents and learning outcomes. International Journal of Human-Computer Studies 66, 2 (2008), 98--112. Google ScholarDigital Library
Recommendations
A query-based multi-document sentiment summarizer
CIKM '11: Proceedings of the 20th ACM international conference on Information and knowledge managementReview websites, such as Epinions.com, which offer users a platform to share their opinions on diverse products and services, provide a valuable source of opinion-rich information. Browsing through archived reviews to locate different opinions on a ...
Exploring how politeness impacts the user experience of chatbots for mental health support
AbstractPoliteness is important in human–human interaction when asking people to engage in sensitive conversations. If politeness manifests similarly in human–chatbot interaction, it may play an important role in the design of sensitive chatbot ...
Highlights- Politeness can both positively and negatively impact the chatbot user experience.
- The Personal politeness chatbot was experienced as caring and encouraging.
- The Passive politeness chatbot was experienced as too apologetic and ...
Effect of politeness strategies in dialogue on negotiation outcomes
IVA '21: Proceedings of the 21st ACM International Conference on Intelligent Virtual AgentsNegotiation is a social interaction aimed at reaching a mutually beneficial agreement among all participants in a conflict situation. Unfortunately, parties often find negotiations threatening or aversive, undermining the chances of reaching good ...
Comments