research-article

MarValous: machine learning based detection of emotions in the valence-arousal space in software engineering text

Authors:

Md Rakibul Islam,

Md Kauser Ahmmed,

Minhaz F. ZibranAuthors Info & Claims

SAC '19: Proceedings of the 34th ACM/SIGAPP Symposium on Applied Computing

Pages 1786 - 1793

https://doi.org/10.1145/3297280.3297455

Published: 08 April 2019 Publication History

Abstract

Emotion analysis in text has drawn recent interests in the software engineering (SE) community. Existing domain-independent techniques for automated emotion/sentiment analysis perform poorly when operated on SE text. Thus, a few SE domain-specific tools are recently developed for detecting sentimental polarities (e.g., positivity, negativity). But, for capturing individual emotional states such as excitation, stress, depression, and relaxation, there is only one recent tool named DEVA, which uses a lexicon-based approach.

We have developed MarValous, the first Machine Learning based tool for improved detection of the aforementioned emotional states in software engineering text. We evaluate MarValous using a dataset containing 5,122 comments collected from JIRA and Stack Overflow. From a quantitative evaluation, MarValous is found to have substantially outperformed DEVA achieving more than 83% precision and more than 79% recall.

References

[1]

Sept 2018. Download MarValous. https://figshare.com/s/a3308b7087df910db38f.

[2]

Verified: Aug 2018. The Social Media Glossary: 226 Essential Definitions. https://blog.hootsuite.com/social-media-glossary-definitions/.

[3]

Verified: Aug 2018. SyntaxNet: Neural Models of Syntax. https://github.com/tensorflow/models/tree/master/researeh/syntaxnet.

[4]

Verified: Aug 2018. TF-IDF. http://www.tfidf.com.

[5]

Verified: Sept 2018. Industrial-Strength Natural Language Processing. https://spacy.io.

[6]

Verified: Sept 2018. Natural Language Toolkit for Sentiment Analysis. http://www.nltk.org/api/nltk.sentiment.html.

[7]

Verified: Sept 2018. Scikit-learn: Machine Learning in Python. http://scikit-learn.org/stable/.

[8]

Verified: Sept 2018. Snowball. http://snowballstem.org.

[9]

Verified: Sept 2018. Stanford Core NLP Sentiment Annotator. http://stanfordnlp.github.io/CoreNLP/sentiment.html.

[10]

T. Ahmed, A. Bosu, A. Iqbal, and S. Rahimi. 2017. SentiCR: a customized sentiment analysis tool for code review interactions. In ASE. 106--111.

Digital Library

[11]

D. Alboaneen, H. Tianfield, and Y. Zhang. 2017. Sentiment Analysis via Multi-Layer Perceptron Trained by Meta-Heuristic Optimisation. In ICBD. 4630--4635.

[12]

N. Bettenburg, B. Adams, and A. Hassan. 2011. A Lightweight Approach to Uncover Technical Information in Unstructured Data. In ICPC. 185--188.

Digital Library

[13]

A. Bifet and E. Frank. 2010. Sentiment knowledge discovery in twitter streaming data. In DS. 1--15.

Digital Library

[14]

C. Blaz and K. Becker. 2016. Sentiment Analysis in Tickets for IT Support. In MSR. 235--246.

Digital Library

[15]

F. Calefato and F. Lanubile. 2016. Affective Trust as a Predictor of Successful Collaboration in Distributed Software Projects. In SEmotion. 3--5.

Digital Library

[16]

F. Calefato, F. Lanubile, F. Maiorano, and N. Novielli. 2017. Sentiment Polarity Detection for Software Development. Emp. Softw. Eng. (2017), 1--31.

Digital Library

[17]

F. Calefato, F. Lanubile, and N. Novielli. 2017. EmoTxt: A Toolkit for Emotion Recognition from Text. In ACII.

[18]

A. Ciurumelea, A. Schaufelbuhl, S. Panichella, and Harald Gall. 2017. Analyzing Reviews and Code of Mobile Apps for Better Release Planning. In SANER. 91--102.

[19]

A. D'Andrea, F. Ferri, P. Grifoni, and T. Guzzo. 2015. Approaches, Tools and Applications for Sentiment Analysis Implementation. International Journal of Computer Applications 125, 3 (2015), 26--33.

[20]

T. Dietterich. 1998. Approximate Statistical Tests for Comparing Supervised Classification Learning Algorithms. J. of Neural Comp. 10, 7 (1998), 1895--1923.

Digital Library

[21]

J. Ding, H. Sun, X. Wang, and X. Liu. 2018. Entity-Level Sentiment Analysis of Issue Comments. In SEmotion. 7--13.

Digital Library

[22]

T. Eerola and J. Vuoskoski. 2010. A Comparison of the Discrete and Dimensional Models of Emotion in Music. Psychology of Music 39, 1 (2010), 18--49.

[23]

B. Martínez; A. Rodrigo; R. Cantabrana; J. García and R. Alcaraz. 2016. Application of Entropy-Based Metrics to Identify Emotional Distress from Electroencephalographic Recordings. Entropy 18, 6 (2016).

[24]

D. Garcia, M. Zanetti, and F. Schweitzer. 2013. The Role of Emotions in Contributors Activity: A Case Study on the GENTOO Community. In CCGC. 410--417.

Digital Library

[25]

D. Graziotin, X. Wang, and P. Abrahamsson. 2013. Are Happy Developers more Productive? The Correlation of Affective States of Software Developers and their self-assessed Productivity. In PROFES. 50--64.

[26]

E. Guzman and B. Bruegge. 2013. Towards Emotional Awareness in Software Development Teams. In ESEC/FSE. 671--674.

Digital Library

[27]

M. Hasan, E. Rundensteiner, and E. Agu. 2014. EMOTEX: Detecting Emotions in Twitter Messages. In ASE. 27--31.

[28]

M. Islam and M. Zibran. 2016. Exploration and Exploitation of Developers' Sentimental Variations in Software Engineering. Internation Journal of Software Innovation 4, 4 (2016), 35--55.

Digital Library

[29]

M. Islam and M. Zibran. 2016. Towards Understanding and Exploiting Developers' Emotional Variations in Software Engineering. In SERA. 185--192.

[30]

M. Islam and M. Zibran. 2017. Leveraging Automated Sentiment Analysis in Software Engineering. In MSR. 203--214.

Digital Library

[31]

M. Islam and M. Zibran. 2018. DEVA: Sensing Emotions in the Valence Arousal Space in Software Engineering Text. In SAC. 1536--1543.

Digital Library

[32]

M. Islam and M. Zibran. 2018. Sentiment Analysis of Software Bug Related Commit Messages. In SEDE.

[33]

M. Islam and M. Zibran. 2018. SentiStrength-SE: Exploiting domain specificity for improved sentiment T analysis in software engineering text. Journal of System and Software 145 (2018), 125--146.

[34]

R. Jongeling, S. Datta, and A. Serebrenik. 2015. Choosing Your Weapons: On Sentiment Analysis Tools for Software Engineering Research. In ICSME. 531--535.

Digital Library

[35]

I. Khan, W. Brinkman, and R. Hierons. 2010. Do Moods Affect Programmers' Debug Performance? Cogn. Technol. Work 13, 4 (2010), 245--258.

Digital Library

[36]

L. Khreisat. 2006. Arabic Text Classification Using N-Gram Frequency Statistics A Comparative Study. In CDM. 78--82.

[37]

D. McDuff, A. Karlson, A. Kapoor, A. Roseway, and M. Czerwinski. 2012. AffectAura: An Intelligent System for Emotional Memory. In CHI. 849--858.

Digital Library

[38]

A. Murgia, M. Ortu, P. Tourani, and B. Adams. 2017. An Exploratory Qualitative and Quantitative Analysis of Emotions in Issue Report Comments of Open Source Systems. Empirical Software Engineering (2017), 1--44.

Digital Library

[39]

A. Murgia, P. Tourani, B. Adams, and M. Ortu. 2014. Do Developers Feel Emotions? An Exploratory Analysis of Emotions in Software Artifacts. In MSR. 261--271.

Digital Library

[40]

M. Nayebi, H. Farrahi, and G. Ruhe. 2017. Which Version Should be Released to the App Store?. In ESEM. 324--333.

Digital Library

[41]

N. Novielli, F. Calefato, and F. Lanubile. 2014. Towards Discovering the Role of Emotions in Stack Overflow. In SSE. 33--40.

Digital Library

[42]

N. Novielli, F. Calefato, and F. Lanubile. 2015. The Challenges of Sentiment Detection in the Social Programmer Ecosystem. In SSE. 33--40.

Digital Library

[43]

N. Novielli, F. Calefato, and F. Lanubile. 2018. A Gold Standard for Emotion Annotation in Stack Overflow. In MSR. 14--17.

Digital Library

[44]

F. Omran and C. Treude. 2017. Choosing an NLP Lib. for Analyzing Softw. Documentation: A Systematic Lit. Review and a Series of Exp. In MSR. 187--197.

Digital Library

[45]

M. Ortu, B. Adams, G. Destefanis, P. Tourani, M. Marchesi, and R. Tonelli. 2015. Are Bullies More Productive? Empirical Study of Affectiveness vs. Issue Fixing Time. In MSR. 303--313.

Digital Library

[46]

B. Pang, L. Lee, and S. Vaithyanathan. 2002. Thumbs up? Sentiment Classification Using Machine Learning Techniques. In EMNLP. 79--86.

Digital Library

[47]

M. Pennacchiotti and A. Popescu Democrats. 2011. Republicans and Starbucks Afficionados: User Classification in Twitter. In KDD. 430--438.

Digital Library

[48]

D. Pletea, B. Vasilescu, and A. Serebrenik. 2014. Security and Emotion: Sentiment Analysis of Security Discussions on GitHub. In MSR. 348--351.

Digital Library

[49]

L. Polanyi and A. Zaenen. 2006. Contextual Valence Shifters. Computing Attitude and Affect in Text: Theory and Applications 20 (2006), 1--10.

[50]

M. Raymer, W Punch, E Goodman, L Kuhn, and A Jain. 2000. Dimensionality reduction using genetic algo. IEEE Trans. on Evol. Comp. 4, 2 (2000), 164--171.

Digital Library

[51]

J. Russell and A. Mehrabian. 1977. Evidence for a Three-factor Theory of Emotions. Journal of Research in Personality 11, 3 (1977), 273--294.

[52]

R. Socher, A. Perelygin, J. Wu, J. Chuang, C. Manning, A. Ng, and C. Potts. 2013. Recursive Deep Models for Semantic Compositionality over a Sentiment Treebank. In EMNLP. 1631--1641.

[53]

M. Thelwall. 2013. Heart and Soul: Sentiment Strength Detection in the Social Web with Sentistrength,. In CyberEmotions. 1--14.

[54]

M. Thelwall. 2017. TensiStrength: Stress and relaxation magnitude detection for social media texts. Information Processing and Management 53, 1 (2017), 106--121.

[55]

P. Tourani and B. Adams. 2016. The Impact of Human Discussions on Just-in-time Quality Assurance. In SANER. 189--200.

[56]

Y. Wan and D. Gao. 2015. An Ensemble Sentiment Classification System of Twitter Data for Airline Services Analysis. In ICDMW. 1318--1325.

Digital Library

[57]

M. Wrobel. 2013. Emotions in the Software Development Process. In HSI. 518--523.

[58]

C. Yang, K. Lin, and H. Chen. 2009. Writer Meets Reader: Emo. Analysis of Soc. Media from both the Writer's and Reader's Perspectives. In WIIAT. 287--290.

Digital Library

[59]

G. Yang, S. Baek, J. Lee, and B. Lee. 2017. Analyzing Emotion Words to Predict Severity of Softw. Bugs: A Case Study of Open Source Proj. In SAC. 1280--1287.

Digital Library

[60]

D. Ye, Z. Xing, C. Foo, Z. Ang, J. Li, and N. Kapre. 2016. Software-Specific Named Entity Recognition in Software Engineering Social Content. In SANER. 90--101.

Cited By

Venigalla AChimalakonda S(2025)What are the emotions of developers towards deep learning documentation? — An exploratory study on Stack Overflow postsInformation and Software Technology10.1016/j.infsof.2024.107655179(107655)Online publication date: Mar-2025
https://doi.org/10.1016/j.infsof.2024.107655
Zhang TIrsan IThung FLo D(2024)Revisiting Sentiment Analysis for Software Engineering in the Era of Large Language ModelsACM Transactions on Software Engineering and Methodology10.1145/369700934:3(1-30)Online publication date: 24-Sep-2024
https://dl.acm.org/doi/10.1145/3697009
Imran MIzadi MDi Sorbo APanichella S(2024)Emotion Classification In Software Engineering Texts: A Comparative Analysis of Pre-trained Transformers Language ModelsProceedings of the Third ACM/IEEE International Workshop on NL-based Software Engineering10.1145/3643787.3648034(73-80)Online publication date: 20-Apr-2024
https://dl.acm.org/doi/10.1145/3643787.3648034
Show More Cited By

Index Terms

MarValous: machine learning based detection of emotions in the valence-arousal space in software engineering text
1. Software and its engineering
  1. Software creation and management
    1. Collaboration in software development
      1. Programming teams

Recommendations

DEVA: sensing emotions in the valence arousal space in software engineering text
SAC '18: Proceedings of the 33rd Annual ACM Symposium on Applied Computing

Existing tools for automated sentiment analysis in software engineering text suffer from either or both of two limitations. First, they are developed for non-technical domain and perform poorly when operated on software engineering text. Second, those ...
PanoEmo, a set of affective 360-degree panoramas: a psychophysiological study
Abstract
There is a significant increase in the use of virtual reality in scientific experiments in the fields of ergonomics, education, and psychology among others. Many researchers successfully provoked different affective states in participants in order ...
The Effect of Thermal Stimuli on the Emotional Perception of Images
CHI '16: Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems

Thermal stimulation is a feedback channel that has the potential to influence the emotional response of people to media such as images. While previous work has demonstrated that thermal stimuli might have an effect on the emotional perception of images, ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

SAC '19: Proceedings of the 34th ACM/SIGAPP Symposium on Applied Computing

April 2019

2682 pages

ISBN:9781450359337

DOI:10.1145/3297280

Conference Chairs:
Chih-Cheng Hung
Kennesaw State University, Marietta, Georgia
,
George A. Papadopoulos
University of Cyprus, Nicosia, Cyprus

Copyright © 2019 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGAPP: ACM Special Interest Group on Applied Computing

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 08 April 2019

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

SAC '19

Sponsor:

SIGAPP

SAC '19: The 34th ACM/SIGAPP Symposium on Applied Computing

April 8 - 12, 2019

Limassol, Cyprus

Acceptance Rates

Overall Acceptance Rate 1,650 of 6,669 submissions, 25%

Upcoming Conference

SAC '25

Sponsor:
sigapp

The 40th ACM/SIGAPP Symposium on Applied Computing

March 31 - April 4, 2025

Catania , Italy

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

15
Total Citations
View Citations
326
Total Downloads

Downloads (Last 12 months)29
Downloads (Last 6 weeks)4

Reflects downloads up to 05 Mar 2025

Other Metrics

View Author Metrics

Citations

Cited By

Venigalla AChimalakonda S(2025)What are the emotions of developers towards deep learning documentation? — An exploratory study on Stack Overflow postsInformation and Software Technology10.1016/j.infsof.2024.107655179(107655)Online publication date: Mar-2025
https://doi.org/10.1016/j.infsof.2024.107655
Zhang TIrsan IThung FLo D(2024)Revisiting Sentiment Analysis for Software Engineering in the Era of Large Language ModelsACM Transactions on Software Engineering and Methodology10.1145/369700934:3(1-30)Online publication date: 24-Sep-2024
https://dl.acm.org/doi/10.1145/3697009
Imran MIzadi MDi Sorbo APanichella S(2024)Emotion Classification In Software Engineering Texts: A Comparative Analysis of Pre-trained Transformers Language ModelsProceedings of the Third ACM/IEEE International Workshop on NL-based Software Engineering10.1145/3643787.3648034(73-80)Online publication date: 20-Apr-2024
https://dl.acm.org/doi/10.1145/3643787.3648034
Rabbi MChampa AZibran M(2024)Phishy? Detecting Phishing Emails Using Machine Learning and Natural Language ProcessingSoftware Engineering and Management: Theory and Application10.1007/978-3-031-55174-1_9(119-137)Online publication date: 3-May-2024
https://doi.org/10.1007/978-3-031-55174-1_9
Sajadi ADamevski KChatterjee PChandra SBlincoe KTonella P(2023)Towards Understanding Emotions in Informal Developer Interactions: A Gitter Chat StudyProceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering10.1145/3611643.3613084(2097-2101)Online publication date: 30-Nov-2023
https://dl.acm.org/doi/10.1145/3611643.3613084
Rabbi MChampa AZibran M(2023)Phishy? Detecting Phishing Emails Using ML and NLP2023 IEEE/ACIS 21st International Conference on Software Engineering Research, Management and Applications (SERA)10.1109/SERA57763.2023.10197758(77-83)Online publication date: 23-May-2023
https://doi.org/10.1109/SERA57763.2023.10197758
Li JLei YLi SZhou HYu YJia ZMa YWang T(2023)A Two-Stage Framework for Ambiguous Classification in Software Engineering2023 IEEE 34th International Symposium on Software Reliability Engineering (ISSRE)10.1109/ISSRE59848.2023.00070(275-286)Online publication date: 9-Oct-2023
https://doi.org/10.1109/ISSRE59848.2023.00070
Swillus MZaidman A(2023)Sentiment overflow in the testing stackJournal of Systems and Software10.1016/j.jss.2023.111804205:COnline publication date: 1-Nov-2023
https://dl.acm.org/doi/10.1016/j.jss.2023.111804
Tulili TCapiluppi ARastogi A(2023)Burnout in software engineeringInformation and Software Technology10.1016/j.infsof.2022.107116155:COnline publication date: 1-Mar-2023
https://dl.acm.org/doi/10.1016/j.infsof.2022.107116
Novielli NSerebrenik A(2023)Emotion Analysis in Software EcosystemsSoftware Ecosystems10.1007/978-3-031-36060-2_5(105-127)Online publication date: 26-May-2023
https://doi.org/10.1007/978-3-031-36060-2_5
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten