Can we predict useful comments in source codes? - Analysis of findings from Information Retrieval in Software Engineering Track @ FIRE 2022
Authors: 
Srijoni Majumdar, Ayan Bandyopadhyay, 
Partha Pratim Das, Paul Clough, Samiran Chattopadhyay, Prasenjit MajumderAuthors Info & Claims
Srijoni Majumdar, Ayan Bandyopadhyay, Partha Pratim Das, Paul Clough, Samiran Chattopadhyay, Prasenjit MajumderAuthors Info & ClaimsPages 15 - 17
Abstract
The Information Retrieval in Software Engineering (IRSE) track aims to develop solutions for automated evaluation of code comments in a machine learning framework. In this track, there is a binary classification task to classify comments as useful and not useful. The dataset consists of 9048 code comments and surrounding code snippet pairs extracted from open source github C based projects. Overall 34 experiments have been submitted by 11 teams from various universities and software companies. The submissions have been evaluated quantitatively using the F1-Score and qualitatively based on the type of features developed, the supervised learning model used and their corresponding hyper-parameters. The best performing architectures mostly have employed transformer architectures coupled with a software development related embedding space.
References
[1]
Amiangshu Bosu, Michaela Greiler, and Christian Bird. 2015. Characteristics of useful code reviews: An empirical study at microsoft(Working Conference on Mining Software Repositories). IEEE, 146–156.
[2]
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805(2018).
[3]
Mingwei Liu, Yanjun Yang, Xin Peng, Chong Wang, Chengyuan Zhao, Xin Wang, and Shuangshuang Xing. 2020. Learning based and Context Aware Non-Informative Comment Detection(International Conference on Software Maintenance and Evolution (ICSME)). IEEE, 866–867.
[4]
Srijoni Majumdar, Ayan Bandyopadhyay, Samiran Chattopadhyay, Partha Pratim Das, Paul D Clough, and Prasenjit Majumder. 2022. Overview of the IRSE track at FIRE 2022: Information Retrieval in Software Engineering. In FIRE (Working Notes).
[5]
Srijoni Majumdar, Ayush Bansal, Partha Pratim Das, Paul D Clough, Kausik Datta, and Soumya Kanti Ghosh. 2022. Automated evaluation of comments to aid software maintenance. Journal of Software: Evolution and Process 34, 7 (2022), e2463.
[6]
Mohammad Masudur Rahman, Chanchal K Roy, and Raula G Kula. 2017. Predicting usefulness of code review comments using textual features and developer experience(International Conference on Mining Software Repositories (MSR)). IEEE, 215–226.
[7]
Daniela Steidl, Benjamin Hummel, and Elmar Juergens. 2013. Quality analysis of source code comments(International Conference on Program Comprehension (ICPC)). IEEE, 83–92.
Index Terms
- Can we predict useful comments in source codes? - Analysis of findings from Information Retrieval in Software Engineering Track @ FIRE 2022
Recommendations
Efficiency of Large Language Models to scale up Ground Truth: Overview of the IRSE Track at Forum for Information Retrieval 2023
FIRE '23: Proceedings of the 15th Annual Meeting of the Forum for Information Retrieval EvaluationThe Software Engineering Information Retrieval (IRSE) track aims to devise solutions for the automated evaluation of code comments within a machine learning framework, with labels generated by both humans and large language models. Within this track, ...
Deriving thresholds of software metrics to predict faults on open source software
We empirically examined if there are effective thresholds for software metrics.Open-source software systems were used as benchmarking datasets.The learner model was created using logistic regression and the Bender method.Experimental results revealed ...
On the impact of multiple source code representations on software engineering tasks — An empirical study
AbstractEfficiently representing source code is crucial for various software engineering tasks such as code classification and clone detection. Existing approaches primarily use Abstract Syntax Tree (AST), and only a few focus on semantic graphs such as ...
Highlights- Emphasizes and demonstrates the idea of integrating multiple source code representations for software engineering tasks.
- Leverages the advantages of semantic code representations (CFG and PDG) along with syntactic code representations (...
Comments
Information & Contributors
Information
Published In
December 2022
101 pages
ISBN:9798400700231
DOI:10.1145/3574318
Copyright © 2022 Owner/Author.
Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for third-party components of this work must be honored. For all other uses, contact the Owner/Author.
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 12 January 2023
Check for updates
Author Tags
Qualifiers
- Abstract
- Research
- Refereed limited
Conference
FIRE '22
Acceptance Rates
Overall Acceptance Rate 19 of 64 submissions, 30%
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 72Total Downloads
- Downloads (Last 12 months)27
- Downloads (Last 6 weeks)3
Reflects downloads up to 05 Mar 2025
Other Metrics
Citations
Cited By
View allView Options
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign inFull Access
View options
View or Download as a PDF file.
PDFeReader
View online with eReader.
eReaderHTML Format
View this article in HTML Format.
HTML Format