research-article

An Empirical Analysis of Issue Templates Usage in Large-Scale Projects on GitHub

Authors:

Metehan Saçakçı,

Eray TüzünAuthors Info & Claims

ACM Transactions on Software Engineering and Methodology, Volume 33, Issue 5

Article No.: 117, Pages 1 - 28

https://doi.org/10.1145/3643673

Published: 03 June 2024 Publication History

Abstract

GitHub Issues is a widely used issue tracking tool in open-source software projects. Originally designed with broad flexibility, its lack of standardization led to incomplete issue reports, impeding software development and maintenance efficiency. To counteract this, GitHub introduced issue templates in 2016, which rapidly became popular. Our study assesses the current use and evolution of these templates in large-scale open-source projects and their impact on issue tracking metrics, including resolution time, number of reopens, and number of issue comments. Employing a comprehensive analysis of 350 templates from 100 projects, we also evaluated over 1.9 million issues for template conformity and impact. Additionally, we solicited insights from open-source software maintainers through a survey. Our findings highlight issue templates’ extensive usage in 99 of the 100 surveyed projects, with a growing preference for YAML-based templates, a more structured template variant. Projects with a template exhibited markedly reduced resolution time (381.02 days to 103.18 days) and reduced issue comment count (4.95 to 4.32) compared to those without. The use of YAML-based templates further significantly decreased resolution time, the number of reopenings, and the discussion extent. Thus, our research underscores issue templates’ positive impact on large-scale open-source projects, offering recommendations for improved effectiveness.

References

[1]

Ben Bleikamp. 2016. Issue and Pull Request templates. https://github.blog/2016-02-17-issue-and-pull-request-templates/

[2]

Oscar Chaparro, Carlos Bernal-Cárdenas, Jing Lu, Kevin Moran, Andrian Marcus, Massimiliano Di Penta, Denys Poshyvanyk, and Vincent Ng. 2019. Assessing the quality of the steps to reproduce in bug reports. In Proceedings of the 2019 27th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering (New York, NY, USA). ACM, 86–96. DOI:

Digital Library

[3]

Songqiang Chen, Xiaoyuan Xie, Bangguo Yin, Yuanxiang Ji, Lin Chen, and Baowen Xu. 2020. Stay professional and efficient: Automatically generate titles for your bug reports. In Proceedings of the 35th IEEE/ACM International Conference on Automated Software Engineering (New York, NY, USA). ACM, 385–397. DOI:

Digital Library

[4]

Bryan Clark. 2018. Multiple issue and pull request templates. https://github.blog/2018-01-25-multiple-issue-and-pull-request-templates/

[5]

Jailton Coelho, Marco Tulio Valente, Luciano Milen, and Luciana L. Silva. 2020. Is this GitHub project maintained? Measuring the level of maintenance activity of open-source projects. Information and Software Technology 122 (62020), 106274. DOI:

[6]

Ozren Dabic, Emad Aghajani, and Gabriele Bavota. 2021. Sampling projects in GitHub for MSR studies. In 2021 IEEE/ACM 18th International Conference on Mining Software Repositories (MSR). IEEE, 560–564. DOI:

[7]

Santiago Dueñas, Valerio Cosentino, Jesus M. Gonzalez-Barahona, Alvaro del Castillo San Felix, Daniel Izquierdo-Cortazar, Luis Cañas-Díaz, and Alberto Pérez García-Plaza. 2021. GrimoireLab: A toolset for software development analytics. PeerJ Computer Science 7 (72021), e601. DOI:

[8]

GitHub. 2024. About issue and pull request templates. https://docs.github.com/en/communities/using-templates-to-encourage-useful-issues-and-pull-requests/about-issue-and-pull-request-templates

[9]

Aigerim Issabayeva, Ariadi Nugroho, and Joost Visser. 2012. Issue handling performance in proprietary software projects. In 2012 9th IEEE Working Conference on Mining Software Repositories (MSR). IEEE, 209–212.

[10]

Maliheh Izadi, Kiana Akbari, and Abbas Heydarnoori. 2022. Predicting the objective and priority of issue reports in software repositories. Empirical Software Engineering 27, 2 (32022), 50. DOI:

Digital Library

[11]

Eirini Kalliamvakou, Georgios Gousios, Kelly Blincoe, Leif Singer, Daniel M. German, and Daniela Damian. 2016. An in-depth study of the promises and perils of mining GitHub. Empirical Software Engineering 21, 5 (102016), 2035–2071. DOI:

Digital Library

[12]

Rafael Kallis, Andrea Di Sorbo, Gerardo Canfora, and Sebastiano Panichella. 2019. Ticket tagger: Machine learning driven issue classification. In 2019 IEEE International Conference on Software Maintenance and Evolution (ICSME). IEEE, 406–409. DOI:

[13]

Rafael Kallis, Andrea Di Sorbo, Gerardo Canfora, and Sebastiano Panichella. 2021. Predicting issue types on GitHub. Science of Computer Programming 205 (52021), 102598. DOI:

[14]

Zhixing Li, Yue Yu, Tao Wang, Yan Lei, Ying Wang, and Huaimin Wang. 2022. To follow or not to follow: Understanding issue/pull-request templates on GitHub. IEEE Transactions on Software Engineering (2022), 1–16. DOI:

Digital Library

[15]

Bart Luijten, Joost Visser, and Andy Zaidman. 2010. Assessment of issue handling efficiency. In 2010 7th IEEE Working Conference on Mining Software Repositories (MSR 2010). IEEE, 94–97.

[16]

H. B. Mann and D. R. Whitney. 1947. On a test of whether one of two random variables is stochastically larger than the other. The Annals of Mathematical Statistics 18, 1 (31947), 50–60. DOI:

[17]

Rahul Mohanani, Paul Ralph, Burak Turhan, and Vladimir Mandic. 2021. How templated requirements specifications inhibit creativity in software engineering. IEEE Transactions on Software Engineering (2021), 1–1. DOI:

Digital Library

[18]

Maintainers of Open-Source Projects. 2016. Dear GitHub. https://github.com/dear-github/dear-github

[19]

Sebastiano Panichella, Gerardo Canfora, and Andrea Di Sorbo. 2021. Won’t we fix this issue? Qualitative characterization and automated identification of wontfix issues on GitHub. Information and Software Technology 139 (112021), 106665. DOI:

Digital Library

[20]

Henrique Rocha, Guilherme de Oliveira, Marco Tulio Valente, and Humberto Marques-Neto. 2016. Characterizing bug workflows in Mozilla Firefox. In Proceedings of the XXX Brazilian Symposium on Software Engineering. 43–52.

Digital Library

[21]

Tommaso Dal Sasso, Andrea Mocci, and Michele Lanza. 2016. What makes a satisficing bug report?. In 2016 IEEE International Conference on Software Quality, Reliability and Security (QRS). IEEE, 164–174. DOI:

[22]

Mozhan Soltani, Felienne Hermans, and Thomas Bäck. 2020. The significance of bug report elements. Empirical Software Engineering 25, 6 (112020), 5255–5294. DOI:

Digital Library

[23]

Yang Song and Oscar Chaparro. 2020. BEE: A tool for structuring and analyzing bug reports. In Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering (New York, NY, USA). ACM, 1551–1555. DOI:

Digital Library

[24]

Mengxi Zhang, Huaxiao Liu, Chunyang Chen, Yuzhou Liu, and Shuotong Bai. 2022. Consistent or not? An investigation of using Pull Request Template in GitHub. Information and Software Technology 144 (42022), 106797. DOI:

Digital Library

[25]

Thomas Zimmermann, Rahul Premraj, Nicolas Bettenburg, Sascha Just, Adrian Schroter, and Cathrin Weiss. 2010. What makes a good bug report? IEEE Transactions on Software Engineering 36, 5 (92010), 618–643. DOI:

Digital Library

Cited By

Butler AFilkov VRay BZhou M(2024)Software Supply Chain Risk: Characterization, Measurement & AttenuationProceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering10.1145/3691620.3695608(2506-2509)Online publication date: 27-Oct-2024
https://dl.acm.org/doi/10.1145/3691620.3695608

Index Terms

An Empirical Analysis of Issue Templates Usage in Large-Scale Projects on GitHub
1. Software and its engineering
  1. Software creation and management
    1. Collaboration in software development
      1. Open source model
    2. Software development process management

Recommendations

Understanding and Enhancing Issue Prioritization in GitHub
ASE '23: Proceedings of the 38th IEEE/ACM International Conference on Automated Software Engineering

GitHub has become a prominent platform for open source software development, facilitating collaboration and communication among a diverse group of contributors. Efficient issue tracking is a crucial aspect of managing projects on GitHub, and labels serve ...
Using dynamic and contextual features to predict issue lifetime in GitHub projects
MSR '16: Proceedings of the 13th International Conference on Mining Software Repositories

Methods for predicting issue lifetime can help software project managers to prioritize issues and allocate resources accordingly. Previous studies on issue lifetime prediction have focused on models built from static features, meaning features ...
“Won’t We Fix this Issue?” Qualitative characterization and automated identification of wontfix issues on GitHub
Abstract Context
: Addressing user requests in the form of bug reports and Github issues represents a crucial task of any successful software project. However, user-submitted issue reports tend to widely differ in their quality, and ...

Comments

Information & Contributors

Information

Published In

cover image ACM Transactions on Software Engineering and Methodology

ACM Transactions on Software Engineering and Methodology Volume 33, Issue 5

June 2024

952 pages

EISSN:1557-7392

DOI:10.1145/3618079

Editor:
Mauro Pezzè
USI Università della Svizzera italiana and SIT Schaffhausen Institute of Technology, Switzerland

Issue’s Table of Contents

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 03 June 2024

Online AM: 31 January 2024

Accepted: 15 January 2024

Revised: 12 January 2024

Received: 12 June 2023

Published in TOSEM Volume 33, Issue 5

Check for updates

Author Tags

Qualifiers

Research-article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

1
Total Citations
View Citations
471
Total Downloads

Downloads (Last 12 months)394
Downloads (Last 6 weeks)19

Reflects downloads up to 03 Mar 2025

Other Metrics

View Author Metrics

Citations

Cited By

Butler AFilkov VRay BZhou M(2024)Software Supply Chain Risk: Characterization, Measurement & AttenuationProceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering10.1145/3691620.3695608(2506-2509)Online publication date: 27-Oct-2024
https://dl.acm.org/doi/10.1145/3691620.3695608

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Full Text

View this article in Full Text.

Figures

Tables

Media

View full text|Download PDF

View Issue’s Table of Contents