Abstract
Release notes (RNs) refer to the technical documentation that offers users, developers, and other stakeholders comprehensive information about the changes and updates of a new software version. Producing high-quality RNs can be challenging, and it remains unknown what issues developers commonly encounter and what effective strategies can be adopted to mitigate them. To bridge this knowledge gap, we conduct a manual analysis of 1,529 latest RN-related issues in the GitHub issue tracker by using multiple rounds of open coding and construct 1) a comprehensive taxonomy of RN-related issues with four dimensions validated through three semi-structured interviews; 2) an effective framework with eight categories of strategies to overcome these challenges. The four dimensions of RN-related issues revealed by the taxonomy and the corresponding strategies from the framework include: 1) Content (419, 25.47%): RN producers tend to overlook information rather than include inaccurate details, especially for breaking changes. To address this, effective completeness validations are recommended, such as managing Pull Requests, issues, and commits related to RNs; 2) Presentation (150, 9.12%): inadequate layout may bury important information and lead to end users’ confusion, which can be mitigated by employing a hierarchical structure, standardized format, rendering RNs, and folding techniques; 3) Accessibility (303, 18.42%): many users find RNs inaccessible due to link deterioration, insufficient notification, and obfuscated RN locations. This can be alleviated by adopting appropriate locations and channels (such as project websites) and standardizing link management.; 4) Production (773, 46.99%): despite the high demand from RN producers, automating and standardizing the RN production process remains challenging. Developers resolve this problem by using some mature tools on GitHub (like Release Drafter). Additionally, offering guidance, clarifying responsibilities, and distributing workloads are effective in improving collaboration within the team. Mechanisms for distributing and verifying RNs are also selected to enhance synchronization management. Our taxonomy provides a comprehensive blueprint to improve RN production in practice and also reveals interesting future research directions.
![](http://media.springernature.com/m312/springer-static/image/art%3A10.1007%2Fs10664-024-10486-0/MediaObjects/10664_2024_10486_Fig1_HTML.png)
![](http://media.springernature.com/m312/springer-static/image/art%3A10.1007%2Fs10664-024-10486-0/MediaObjects/10664_2024_10486_Fig2_HTML.png)
![](http://media.springernature.com/m312/springer-static/image/art%3A10.1007%2Fs10664-024-10486-0/MediaObjects/10664_2024_10486_Fig3_HTML.png)
![](http://media.springernature.com/m312/springer-static/image/art%3A10.1007%2Fs10664-024-10486-0/MediaObjects/10664_2024_10486_Fig4_HTML.png)
![](http://media.springernature.com/m312/springer-static/image/art%3A10.1007%2Fs10664-024-10486-0/MediaObjects/10664_2024_10486_Fig5_HTML.png)
![](http://media.springernature.com/m312/springer-static/image/art%3A10.1007%2Fs10664-024-10486-0/MediaObjects/10664_2024_10486_Fig6_HTML.png)
![](http://media.springernature.com/m312/springer-static/image/art%3A10.1007%2Fs10664-024-10486-0/MediaObjects/10664_2024_10486_Fig7_HTML.png)
![](http://media.springernature.com/m312/springer-static/image/art%3A10.1007%2Fs10664-024-10486-0/MediaObjects/10664_2024_10486_Fig8_HTML.png)
![](http://media.springernature.com/m312/springer-static/image/art%3A10.1007%2Fs10664-024-10486-0/MediaObjects/10664_2024_10486_Fig9_HTML.png)
Similar content being viewed by others
Data Availability
A replication package includes all supplementary materials, which can be found at https://figshare.com/s/b86957f784a8c872c042. This package includes:
– An Excel spreadsheet contains the complete issues and all of our analysis of the RN-related issues;
– The scripts we used to collect, analyze, and visualize the data;
– A markdown contains the practitioner-oriented checklist for producing a high-quality RN.
Notes
Throughout this paper, we use the term “user” to refer to anyone reading or referring to RNs for their tasks, including internal developers, downstream developers, and software end users.
In this paper, the term “RN-related issues” refers to the issues proposed by developers in the GitHub issue tracker system for software release notes.
An IssueEvent can refer to any activity associated with issues, e.g., creating an issue and commenting on an issue. https://docs.github.com/en/rest/issues/events
We exclude issues whose status is open and whose duration time is over one year in order to avoid bias introduced by forgetting to close and inactivity.
We do not assign more because inspecting, comprehending, and labeling issues takes significant time and energy, which they lack to label more.
Considering a large number of leaf nodes, we only plot left nodes whose issue number is greater than 30, which cover all four dimensions and account for 82.54% of issues in our dataset.
Most automation tools primarily focus on automating RN generation, while there are also several tools recommended to assist RN management. In the following strategies, we will introduce other tools according to their goals.
Cake.recipe is exclusively deployed across multiple projects but only within their organization and with 74 stars.
References
Abebe SL, Ali N, Hassan AE (2016) An empirical study of software release notes. Empirical Software Engineering 21(3):1107–1142. https://doi.org/10.1007/s10664-015-9377-5
Aghajani E, Nagy C, Vega-Márquez OL, Linares-Vásquez M, Moreno L, Bavota G, Lanza M (2019) Software documentation issues unveiled. In: 2019 IEEE/ACM 41st International Conference on Software Engineering (ICSE), IEEE, pp 1199–1210. https://doi.org/10.1109/ICSE.2019.00122
Aghajani E, Nagy C, Linares-Vásquez M, Moreno L, Bavota G, Lanza M, Shepherd DC (2020) Software documentation: the practitioners’ perspective. In: 2020 IEEE/ACM 42nd International Conference on Software Engineering (ICSE), IEEE, pp 590–601. https://doi.org/10.1145/3377811.3380405
Ahire JB (2021) Issue #156 of hypertrace/hypertrace. https://github.com/hypertrace/hypertrace/issues/156
Alali A, Kagdi H, Maletic JI (2008) What’s a typical commit? a characterization of open source software repositories. In: 2008 16th IEEE International conference on program comprehension, IEEE, pp 182–191. https://doi.org/10.1109/ICPC.2008.24
Beyer S, Macho C, Di Penta M, Pinzger M (2018) Automatically classifying posts into question categories on Stack Overflow. In: 2018 IEEE/ACM 26th International Conference on Program Comprehension (ICPC), IEEE, pp 211–21110. https://doi.org/10.1145/3196321.3196333
Bi T, Xia X, Lo D, Grundy J, Zimmermann T (2020) An empirical study of release note production and usage in practice. IEEE Trans Soft Eng. https://doi.org/10.1109/TSE.2020.3038881
Chen Z, Cao Y, Liu Y, Wang H, Xie T, Liu X (2020) A comprehensive study on challenges in deploying deep learning based software. In: Proceedings of the 28th ACM joint meeting on European software engineering conference and symposium on the foundations of software engineering, pp 750–762. https://doi.org/10.1145/3368089.3409759
Coelho R, Almeida L, Gousios G, Van Deursen A (2015) Unveiling exception handling bug hazards in Android based on github and google code issues. In: 2015 IEEE/ACM 12th Working conference on mining software repositories, IEEE, pp 134–145. https://doi.org/10.1109/MSR.2015.20
El-Kassas WS, Salama CR, Rafea AA, Mohamed HK (2021) Automatic text summarization: a comprehensive survey. Expert Syst Appl 165:113679. https://doi.org/10.1016/j.eswa.2020.113679
Ghadhab L, Jenhani I, Mkaouer MW, Ben Messaoud M (2021) Augmenting commit classification by using fine-grained source code changes and a pre-trained deep neural language model. Inf Softw Technol 135:106566. https://doi.org/10.1016/j.infsof.2021.106566, https://www.sciencedirect.com/science/article/pii/S0950584921000495
He H, He R, Gu H, Zhou M (2021) A large-scale empirical study on java library migrations: prevalence, trends, and rationales. In: Proceedings of the 29th ACM joint meeting on European software engineering conference and symposium on the foundations of software engineering, pp 478–490
Holloway FW (1985) Praxis release notes, versions 7. 4 and 7. 5 https://www.osti.gov/biblio/5141606
Hove SE, Anda B (2005) Experiences from conducting semi-structured interviews in empirical software engineering research. In: 11th IEEE International software metrics symposium (METRICS’05), IEEE, pp 10–pp
Humbatova N, Jahangirova G, Bavota G, Riccio V, Stocco A, Tonella P (2020) Taxonomy of real faults in deep learning systems. In: Proceedings of the ACM/IEEE 42nd international conference on software engineering, pp 1110–1121. https://doi.org/10.1145/3377811.3380395
Jiang H, Zhu J, Yang L, Liang G, Zuo C (2021a) Deeprelease: language-agnostic release notes generation from pull requests of open-source software. In: 2021 28th Asia-Pacific software engineering conference (APSEC), IEEE, pp 101–110. https://doi.org/10.1109/APSEC53868.2021.00018
Jiang J, Wu Q, Cao J, Xia X, Zhang L (2021) Recommending tags for pull requests in github. Inf Soft Technol 129:106394. https://doi.org/10.1016/j.infsof.2020.106394
Kamezawa H, Nishida N, Shimizu N, Miyazaki T, Nakayama H (2022) Rnsum: a large-scale dataset for automatic release note generation via commit logs summarization. In: Proceedings of the 60th annual meeting of the association for computational linguistics (vol 1: Long Papers), pp 8718–8735
Klepper S, Krusche S, Bruegge B (2016) Semi-automatic generation of audience-specific release notes. In: 2016 IEEE/ACM International workshop on Continuous Software Evolution and Delivery (CSED), IEEE, pp 19–22
Lam P, Dietrich J, Pearce DJ (2020) Putting the semantics into semantic versioning. In: Proceedings of the 2020 ACM SIGPLAN International Symposium on New Ideas, New Paradigms, and Reflections on Programming and Software, pp 157–179, https://doi.org/10.1145/3426428.3426922
Levin S, Yehudai A (2017) Boosting automatic commit classification into maintenance activities by utilizing source code changes. Association for Computing Machinery, New York, USA, PROMISE, pp 97–106. https://doi.org/10.1145/3127005.3127016
Li M, Gao R, Hu X, Chen Y (2017) Comparing infovis designs with different information architecture for communicating complex information. Commun Design Quarterly Rev 5(1):43–56
Maalej W, Happel HJ (2010) Can development work describe itself? In: 2010 7th IEEE Working conference on Mining Software Repositories (MSR 2010), IEEE, pp 191–200. https://doi.org/10.1109/MSR.2010.5463344
Michlmayr M (2007) Quality improvement in volunteer free and open source software projects: exploring the impact of release management. PhD thesis, University of Cambridge
Moreno L, Bavota G, Penta MD, Oliveto R, Marcus A, Canfora G (2017) Arena: an approach for the automated generation of release notes. IEEE Trans Software Eng 43(2):106–127. https://doi.org/10.1109/TSE.2016.2591536
Nath SS, Roy B (2021) Towards automatically generating release notes using extractive summarization technique. In: International conference on Software Engineering & Knowledge Engineering, SEKE, pp 241–248
Nath SS, Roy B (2022) Exploring relevant artifacts of release notes: the practitioners’ perspective. In: IEEE International conference on software analysis, evolution and reengineering, SANER 2022, Honolulu, HI, USA, March 15-18, 2022, IEEE, pp 1270–1277. https://doi.org/10.1109/SANER53432.2022.00152, https://doi.org/10.1109/SANER53432.2022.00152
Olsson HH, Bosch J (2014) From opinions to data-driven software r &d: a multi-case study on how to close the ’open loop’ problem. In: 2014 40th EUROMICRO Conference on software engineering and advanced applications, IEEE, pp 9–16. https://doi.org/10.1109/SEAA.2014.75
Rossi B, Russo B, Succi G (2009) Analysis of open source software development iterations by means of burst detection techniques. In: Boldyreff C, Crowston K, Lundell B, Wasserman AI (eds) Open source ecosystems: diverse communities interacting, 5th IFIP WG 2.13 International Conference on Open Source Systems, OSS 2009, Skövde, Sweden, June 3-6, 2009. Proceedings, Springer, IFIP Advances in Information and Communication Technology, vol 299, pp 83–93, https://doi.org/10.1007/978-3-642-02032-2_9, https://doi.org/10.1007/978-3-642-02032-2_9
Seaman CB (1999) Qualitative methods in empirical studies of software engineering. IEEE Trans Soft Eng 25(4):557–572
Shihab E, Ihara A, Kamei Y, Ibrahim WM, Ohira M, Adams B, Hassan AE, Ki Matsumoto (2013) Studying re-opened bugs in open source software. Empirical Soft Eng 18(5):1005–1042. https://doi.org/10.1007/s10664-012-9228-6
Swanson EB (1976) The dimensions of maintenance. In: Proceedings of the 2nd international conference on software engineering, IEEE Computer Society Press, Washington, DC, USA, ICSE ’76, pp 492–497
Tan L, Yuan D, Krishna G, Zhou Y (2007) /* icomment: bugs or bad comments? */. In: Proceedings of 21st ACM SIGOPS symposium on operating systems principles, pp 145–158. https://doi.org/10.1145/1294261.1294276
Tan X, Zhou M (2019) How to communicate when submitting patches: an empirical study of the Linux kernel. Proceedings of the ACM on Human-Computer Interaction 3(CSCW):1–26
Toshevska M, Gievska S (2021) A review of text style transfer using deep learning. IEEE Trans Artif Intell pp 1–1, https://doi.org/10.1109/TAI.2021.3115992
Wu J, He H, Xiao W, Gao K, Zhou M (2022) Demystifying software release note issues on github. In: 2022 IEEE/ACM 30th International Conference on Program Comprehension (ICPC), pp 602–613. https://doi.org/10.1145/3524610.3527919
Wu J, Xu W, Gao K, Li J, Zhou M (2023) Characterize software release notes of github projects: structure, writing style, and content. In: Zhang T, Xia X, Novielli N (eds) IEEE International conference on Software analysis, evolution and reengineering, SANER 2023, Taipa, Macao, March 21-24, 2023, IEEE, pp 473–484. https://doi.org/10.1109/SANER56733.2023.00051
Yang AZ, Hassan S, Zou Y, Hassan AE (2021) An empirical study on release notes patterns of popular apps in the google play store. Empirical Soft Eng pp 1–41
Yang AZH, Hassan S, Zou Y, Hassan AE (2022) An empirical study on release notes patterns of popular apps in the google play store. Empir Softw Eng 27(2):55. https://doi.org/10.1007/s10664-021-10086-2
Yu L (2009) Mining change logs and release notes to understand software maintenance and evolution. CLEI Electron J 12(2):1–10
Yu S, Xu L, Zhang Y, Wu J, Liao Z, Li Y (2018) Nbsl: A supervised classification model of pull request in github. In: 2018 IEEE International conference on communications (ICC), pp 1–6
Zhai J, Shi Y, Pan M, Zhou G, Liu Y, Fang C, Ma S, Tan L, Zhang X (2020) C2s: translating natural language comments to formal program specifications. In: Proceedings of the 28th ACM joint meeting on European software engineering conference and symposium on the foundations of software engineering, pp 25–37
Zhang T, Gao C, Ma L, Lyu M, Kim M (2019a) An empirical study of common challenges in developing deep learning applications. In: 2019 IEEE 30th International symposium on software reliability engineering (ISSRE), IEEE, pp 104–115. https://doi.org/10.1109/ISSRE.2019.00020
Zhang Y, Zhou M, Mockus A, Jin Z (2019) Companies’ participation in oss development-an empirical study of openstack. IEEE Trans Soft Eng 47(10):2242–2259
Acknowledgements
This work is supported by the National Natural Science Foundation of China Grant 61825201 and 62332001.
Author information
Authors and Affiliations
Corresponding author
Additional information
Communicated by: Venera Arnaoudova, Sonia Haiduc, Gabriele Bavota
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
This article belongs to the Topical Collection: International Conference on Program Coprehension (ICPC).
Rights and permissions
Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.
About this article
Cite this article
Wu, J., He, H., Gao, K. et al. A comprehensive analysis of challenges and strategies for software release notes on GitHub. Empir Software Eng 29, 104 (2024). https://doi.org/10.1007/s10664-024-10486-0
Accepted:
Published:
DOI: https://doi.org/10.1007/s10664-024-10486-0