research-article

Evaluating usage and quality of technical software documentation: an empirical study

Authors:

Golara Garousi,

Mahmoud Moussavi,

Brian SmithAuthors Info & Claims

EASE '13: Proceedings of the 17th International Conference on Evaluation and Assessment in Software Engineering

Pages 24 - 35

https://doi.org/10.1145/2460999.2461003

Published: 14 April 2013 Publication History

Abstract

Context: Software documentation is an integral part of any software development process. However, software practitioners are often concerned about the lack of usage and quality of documentation in practice. Unfortunately, in many projects, practitioners find that software documentation artifacts are outdated, incomplete and sometimes not beneficial. Objective: Motivated by the needs of NovAtel Inc. (NovAtel), a world-leading company of GPS software systems, we propose in this paper an approach to analyze the usage and quality of software documentation in development and maintenance phases. Method: The approach incorporates inputs from automated analysis (e.g., mining of project's data) and also experts' opinion extracted from survey-based questionnaire. The approach has been designed based on the "action-research" approach and in close collaboration between industry and academia. Results: To evaluate the feasibility and usefulness of the proposed approach, we have applied it in an industrial setting and results are presented in this paper. One of the results is that, in the context of our case-study, usage of documentation for an implementation purpose is higher than the usage for maintenance purposes. Conclusion: It is concluded that the usage of documentation differs for various purposes and it depends on the type of the information needs as well as the task to be completed (e.g. development and maintenance). In addition, we identify the most important and relevant quality attributes which are critical to improving documentation quality.

References

[1]

D. L. Parnas, "Precise Documentation: The Key To Better Software," in The Future of Software Engineering, S. Nanz, Ed.: Springer, 2011.

[2]

S. Cozzetti, B. d. Souza, N. Anquetil, and K. M. d. Oliveira, "Which documentation for software maintenance?," Journal of the Brazilian Computer Society, vol. 12, 2006.

[3]

L. C. Briand., "Software documentation: How much is enough?," in Proceedings of the European Conference on Software Maintenance and Reengineering, 2003, pp. 13--17.

Digital Library

[4]

P. S. M. d. Santos and G. H. Travassos, "Action research use in software engineering: An initial survey," in Proceedings of the International Symposium on Empirical Software Engineering and Measurement, 2009

Digital Library

[5]

A. Forward, "Software Documentation: Building and Maintaining Artefacts of Communication," University of Ottawa, MSc Thesis, 2002.

[6]

M. Visconti and C. R. Cook, "Assessing the State of Software Documentation Practices," in Conf. on Product Focused Software Process Improvement, 2004, pp. 485--496.

[7]

E. Arisholm, L. C. Briand, S. E. Hove, and Y. Labiche, "The Impact of UML Documentation on Software Maintenance: An Experimental Evaluation," IEEE Transactions on Software Engineering vol. 32, 2006

Digital Library

[8]

A. Von Mayrhauser, "Program comprehension during software maintenance and evolution," IEEE Computer, vol. 28, pp. 44--55, 1995.

Digital Library

[9]

S. C. B. d. Souza, N. Anquetil, and K. M. d. Oliveira, "A study of the documentation essential to software maintenance," in Proceedings of the annual international conference on Design of communication, 2005.

Digital Library

[10]

E. Tryggeseth, "Report from an Experiment: Impact of Documentation on Maintenance," Empirical Software Engineering, pp. 201--207, 1997.

Digital Library

[11]

M. Kajko-Mattsson, "A Survey of Documentation Practice within Corrective Maintenance," Empirical Software Engineering, vol. 10, pp. 31--55, 2005.

Digital Library

[12]

W. J. Dzidek, E. Arisholm, and L. C. Briand, "A Realistic Empirical Evaluation of the Costs and Benefits of UML in Software Maintenance," IEEE Transactions on Software Engineering, vol. 34, pp. 407--432 2008.

Digital Library

[13]

A. Wingkvist, M. Ericsson, R. Lincke, and W. Lowe, "A Metrics-Based Approach to Technical Documentation Quality," in Proceedings of the International Conference on the Quality of Information and Communications Technology, 2010, pp. 476--481

Digital Library

[14]

J. Zhi, V. Garousi, B. Sun, G. Garousi, S. Shahnewaz, and G. Ruhe, "Cost, Benefits and Quality of Technical Software Documentation: A Systematic Mapping " Journal of Systems and Software, Under Review, 2012.

[15]

J. Zhi, V. Garousi, B. Sun, G. Garousi, S. Shahnewaz, and G. Ruhe, "Online paper repository for: Cost, Benefits and Quality of Technical Software Documentation: A Systematic Mapping" http://www.softqual.ucalgary.ca/projects/SM/doc_cost_benefit_usage_quality, Last accessed: Nov. 2012.

[16]

B. G. Freimut, L. C. Briand, and F. Vollei, "Determining Inspection Cost-Effectiveness by Combining Project Data and Expert Opinion," IEEE Trans. Software Eng., vol. 31, pp. 1074--1092, 2005.

Digital Library

[17]

P. Runeson and M. Höst, "Guidelines for conducting and reporting case study research in software engineering," Empirical Software Engineering, vol. 14, pp. 131--164, 2009.

Digital Library

[18]

F. Shull, J. Singer, and D. I. K. Sjøberg, Guide to Advanced Empirical Software Engineering: Springer, 2007.

Digital Library

[19]

D. E. Avison, F. Lau, M. D. Myers, and P. A. Nielsen, "Action Research," Communications of the ACM, vol. 42, pp. 94--97, 1999.

Digital Library

[20]

D. Avison, R. Baskerville, and M. Myers, "Controlling action research projects," Information Technology & People, vol. 14, pp. 28--45, 2001.

[21]

V. Garousi, "Evidence-based Insights about Issue Management Processes: An Exploratory Study," in Proceedings of the International Conference on Software Process (ICSP), 2009, pp. 112--123.

Digital Library

[22]

V. Basili, G. Caldiera, H. D Rombach, and R. v. Solingen, The Goal Question Metric Approach (Encyclopedia of Software Engineering 1): John Wiley and Sons, 2001.

[23]

O. L. Marcus Ciolkowski, Sira Vegas, Stefan Biffl, "Practical Experiences in the Design and Conduct of Surveys in Empirical Software Engineering," in Empirical Methods and Studies in Software Engineering, 2003, pp. 104--128.

[24]

V. Garousi and T. Varma, "A Replicated Survey of Software Testing Practices in the Canadian Province of Alberta: What has Changed from 2004 to 2009?," Journal of Systems and Software, vol. 83, pp. 2251--2262, 2010.

Digital Library

[25]

F. Shull, J. Singer, and D. I. K. Sjoberg, Guide To Advanced Empirical Software Engineering: Springer, 2008.

Digital Library

[26]

L. Wasserman, All of nonparametric statistics: Springer, 2007.

Digital Library

[27]

E. McCrum-Gardner, "Which is the correct statistical test to use?," British Journal of Oral and Maxillofacial Surgery, pp. 38--41, 2007.

[28]

F. Wilcoxon, "Individual comparisons by ranking methods," Biometrics Bulletin, pp. 80--83, 1945.

[29]

L. C. Briand, C. M. Differding, and H. D. Rombach, "Practical guidelines for measurement-based process improvement," Software Process Improvement and Practice Journal, vol. 2, 1997.

[30]

M. Umarji and C. Seaman, "Predicting acceptance of Software Process Improvement," ACM SIGSOFT Software Engineering Notes, vol. 30, pp. 1--6, 2005

Digital Library

[31]

A. Rainer and T. Hall, "Key success factors for implementing software process improvement: a maturity-based analysis," Journal of Systems and Software, vol. 62, pp. 71--84, 2002.

[32]

D. J. Gilmore, R. L. Winder, and F. Detienne, User-Centred Requirements for Software Engineering Environments: Springer, 1994.

Digital Library

[33]

G. Garousi, "A Hybrid Methodology for Analyzing Software Documentation Quality and Usage," MSc Thesis, University of Calgary, Oct. 2012.

[34]

B. Sun, "A Methodology for Analyzing Cost and Cost-Drivers of Technical Software Documentation," MSc Thesis, University of Calgary, 2012.

Cited By

Bala SSahling KHaase JMendling J(2025)ChatGPT for Tailoring Software Documentation for Managers and DevelopersAgile Processes in Software Engineering and Extreme Programming – Workshops10.1007/978-3-031-72781-8_11(103-109)Online publication date: 11-Jan-2025
https://doi.org/10.1007/978-3-031-72781-8_11
Karas ZBansal AZhang YLi TMcMillan CHuang Y(2024)A Tale of Two Comprehensions? Analyzing Student Programmer Attention during Code SummarizationACM Transactions on Software Engineering and Methodology10.1145/366480833:7(1-37)Online publication date: 26-Aug-2024
https://dl.acm.org/doi/10.1145/3664808
Sovrano FLognoul MBacchelli ARoychoudhury PPaiva AAbreu RStorey MSharif BXia X(2024)An Empirical Study on Compliance with Ranking Transparency in the Software Documentation of EU Online PlatformsProceedings of the 46th International Conference on Software Engineering: Software Engineering in Society10.1145/3639475.3640112(46-56)Online publication date: 14-Apr-2024
https://dl.acm.org/doi/10.1145/3639475.3640112
Show More Cited By

Index Terms

Evaluating usage and quality of technical software documentation: an empirical study
1. Software and its engineering
  1. Software creation and management
    1. Software post-development issues
      1. Documentation

Recommendations

Beyond accuracy: assessing software documentation quality
ESEC/FSE 2020: Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering

Good software documentation encourages good software engineering, but the meaning of "good" documentation is vaguely defined in the software engineering literature. To clarify this ambiguity, we draw on work from the data and information quality ...
The relevance of software documentation, tools and technologies: a survey
DocEng '02: Proceedings of the 2002 ACM symposium on Document engineering

This paper highlights the results of a survey of software professionals. One of the goals of this survey was to uncover the perceived relevance (or lack thereof) of software documentation, and the tools and technologies used to maintain, verify and ...
Knowledge-based approaches in software documentation: A systematic literature review

Context: Software documents are core artifacts produced and consumed in documentation activity in the software lifecycle. Meanwhile, knowledge-based approaches have been extensively used in software development for decades, however, the software ...

Comments

Information & Contributors

Information

Published In

cover image ACM Other conferences

EASE '13: Proceedings of the 17th International Conference on Evaluation and Assessment in Software Engineering

April 2013

268 pages

ISBN:9781450318488

DOI:10.1145/2460999

General Chair:
Fabio Q. B. da Silva
Universidade Federal de Pernambuco, Brazil
,
Program Chairs:
Natalia Juristo
Universidad Politécnica de Madrid, Spain
,
Guilherme Horta Travassos
Universidade Federal do Rio de Janeiro, Brazil

Copyright © 2013 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Centro de Informatica - UFPE: Centro de Informatica - UFPE
SBC: Brazilian Computer Society
CNPq: Conselho Nacional de Desenvolvimento Cientifico e Tecn
CAPES: Brazilian Higher Education Funding Council

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 14 April 2013

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

Conference

EASE '13

Sponsor:

Centro de Informatica - UFPE
SBC
CNPq
CAPES

EASE '13: 17th International Conference on Evaluation and Assessment in Software Engineering

April 14 - 16, 2013

Porto de Galinhas, Brazil

Acceptance Rates

EASE '13 Paper Acceptance Rate 31 of 94 submissions, 33%;

Overall Acceptance Rate 71 of 232 submissions, 31%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

27
Total Citations
View Citations
1,060
Total Downloads

Downloads (Last 12 months)93
Downloads (Last 6 weeks)16

Reflects downloads up to 08 Mar 2025

Other Metrics

View Author Metrics

Citations

Cited By

Bala SSahling KHaase JMendling J(2025)ChatGPT for Tailoring Software Documentation for Managers and DevelopersAgile Processes in Software Engineering and Extreme Programming – Workshops10.1007/978-3-031-72781-8_11(103-109)Online publication date: 11-Jan-2025
https://doi.org/10.1007/978-3-031-72781-8_11
Karas ZBansal AZhang YLi TMcMillan CHuang Y(2024)A Tale of Two Comprehensions? Analyzing Student Programmer Attention during Code SummarizationACM Transactions on Software Engineering and Methodology10.1145/366480833:7(1-37)Online publication date: 26-Aug-2024
https://dl.acm.org/doi/10.1145/3664808
Sovrano FLognoul MBacchelli ARoychoudhury PPaiva AAbreu RStorey MSharif BXia X(2024)An Empirical Study on Compliance with Ranking Transparency in the Software Documentation of EU Online PlatformsProceedings of the 46th International Conference on Software Engineering: Software Engineering in Society10.1145/3639475.3640112(46-56)Online publication date: 14-Apr-2024
https://dl.acm.org/doi/10.1145/3639475.3640112
Naimi LBouziane EJakimi ASaadane RChehri A(2024)Automating Software Documentation: Employing LLMs for Precise Use Case DescriptionProcedia Computer Science10.1016/j.procs.2024.09.568246(1346-1354)Online publication date: 2024
https://doi.org/10.1016/j.procs.2024.09.568
Venigalla AChimalakonda S(2024)An exploratory study of software artifacts on GitHub from the lens of documentationInformation and Software Technology10.1016/j.infsof.2024.107425(107425)Online publication date: Feb-2024
https://doi.org/10.1016/j.infsof.2024.107425
Latypova V(2024)Intelligent System of Approving Person Information Support Based on Association Rule Mining Between Errors in Design DocumentationAdvances in Automation V10.1007/978-3-031-51127-1_30(311-320)Online publication date: 4-Jan-2024
https://doi.org/10.1007/978-3-031-51127-1_30
Tang HNadi S(2023)Evaluating Software Documentation Quality2023 IEEE/ACM 20th International Conference on Mining Software Repositories (MSR)10.1109/MSR59073.2023.00023(67-78)Online publication date: May-2023
https://doi.org/10.1109/MSR59073.2023.00023
Rani PBlasi AStulova NPanichella SGorla ANierstrasz O(2023)A decade of code comment quality assessmentJournal of Systems and Software10.1016/j.jss.2022.111515195:COnline publication date: 1-Jan-2023
https://dl.acm.org/doi/10.1016/j.jss.2022.111515
Shetty MBansal CUpadhyayula SRadhakrishna AGupta ARoychoudhury ACadar CKim M(2022)AutoTSG: learning and synthesis for incident troubleshootingProceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering10.1145/3540250.3558958(1477-1488)Online publication date: 7-Nov-2022
https://dl.acm.org/doi/10.1145/3540250.3558958
Wang AWang DDrozdal JMuller MPark SWeisz JLiu XWu LDugan C(2022)Documentation Matters: Human-Centered AI System to Assist Data Science Code Documentation in Computational NotebooksACM Transactions on Computer-Human Interaction10.1145/348946529:2(1-33)Online publication date: 16-Jan-2022
https://dl.acm.org/doi/10.1145/3489465
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten