Cited By
View all- Nassif MRobillard M(2025)Evaluating interactive documentation for programmersEmpirical Software Engineering10.1007/s10664-025-10618-030:3Online publication date: 26-Feb-2025
A Field Study of Developer Documentation Format
Article No.: 7, Pages 1 - 7
Abstract
Documentation facilitates the transfer of knowledge among programmers and helps them become familiar with new technologies. However, the effectiveness with which a reader can find information in a document depends on its presentation format. In a prior publication, we presented Casdoc, a novel dynamic format for code examples. In this work, we synthesized five documentation presentation guidelines from prior research on programmer information needs and search behaviors. We then used Casdoc as an instrument to evaluate the impact of these guidelines in a field study with 326 students who used 126 documents over several months. Participants overwhelmingly chose to use Casdoc instead of a static baseline format. We observed that interactive documents can contain more information without distracting its readers. We also found some limitations that authors should consider when applying the guidelines, such as the large impact of small differences in visual cues to help readers navigate a document.
Supplementary Material
References
[1]
Emad Aghajani, Csaba Nagy, Mario Linares-Vásquez, Laura Moreno, Gabriele Bavota, Michele Lanza, and David C. Shepherd. 2020. Software Documentation: The Practitioners’ Perspective. In Proceedings of the IEEE/ACM 42nd International Conference on Software Engineering. Association for Computing Machinery, New York, NY, USA, 590–601.
[2]
Emad Aghajani, Csaba Nagy, Olga Lucero Vega-Márquez, Mario Linares-Vásquez, Laura Moreno, Gabriele Bavota, and Michele Lanza. 2019. Software Documentation Issues Unveiled. In Proceedings of the IEEE/ACM 41st International Conference on Software Engineering. IEEE Computer Society, Los Alamitos, CA, USA, 1199–1210.
[3]
Deeksha Arya, Jin L. C. Guo, and Martin P. Robillard. 2020. Information Correspondence between Types of Documentation for APIs. Empirical Software Engineering 25, 5 (2020), 4069–4096.
[4]
Deeksha M. Arya, Mathieu Nassif, and Martin P. Robillard. 2020. A Data-Centric Study of Software Tutorial Design. IEEE Software 39, 3 (2020), 106–115.
[5]
Stefanie Beyer, Christian Macho, Massimiliano Di Penta, and Martin Pinzger. 2020. What kind of questions do developers ask on Stack Overflow? A comparison of automated approaches to classify posts into question categories. Empirical Software Engineering 25, 3 (2020), 2258–2301.
[6]
Abir Bouraffa and Walid Maalej. 2020. Two Decades of Empirical Research on Developers’ Information Needs: A Preliminary Analysis. In Proceedings of the IEEE/ACM 42nd International Conference on Software Engineering Workshops. Association for Computing Machinery, New York, NY, USA, 71–77.
[7]
Joel Brandt, Philip J. Guo, Joel Lewenstein, Mira Dontcheva, and Scott R. Klemmer. 2009. Two Studies of Opportunistic Programming: Interleaving Web Foraging, Learning, and Writing Code. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. Association for Computing Machinery, New York, NY, USA, 1589–1598.
[8]
Margaret Burnett, Anicia Peters, Charles Hill, and Noha Elarief. 2016. Finding Gender-Inclusiveness Software Issues with GenderMag: A Field Investigation. In Proceedings of the CHI Conference on Human Factors in Computing Systems. Association for Computing Machinery, New York, NY, USA, 2586–2598.
[9]
Raymond P. L. Buse and Westley Weimer. 2012. Synthesizing API Usage Examples. In Proceedings of the 34th International Conference on Software Engineering. IEEE Computer Society, Los Alamitos, CA, USA, 782–792.
[10]
Kaibo Cao, Chunyang Chen, Sebastian Baltes, Christoph Treude, and Xiang Chen. 2021. Automated Query Reformulation for Efficient Search Based on Query Logs From Stack Overflow. In Proceedings of the IEEE/ACM 43rd International Conference on Software Engineering. IEEE Computer Society, Los Alamitos, CA, USA, 1273–1285.
[11]
Pei-Yu Chi, Sally Ahn, Amanda Ren, Mira Dontcheva, Wilmot Li, and Björn Hartmann. 2012. MixT: Automatic Generation of Step-ty-Step Mixed Media Tutorials. In Proceedings of the 25th annual ACM symposium on User interface software and technology. Association for Computing Machinery, New York, NY, USA, 93–102.
[12]
Bill Curtis, Sylvia B. Sheppard, Elizabeth Kruesi-Bailey, John Bailey, and Deborah A. Boehm-Davis. 1989. Experimental Evaluation of Software Documentation Formats. Journal of Systems and Software 9, 2 (1989), 167–207.
[13]
Rodrigo Fernandes Gomes da Silva, Chanchal K. Roy, Mohammad Masudur Rahman, Kevin A. Schneider, Klérisson Paixão, Carlos Eduardo de Carvalho Dantas, and Marcelo de Almeida Maia. 2020. CROKAGE: effective solution recommendation for programming tasks by leveraging crowd knowledge. Empirical Software Engineering 25, 6 (2020), 4707–4758.
[14]
Sonia Haiduc, Jairo Aponte, Laura Moreno, and Andrian Marcus. 2010. On the Use of Automated Text Summarization Techniques for Summarizing Source Code. In Proceedings of the 17th Working Conference on Reverse Engineering. IEEE Computer Society, Los Alamitos, CA, USA, 35–44.
[15]
Andrew Head, Jason Jiang, James Smith, Marti A. Hearst, and Björn Hartmann. 2020. Composing Flexibly-Organized Step-by-Step Tutorials from Linked Source Code, Snippets, and Outputs. In Proceedings of the CHI Conference on Human Factors in Computing Systems. Association for Computing Machinery, New York, NY, USA, 1–12.
[16]
Andrew Head, Caitlin Sadowski, Emerson Murphy-Hill, and Andrea Knight. 2018. When Not to Comment: Questions and Tradeoffs with API Documentation for C++ Projects. In Proceedings of the ACM/IEEE 40th International Conference on Software Engineering. Association for Computing Machinery, New York, NY, USA, 643–653.
[17]
Fred Hohman, Matthew Conlen, Jeffrey Heer, and Duen Horng (Polo) Chau. 2020. Communicating with Interactive Articles. Distill 5, 9 (2020), e28. https://distill.pub/2020/communicating-with-interactive-articles
[18]
Amber Horvath, Michael Xieyang Liu, River Hendriksen, Connor Shannon, Emma Paterson, Kazi Jawad, Andrew Macvean, and Brad A Myers. 2022. Understanding How Programmers Can Use Annotations on Documentation. In Proceedings of the CHI Conference on Human Factors in Computing Systems. Association for Computing Machinery, New York, NY, USA, 69:1–16.
[19]
Qiao Huang, Xin Xia, Zhenchang Xing, David Lo, and Xinyu Wang. 2018. API Method Recommendation without Worrying about the Task-API Knowledge Gap. In Proceedings of the 33rd ACM/IEEE International Conference on Automated Software Engineering. Association for Computing Machinery, New York, NY, USA, 293–304.
[20]
Maurice G Kendall. 1938. A new measure of rank correlation. Biometrika 30, 1/2 (1938), 81–93.
[21]
Amy J. Ko, Brad A. Myers, Michael J. Coblenz, and Htet Htet Aung. 2006. An Exploratory Study of How Developers Seek, Relate, and Collect Relevant Information during Software Maintenance Tasks. IEEE Transactions on Software Engineering 32, 12 (2006), 971–987.
[22]
Amy J. Ko and Bob Uttl. 2003. Individual Differences in Program Comprehension Strategies in Unfamiliar Programming Systems. In Proceedings of the 11th IEEE International Workshop on Program Comprehension. IEEE Computer Society, Los Alamitos, CA, USA, 175–184.
[23]
Timothy C. Lethbridge, Janice Singer, and Andrew Forward. 2003. How Software Engineers Use Documentation: The State of the Practice. IEEE Software 20, 6 (2003), 35–39.
[24]
Jiakun Liu, Sebastian Baltes, Christoph Treude, David Lo, Yun Zhang, and Xin Xia. 2021. Characterizing Search Activities on Stack Overflow. In Proceedings of the 29th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering. Association for Computing Machinery, New York, NY, USA, 919–931.
[25]
Mingwei Liu, Xin Peng, Andrian Marcus, Shuangshuang Xing, Christoph Treude, and Chengyuan Zhao. 2021. API-Related Developer Information Needs in Stack Overflow. IEEE Transactions on Software Engineering 48, 11 (2021), 4485–4500.
[26]
Mingwei Liu, Xin Peng, Andrian Marcus, Zhenchang Xing, Wenkai Xie, Shuangshuang Xing, and Yang Liu. 2019. Generating Query-Specific Class API Summaries. In Proceedings of the 27th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering. Association for Computing Machinery, New York, NY, USA, 120–130.
[27]
Lori Lorigo, Bing Pan, Helene Hembrooke, Thorsten Joachims, Laura Granka, and Geri Gay. 2006. The influence of task and gender on search and evaluation behavior using Google. Information Processing & Management 42, 4 (2006), 1123–1131.
[28]
Walid Maalej and Martin P. Robillard. 2013. Patterns of Knowledge in API Reference Documentation. IEEE Transactions on Software Engineering 39, 9 (2013), 1264–1282.
[29]
Laura MacLeod, Andreas Bergen, and Margaret-Anne Storey. 2017. Documenting and sharing software knowledge using screencasts. Empirical Software Engineering 22, 3 (2017), 1478–1507.
[30]
Paul W. McBurney and Collin McMillan. 2014. Automatic Documentation Generation via Source Code Summarization of Method Context. In Proceedings of the 22nd International Conference on Program Comprehension. Association for Computing Machinery, New York, NY, USA, 279–290.
[31]
Parisa Moslehi, Juergen Rilling, and Bram Adams. 2022. A user survey on the adoption of crowd-based software engineering instructional screencasts by the new generation of software developers. Journal of Systems and Software 185 (2022), 111144.
[32]
Sarah Nadi and Christoph Treude. 2020. Essential Sentences for Navigating Stack Overflow Answers. In Proceedings of the IEEE 27th International Conference on Software Analysis, Evolution and Reengineering. IEEE Computer Society, Los Alamitos, CA, USA, 229–239.
[33]
Seyed Mehdi Nasehi, Jonathan Sillito, Frank Maurer, and Chris Burns. 2012. What Makes a Good Code Example? A Study of Programming Q&A in StackOverflow. In Proceedings of the 28th IEEE International Conference on Software Maintenance. IEEE Computer Society, Los Alamitos, CA, USA, 25–34.
[34]
Mathieu Nassif, Zara Horlacher, and Martin P. Robillard. 2022. Casdoc: Unobtrusive Explanations in Code Examples. In Proceedings of the 30th IEEE/ACM International Conference on Program Comprehension. Association for Computing Machinery, New York, NY, USA, 631–635.
[35]
Stephen Oney and Joel Brandt. 2012. Codelets: Linking Interactive Documentation and Example Code in the Editor. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. Association for Computing Machinery, New York, NY, USA, 2697–2706.
[36]
Peter Pirolli and Stuart Card. 1999. Information Foraging. Psychological Review 106, 4 (1999), 643–675.
[37]
Plotly, Inc. 2021. Plotly JavaScript Open Source Graphing Library. https://plotly.com/javascript/ Last accessed 2023-03-03.
[38]
Daniele Procida. 2017. Diátaxis documentation framework. https://diataxis.fr/ Accessed 2022-07-30.
[39]
Martin P. Robillard. 2009. What makes APIs hard to learn? Answers from Developers. IEEE Software 26, 6 (2009), 27–34.
[40]
Martin P. Robillard and Christoph Treude. 2020. Understanding Wikipedia as a Resource for Opportunistic Learning of Computing Concepts. In Proceedings of the 51st ACM Technical Symposium on Computer Science Education. Association for Computing Machinery, New York, NY, USA, 72–78.
[41]
Christoffer Rosen and Emad Shihab. 2016. What are mobile developers asking about? A large scale study using stack overflow. Empirical Software Engineering 21, 3 (2016), 1192–1223.
[42]
Jaime Teevan, Christine Alvarado, Mark S. Ackerman, and David R. Karger. 2004. The Perfect Search Engine Is Not Enough: A Study of Orienteering Behavior in Directed Search. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. Association for Computing Machinery, New York, NY, USA, 415–422.
[43]
Christoph Treude, Ohad Barzilay, and Margaret-Anne Storey. 2011. How Do Programmers Ask and Answer Questions on the Web? (NIER Track). In Proceedings fo the 33rd International Conference on Software Engineering. Association for Computing Machinery, New York, NY, USA, 804–807.
[44]
Christoph Treude and Martin P. Robillard. 2016. Augmenting API Documentation with Insights from Stack Overflow. In Proceedings of the 38th ACM/IEEE International Conference on Software Engineering. Association for Computing Machinery, New York, NY, USA, 392–403.
[45]
Hans van der Meij, Joyce Karreman, and Michaël Steehouder. 2009. Three Decades of Research and Professional Practice on Printed Software Tutorials for Novices. Technical Communication 56, 3 (2009), 265–292.
[46]
Hans van der Meij and Jan van der Meij. 2013. Eight Guidelines for the Design of Instructional Videos for Software Training. Technical Communication 60, 3 (2013), 205–228.
[47]
Hans van der Meij and Jan van der Meij. 2014. A comparison of paper-based and video tutorials for software learning. Computers & Education 78 (2014), 150–159.
[48]
Bret Victor. 2011. Explorable Explanations. http://worrydream.com/ExplorableExplanations/ Last accessed 2023-03-03.
[49]
Bret Victor. 2012. Learnable Programming. http://worrydream.com/LearnableProgramming/ Last accessed 2023-03-03.
[50]
Wenting Wang, Deeksha Arya, Nicole Novielli, Jinghui Cheng, and Jin L. C. Guo. 2020. ArguLens: Anatomy of Community Opinions On Usability Issues Using Argumentation Models. In Proceedings of the CHI Conference on Human Factors in Computing Systems. Association for Computing Machinery, New York, NY, USA, 1–14.
[51]
Wan-Ching Wu, Diane Kelly, and Avneesh Sud. 2014. Using Information Scent and Need for Cognition to Understand Online Search Behavior. In Proceedings of the 37th International ACM SIGIR conference on Research & development in information retrieval. Association for Computing Machinery, New York, NY, USA, 557–566.
Index Terms
- A Field Study of Developer Documentation Format
Recommendations
Non-Linear Software Documentation with Interactive Code Examples
Documentation enables sharing knowledge between the developers of a technology and its users. Creating quality documents, however, is challenging: Documents must satisfy the needs of a large audience without being overwhelming for individuals. We address ...
Dynamically assembled documentation
SIGDOC '99: Proceedings of the 17th annual international conference on Computer documentationAs online information becomes more comprehensive in its scope, the sheer wealth of information can be overwhelming. Information needs to be both browsable and searchable, and both needs are best met with a structured information approach (such as SGML ...
Casdoc: unobtrusive explanations in code examples
ICPC '22: Proceedings of the 30th IEEE/ACM International Conference on Program ComprehensionCode examples are of great value to programmers trying to learn an unfamiliar API. Effective code examples are often surrounded with plain text explanations of the relevant concepts, techniques, and API elements involved in the example. However, ...
Comments
Information & Contributors
Information
Published In
April 2023
3914 pages
ISBN:9781450394222
DOI:10.1145/3544549
Copyright © 2023 Owner/Author.
Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for third-party components of this work must be honored. For all other uses, contact the Owner/Author.
Sponsors
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 19 April 2023
Check for updates
Author Tags
Qualifiers
- Work in progress
- Research
- Refereed limited
Funding Sources
Conference
CHI '23
Sponsor:
Acceptance Rates
Overall Acceptance Rate 6,164 of 23,696 submissions, 26%
Upcoming Conference
CHI 2025
- Sponsor:
- sigchi
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- View Citations1Total Citations
- 296Total Downloads
- Downloads (Last 12 months)149
- Downloads (Last 6 weeks)21
Reflects downloads up to 02 Mar 2025
Other Metrics
Citations
Cited By
View all- Nassif MRobillard M(2025)Evaluating interactive documentation for programmersEmpirical Software Engineering10.1007/s10664-025-10618-030:3Online publication date: 26-Feb-2025
View Options
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign inFull Access
View options
View or Download as a PDF file.
PDFeReader
View online with eReader.
eReaderFull Text
View this article in Full Text.
Full TextHTML Format
View this article in HTML Format.
HTML Format