Skip to main content
SpringerLink
Log in

An Empirical Comparison Between Tutorials and Crowd Documentation of Application Programming Interface

  • Regular Paper
  • Published:
Journal of Computer Science and Technology Aims and scope Submit manuscript

Abstract

API (application programming interface) documentation is critical for developers to learn APIs. However, it is unclear whether API documentation indeed improves the API learnability for developers. In this paper, we focus on two types of API documentation, i.e., official API tutorials and API crowd documentation. First, we analyze API coverage and check API consistencies in API documentation based on the API traceability. Then, we conduct a survey and extract several characteristics to analyze which API documentation can help developers learn APIs. Our findings show that: 1) API crowd documentation can be regarded as a supplement to the official API tutorials to some extent; 2) the concerns for frequently-used APIs between different types of API documentation show a huge mismatch, which may prevent developers from deeply understanding the usages of APIs through only one type of API documentation; 3) official API tutorials can help developers seek API information on a long page and API crowd documentation could provide long codes for a particular programming task. These findings may help developers select the suitable API documentation and find the useful information they need.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. Subramanian S, Inozemtseva L, Holmes R. Live API documentation. In Proc. the 36th Int. Conf. Softw. Eng., May 2014, pp.643-652. DOI: https://doi.org/10.1145/2568225.2568313.

  2. Petrosyan G, Robillard M P, De Mori R. Discovering information explaining API types using text classification. In Proc. the 37th IEEE/ACM Int. Conf. Softw. Eng., May 2015, pp.869-879. DOI: https://doi.org/10.1109/ICSE.2015.97.

  3. Maalej W, Robillard M P. Patterns of knowledge in API reference documentation. IEEE Trans. Softw. Eng., 2013, 39(9): 1264-1282. DOI: https://doi.org/10.1109/TSE.2013.12.

    Article  Google Scholar 

  4. Robillard M P. What makes APIs hard to learn? Answers from developers. IEEE Softw., 2009, 26(6): 27-34. DOI: https://doi.org/10.1109/MS.2009.193.

    Article  Google Scholar 

  5. Thayer K. Using program analysis to improve API learnability. In Proc. the 2018 ACM Conf. Int. Computing Education Research, Aug. 2018, pp.292-293. DOI: https://doi.org/10.1145/3230977.3231009.

  6. Jiang J, Koskinen J, Ruokonen A, Systa T. Constructing usage scenarios for API redocumentation. In Proc. the 15th IEEE Int. Conf. Prog. Comprehension, Jun. 2007, pp.259-264. DOI: https://doi.org/10.1109/ICPC.2007.16.

  7. Jiang H, Zhang J, Li X, Ren Z, Lo D. A more accurate model for finding tutorial segments explaining APIs. In Proc. the 23rd IEEE Int. Conf. Softw. Analysis, Evolution, and Reengineering, Mar. 2016, pp.157-167. DOI: https://doi.org/10.1109/SANER.2016.59.

  8. Zhang J, Jiang H, Ren Z, Chen X. Recommending APIs for API related questions in Stack Overow. IEEE Access, 2018, 6: 6205-6219. DOI: https://doi.org/10.1109/ACCESS.2017.2777845.

    Article  Google Scholar 

  9. Treude C, Storey M A. Effective communication of software development knowledge through community portals. In Proc. the 19th ACM SIGSOFT Symposium and the 13th European Conf. Foundations of Softw. Eng., Sept. 2011, pp.91-101. DOI: https://doi.org/10.1145/2025113.2025129.

  10. Robillard M P, Deline R. A field study of API learning obstacles. Empir. Softw. Eng., 2011, 16(6): 703-732. DOI: https://doi.org/10.1007/s10664-010-9150-8.

    Article  Google Scholar 

  11. Scaffidi C. Why are APIs difficult to learn and use. ACM Crossroads Student Magazine, 2006, 12(4): 4-10. DOI: https://doi.org/10.1145/1144359.1144363.

    Article  Google Scholar 

  12. Jiang H, Zhang J, Ren Z, Zhang T. An unsupervised approach for discovering relevant tutorial fragments for APIs. In Proc. the 39th IEEE/ACM Int. Conf. Softw. Eng., May 2017, pp.38-48. DOI: https://doi.org/10.1109/ICSE.2017.12.

  13. Ye X, Shen H, Ma X, Bunescu R, Liu C. From word embeddings to document similarities for improved information retrieval in software engineering. In Proc. the 38th Int. Conf. Softw. Eng., May 2016, pp.404-415. DOI: https://doi.org/10.1145/2884781.2884862.

  14. Uddin G, Robillard M P. How API documentation fails. IEEE Softw., 2015, 32(4): 68-75. DOI: https://doi.org/10.1109/MS.2014.80.

    Article  Google Scholar 

  15. Zhou Y, Gu R, Chen T, Huang Z, Panichella S, Gall H. Analyzing APIs documentation and code to detect directive defects. In Proc. the 39th IEEE/ACM Int. Conf. Softw. Eng., May 2017, pp.27-37. DOI: https://doi.org/10.1109/ICSE.2017.11.

  16. Parnin C, Treude C, Grammel L, Storey M A. Crowd documentation: Exploring the coverage and the dynamics of API discussions on Stack Overow. Technical Report, Georgia Institute of Technology, 2012. http://chrisparnin.me/pdf/crowddoc.pdf, Jan. 2021.

  17. Wang X, Huang C, Yao L, Benatallah B, Dong M. A survey on expert recommendation in community question answering. Journal of Computer Science and Tech., 2018, 33(4): 625-653. DOI: https://doi.org/10.1007/s11390-018-1845-0.

    Article  Google Scholar 

  18. Mamykina L, Manoim B, Mittal M, Hripcsak G, Hartmann B. Design lessons from the fastest Q&A site in the west. In Proc. the 2011 Annual Conf. Human Factors in Computing Systems, May 2011, pp.2857-2866. DOI: https://doi.org/10.1145/1978942.1979366.

  19. Beyer S, Macho C, Pinzger M. On Android API classes and their references on Stack Overow. Technical Report, University of Klagenfurt, 2016. https://serg.aau.at/pub/StefanieBeyer/Publications/techreport AAU-SERG-2016-0-01.pdf, Jan. 2021.

  20. Zhang Y, Lo D, Xia X, Sun J L. Multi-factor duplicate question detection in Stack Overow. Journal of Computer Science and Tech., 2015, 30(5): 981-997. DOI: https://doi.org/10.1007/s11390-015-1576-4.

    Article  Google Scholar 

  21. Yang X L, Lo D, Xia X, Wan Z Y, Sun J L. What security questions do developers ask? A large-scale study of Stack Overow posts. Journal of Computer Science and Tech., 2016, 31(5): 910-924. DOI: https://doi.org/10.1007/s11390-016-1672-0.

    Article  Google Scholar 

  22. Rosen C, Shihab E. What are mobile developers asking about? A large scale study using Stack Overow. Empir. Softw. Eng., 2016, 21(3): 1192-1223. DOI: https://doi.org/10.1007/s10664-015-9379-3.

    Article  Google Scholar 

  23. Barua A, Thomas S W, Hassan A E. What are developers talking about? An analysis of topics and trends in Stack Overow. Empir. Softw. Eng., 2014, 19(3): 619-654. DOI: https://doi.org/10.1007/s10664-012-9231-y.

    Article  Google Scholar 

  24. Chen C, Wu K, Srinivasan V, Bharadwaj R K. The best answers? Think twice: Identifying commercial campaigns in the CQA forums. Journal of Computer Science and Tech., 2015, 30(4): 810-828. DOI: https://doi.org/10.1007/s11390-015-1562-x.

    Article  Google Scholar 

  25. Brito G, Hora A C, Valente M T, Romain R. On the use of replacement messages in API deprecation: An empirical study. Journal Syst. Softw., 2018, 137: 306-321. DOI: https://doi.org/10.1016/j.jss.2017.12.007.

    Article  Google Scholar 

  26. Dagenais B, Robillard M P. Recovering traceability links between an API and its learning resources. In Proc. the 34th Int. Conf. Softw. Eng., June 2012, pp.47-57. DOI: https://doi.org/10.1109/ICSE.2012.6227207.

  27. Rastkar S, Murphy G C, Murray G. Summarizing software artifacts: A case study of bug reports. In Proc. the 32nd ACM/IEEE Int. Conf. Softw. Eng., May 2010, pp.505-514. DOI: https://doi.org/10.1145/1806799.1806872.

  28. Bettenburg N, Just S, Schröter A, Weiss C, Premraj R, Zimmermann T. What makes a good bug report? In Proc. the 16th ACM SIGSOFT Int. Symposium on Foundations of Softw. Eng., November 2008, pp.308-318. DOI: https://doi.org/10.1145/1453101.1453146.

  29. Schneider T D. Information theory primer with an appendix on logarithms. http://users.fred.net/tds/lab/papers/primer/, Jan. 2021.

  30. Smith E A, Senter R J. Automated readability index. Wright-Patterson Air Force Base, 1967. https://apps.dtic.mil/dtic/tr/fulltext/u2/667273.pdf, Jan. 2021.

  31. Jay G T, Hale J E, Smith R K, Hale D P, Kraft N A, Ward C. Cyclomatic complexity and lines of code: Empirical evidence of a stable linear relationship. Journal of Softw. Eng. and Applications, 2009, 2(3): 137-143. DOI: https://doi.org/10.4236/jsea.2009.23020.

    Article  Google Scholar 

  32. Nykaza J, Messinger R, Boehme F, Norman CL, Mace M, Gordon M. What programmers really want: Results of a needs assessment for SDK documentation. In Proc. the 20th Annual ACM SIGDOC Int. Conf. Computer Documentation, Oct. 2002, pp.133-141. DOI: https://doi.org/10.1145/584955.584976.

  33. Mclellan S G, Roesler A W, Tempest J T, Spinuzzi C I. Building more usable APIs. IEEE Softw., 1998, 15(3): 78-86. DOI: https://doi.org/10.1109/52.676963.

    Article  Google Scholar 

  34. Santos A L, Myers B A. Design annotations to improve API discoverability. Journal of Systems and Softw., 2017, 126: 17-33. DOI: https://doi.org/10.1016/j.jss.2016.12.036.

    Article  Google Scholar 

  35. McCabe T J. A complexity measure. IEEE Trans. Softw. Eng., 1976, SE-2(4): 308-320. DOI: https://doi.org/10.1109/TSE.197-6.233837.

    Article  MathSciNet  MATH  Google Scholar 

  36. Yuan T, Thung F, Sharma A, Lo D. APIBot: Question answering bot for API documentation. In Proc. the 32nd IEEE/ACM Int. Conf. Automated Softw. Eng., Oct. 30{Nov. 3, 2017, pp.153-158. DOI: https://doi.org/10.1109/ASE.2017.8115628.

  37. Zar J H. Spearman rank correlation. Encyclopedia of Bio-statistics. DOI: https://doi.org/10.1002/0470011815.b2a15150.

  38. Mandelin D, Xu L, Bodík R, Kimelman D. Jungloid mining: Helping to navigate the API jungle. In Proc. the ACM SIG-PLAN Conf. Prog. Language Design and Implementation, June 2005, 40(6): 48-61. DOI: https://doi.org/10.1145/1065010.1065018.

  39. Mann H B, Whitney D R. On a test of whether one of two random variables is stochastically larger than the other. The Annals of Mathematical Statistics, 1947, 18(1): 50-60. DOI: https://doi.org/10.1214/AOMS/1177730491.

    Article  MathSciNet  MATH  Google Scholar 

  40. Kitchenham B A, Peeger S L. Personal opinion surveys. In Guide to Advanced Empirical Software Engineering, Shull F, Singer J, Sjøberg D (eds.), Springer, 2008, pp.63-92. DOI: https://doi.org/10.1007/978-1-84800-044-5_3.

  41. Zou W, Lo D, Chen Z, Xia X, Feng Y, Xu B. How practitioners perceive automated bug report management techniques. IEEE Trans. Softw. Eng., 2020, 46(8): 836-862. DOI: https://doi.org/10.1109/TSE.2018.2870414.

    Article  Google Scholar 

  42. David L O, Nagappan N, Zimmermann T. How practitioners perceive the relevance of software engineering research. In Proc. the 10th Joint Meeting on Foundations of Soft. Eng., Aug. 30{Sept. 4, 2015, pp.415-425. DOI: https://doi.org/10.1145/2786805.2786809.

  43. Fisher R A. On the interpretation of χ2 from contingency tables, and the calculation of P. Journal of the Royal Statistical Society, 1922, 85(1): 87-94. DOI: https://doi.org/10.1111/j.2397-2335.1922.tb00768.x.

    Article  Google Scholar 

  44. McDonald J H. Handbook of Biological Statistics (3rd edition). Sparky House Publisher, 2009.

  45. Rocha A M, Maia M A. Automated API documentation with tutorials generated from Stack Overow. In Proc. the 30th Brazilian Symposium on Softw. Eng., Sept. 2016, pp.33-42. DOI: https://doi.org/10.1145/2973839.2973847.

  46. Kim J, Lee S, Hwang S, Kim S. Enriching documents with examples: A corpus mining approach. ACM Trans. Information Systems, 2013, 31(1): Article No. 1. DOI: https://doi.org/10.1145/2414782.2414783.

  47. Treude C, Robillard M P. Augmenting API documentation with insights from Stack Overow. In Proc. the 38th IEEE/ACM Int. Conf. Softw. Eng., May 2016, pp.392-403. DOI: https://doi.org/10.1145/2884781.2884800.

Download references

Author information

Authors and Affiliations

  1. School of Software, Dalian University of Technology, Dalian, 116000, China

    Yi-Xuan Tang, Zhi-Lei Ren, He Jiang, Xiao-Chen Li & Wei-Qiang Kong

  2. Key Laboratory for Ubiquitous Network and Service Software of Liaoning Province, Dalian, 116000, China

    He Jiang

  3. School of Computer Science & Technology, Beijing Institute of Technology, Beijing, 100000, China

    He Jiang

Authors
  1. Yi-Xuan Tang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Zhi-Lei Ren
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. He Jiang
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Xiao-Chen Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Wei-Qiang Kong
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Zhi-Lei Ren.

Supplementary Information

ESM 1

(PDF 399 kb)

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Tang, YX., Ren, ZL., Jiang, H. et al. An Empirical Comparison Between Tutorials and Crowd Documentation of Application Programming Interface. J. Comput. Sci. Technol. 36, 856–876 (2021). https://doi.org/10.1007/s11390-020-0042-0

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11390-020-0042-0

Keywords

Search

Navigation