skip to main content
10.1145/3555228.3555250acmotherconferencesArticle/Chapter ViewAbstractPublication PagessbesConference Proceedingsconference-collections
research-article

Assessing the Impact of Code Samples Evolution on Developers’ Questions

Published: 05 October 2022 Publication History

Abstract

To accelerate the learning process of technical resources by developers, organizations such as Amazon, Google, and Microsoft have been provided and maintained the so-called code samples. Code samples are special software projects made available in public repositories with the main goal of teaching and clarifying developers’ doubts related to the use of frameworks, libraries, or other technical resources. Since code samples are software projects themselves, they change over time. If these changes do not preserve essential characteristics of code samples such as size and code simplicity throughout evolution, they may lose their usefulness and can be another source of questions. We investigated the evolution of code samples as well as whether and how it may raise questions from the developers about the code samples themselves. In this way, we analyzed the evolution of 186 code samples provided by Android, AWS, Azure, Spring Boot, and Spring Cloud, assessing metrics related to the size and complexity of over 15,184 commits. We also explored more than 1,500 developers’ questions in Stack Overflow to understand the relation between these questions and code samples’ evolution. We observed that organizations may not be preserving key characteristics of code samples during evolution. Our analysis also indicates that some untamed metrics are related to a rise of questions about code samples.

References

[1]
Hadeel Alsolai, Marc Roper, and Dua Nassar. 2018. Predicting software maintainability in object-oriented systems using ensemble techniques. In 2018 IEEE International Conference on Software Maintenance and Evolution (ICSME). 716–721.
[2]
Amazon. 2022. AWS Samples. https://amazon.com/aws/
[3]
Gabriele Bavota and Barbara Russo. 2016. A large-scale empirical study on self-admitted technical debt. In Working Conference on Mining Software Repositories (MSR). 315–326.
[4]
Jacob Benesty, Jingdong Chen, Yiteng Huang, and Israel Cohen. 2009. Noise reduction in speech processing. Springer Science & Business Media.
[5]
Willian Marotzki Braga, Gabriel Menezes, Awdren Fontao, Andre Hora, and Bruno Cafeo. 2020. Quero lhe usar! Uma Análise do Público Alvo de Code Samples. In Anais do VIII Workshop de Visualização, Evolução e Manutenção de Software. 33–40.
[6]
Hongyu Pei Breivold, Ivica Crnkovic, and Magnus Larsson. 2012. A systematic review of software architecture evolution research. Information and Software Technology(2012).
[7]
Raymond PL Buse and Westley R Weimer. 2008. A metric for software readability. In Proceedings of the 2008 international symposium on Software testing and analysis.
[8]
Shyam R Chidamber and Chris F Kemerer. 1994. A metrics suite for object oriented design. IEEE Transactions on software engineering 20 (1994), 476–493.
[9]
Awdren Fontão, Sergio Cleger-Tamayo, Igor Wiese, Rodrigo Pereira dos Santos, and Arilo Claudio Dias-Neto. 2020. On value creation in developer relations (DevRel) a practitioners’ perspective. In Proceedings of the 15th International Conference on Global Software Engineering. 33–42.
[10]
The Apache Software Foundation. 2022. Apache Maven Project. https://maven.apache.org/
[11]
Inc. GitHub. 2022. Example of extraction. https://github.com/azure-samples?language=java
[12]
Google. 2022. Android Samples. https://developer.android.com/samples
[13]
Google. 2022. Google Maps Samples. https://developers.google.com/maps/documentation/javascript/examples/
[14]
Khaled H Hamed. 2008. Trend detection in hydrologic data: the Mann–Kendall trend test under the scaling hypothesis. Journal of hydrology 349(2008), 350–363.
[15]
Andre Hora. 2021. Characterizing top ranked code examples in Google. Journal of Systems and Software 178 (2021), 110971.
[16]
Md. Hussain and Ishtiak Mahmud. 2019. pyMannKendall: a python package for non parametric Mann Kendall family of trend tests.Journal of Open Source Software 4 (2019), 1556.
[17]
Gradle Inc. 2022. Gradle Build Tool. https://gradle.org/
[18]
Stack Exchange Inc. 2022. Stack Exchange Data Explorer. https://archive.org/download/stackexchange/
[19]
Ivar Jacobson, Magnus Christerson, Patrik Jonsson, and Gunnar Övergaard. 1992. Object-Oriented Software Engineering: A Use Case Driven Approach. Addison-Wesley.
[20]
Kalpana Johari and Arvinder Kaur. 2011. Effect of software evolution on software metrics: an open source case study. ACM SIGSOFT Software Engineering Notes 36 (2011), 1–8.
[21]
Maurice Kendall and Jean D. Gibbons. 1990. Rank Correlation Methods(5 ed.). A Charles Griffin Title.
[22]
Dino Konstantopoulos, John Marien, Mike Pinkerton, and Eric Braude. 2009. Best principles in the design of shared software. In International Computer Software and Applications Conference. 287–292.
[23]
Manny M Lehman. 1996. Laws of software evolution revisited. In European Workshop on Software Process Technology.
[24]
Henry B Mann. 1945. Nonparametric tests against trend. Econometrica: Journal of the econometric society (1945), 245–259.
[25]
Robert C Martin. 2009. Clean code: a handbook of agile software craftsmanship. Pearson Education.
[26]
Sarah Meldrum, Sherlock A Licorish, and Bastin Tony Roy Savarimuthu. 2017. Crowdsourced knowledge on stack overflow: A systematic mapping study. In Proceedings of the 21st International Conference on Evaluation and Assessment in Software Engineering.
[27]
Gabriel Menezes, Bruno Cafeo, and Andre Hora. 2019. Framework code samples: How are they maintained and used by developers?. In 2019 ACM/IEEE International Symposium on Empirical Software Engineering and Measurement (ESEM).
[28]
Gabriel Menezes, Bruno Cafeo, and Andre Hora. 2022. How are framework code samples maintained and used by developers? The case of Android and Spring Boot. Journal of Systems and Software 185 (2022), 111146.
[29]
Microsoft. 2022. Azure Samples. https://azure.microsoft.com/documentation/samples/
[30]
Microsoft. 2022. Microsoft Samples. https://code.msdn.microsoft.com
[31]
Laura Moreno, Gabriele Bavota, Massimiliano Di Penta, Rocco Oliveto, and Andrian Marcus. 2015. How can I use this method?. In International Conference on Software Engineering. 880–890.
[32]
Simon Moser and Oscar Nierstrasz. 1996. The effect of object-oriented frameworks on developer productivity. Computer 29(1996).
[33]
S. M. Nasehi, J. Sillito, F. Maurer, and C. Burns. 2012. What makes a good code example?: A study of programming Q A in StackOverflow. In 2012 28th IEEE International Conference on Software Maintenance (ICSM). 25–34.
[34]
Hector M Olague, Letha H Etzkorn, and Glenn W Cox. 2006. An Entropy-Based Approach to Assessing Object-Oriented Software Maintainability and Degradation-A Method and Case Study. In Software Engineering Research and Practice. 442–452.
[35]
Antoine Picard. 2014. Managing code samples in documentation.
[36]
Daryl Posnett, Abram Hindle, and Premkumar Devanbu. 2011. A Simpler Model of Software Readability. In Proceedings of the 8th Working Conference on Mining Software Repositories.
[37]
Steven Raemaekers, Arie van Deursen, and Joost Visser. 2012. Measuring software library stability through historical version analysis. In International Conference on Software Maintenance. 378–387.
[38]
Martin P Robillard. 2009. What makes APIs hard to learn? Answers from developers. IEEE software 26(2009), 27–34.
[39]
Martin P Robillard and Robert DeLine. 2011. A field study of API learning obstacles. Empirical Software Engineering 16 (2011), 703–732.
[40]
Simone Scalabrino, Mario Linares-Vásquez, Rocco Oliveto, and Denys Poshyvanyk. 2018. A comprehensive model for code readability. Journal of Software: Evolution and Process 30 (2018), e1958.
[41]
Scitools. 2022. Understand Documentation. https://scitools.com/features/
[42]
Samuel Sanford Shapiro and Martin B Wilk. 1965. An analysis of variance test for normality (complete samples). Biometrika 52(1965), 591–611.
[43]
Inc. Sonatype. 2022. Stats - Central Repository for Maven, Gradle, Kotlin, Scala and more. https://search.maven.org/stats
[44]
C. Spearman. 1904. The Proof and Measurement of Association between Two Things. The American Journal of Psychology 15 (1904), 72–101.
[45]
Spring. 2022. Spring Cloud Samples. https://spring.io/projects/spring-cloud#samples
[46]
Spring. 2022. Spring Samples. https://spring.io/guides/
[47]
Jeffrey Stylos and Brad A Myers. 2006. Mica: A web-search tool for finding api components and examples. In Visual Languages and Human-Centric Computing (VL/HCC’06). 195–202.
[48]
Kazi Zakia Sultana, Vaibhav Anu, and Tai-Yin Chong. 2021. Using software metrics for predicting vulnerable classes and methods in Java projects: A machine learning approach. Journal of Software: Evolution and Process 33 (2021), e2303.
[49]
Yuan Tian, Ferdian Thung, Abhishek Sharma, and David Lo. 2017. APIBot: question answering bot for API documentation. In 2017 32nd IEEE/ACM International Conference on Automated Software Engineering (ASE). 153–158.
[50]
Alexander Trautsch, Steffen Herbold, and Jens Grabowski. 2020. Static source code metrics and static analysis warnings for fine-grained just-in-time defect prediction. In 2020 IEEE International Conference on Software Maintenance and Evolution (ICSME). 127–138.
[51]
Michael Trier. 2022. GitPython Documentation. https://gitpython.readthedocs.io/en/stable/index.html
[52]
Twitter. 2022. Twitter Samples. http://twitterdev.github.io
[53]
Gias Uddin and Martin P Robillard. 2015. How API documentation fails. IEEE Software 32(2015), 68–75.
[54]
Danielle Vincent. 2018. Code example guidelines. https://developer.mozilla.org/en-US/docs/MDN/Contribute/Guidelines/Code_guidelines.
[55]
Claes Wohlin, Per Runeson, Martin Hst, Magnus C. Ohlsson, Bjrn Regnell, and Anders Wessln. 2012. Experimentation in Software Engineering. Springer Publishing Company, Incorporated.
[56]
Xiao Yu, Kwabena Ebo Bennin, Jin Liu, Jacky Wai Keung, Xiaofei Yin, and Zhou Xu. 2019. An Empirical Study of Learning to Rank Techniques for Effort-Aware Defect Prediction. In 2019 IEEE 26th International Conference on Software Analysis, Evolution and Reengineering (SANER). 298–309.
[57]
Feng Zhang, Audris Mockus, Ying Zou, Foutse Khomh, and Ahmed E Hassan. 2013. How does context affect the distribution of software maintainability metrics?. In 2013 IEEE International Conference on Software Maintenance. 350–359.
[58]
Jingxuan Zhang, Jiang He, Zhilei Ren, Tao Zhang, and Zhiqiu Huang. 2019. Enriching API Documentation with Code Samples and Usage Scenarios from Crowd Knowledge. IEEE Transactions on Software Engineering(2019).
[59]
Jack Zhang, Shikhar Sagar, and Emad Shihab. 2013. The evolution of mobile apps: An exploratory study. In Proceedings of the 2013 International Workshop on Software Development Lifecycle for Mobile. 1–8.
[60]
Shufan Zhou, Beijun Shen, and Hao Zhong. 2019. Lancer: Your code tell me what you need. In 2019 34th IEEE/ACM International Conference on Automated Software Engineering (ASE).
[61]
Zixiao Zhu, Chenyan Hua, Yanzhen Zou, Bing Xie, and Junfeng Zhao. 2017. Automatically generating task-oriented api learning guide. In Proceedings of the 9th Asia-Pacific Symposium on Internetware. 1–10.

Index Terms

  1. Assessing the Impact of Code Samples Evolution on Developers’ Questions

    Recommendations

    Comments

    Information & Contributors

    Information

    Published In

    cover image ACM Other conferences
    SBES '22: Proceedings of the XXXVI Brazilian Symposium on Software Engineering
    October 2022
    457 pages
    ISBN:9781450397353
    DOI:10.1145/3555228
    Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

    Publisher

    Association for Computing Machinery

    New York, NY, United States

    Publication History

    Published: 05 October 2022

    Permissions

    Request permissions for this article.

    Check for updates

    Author Tags

    1. code samples
    2. mining software repositories
    3. software evolution
    4. stack overflow

    Qualifiers

    • Research-article
    • Research
    • Refereed limited

    Funding Sources

    Conference

    SBES 2022
    SBES 2022: XXXVI Brazilian Symposium on Software Engineering
    October 5 - 7, 2022
    Virtual Event, Brazil

    Acceptance Rates

    Overall Acceptance Rate 147 of 427 submissions, 34%

    Contributors

    Other Metrics

    Bibliometrics & Citations

    Bibliometrics

    Article Metrics

    • 0
      Total Citations
    • 58
      Total Downloads
    • Downloads (Last 12 months)17
    • Downloads (Last 6 weeks)0
    Reflects downloads up to 15 Feb 2025

    Other Metrics

    Citations

    View Options

    Login options

    View options

    PDF

    View or Download as a PDF file.

    PDF

    eReader

    View online with eReader.

    eReader

    HTML Format

    View this article in HTML Format.

    HTML Format

    Figures

    Tables

    Media

    Share

    Share

    Share this Publication link

    Share on social media