Skip to main content
SpringerLink
Log in

How programmers find online learning resources

  • Published:
Empirical Software Engineering Aims and scope Submit manuscript

Abstract

When learning a new technology, programmers often have to sift through multiple online resources to find information that addresses their questions. Prior work has reported that information seekers use a number of different strategies, including following scents, or indicators, to locate appropriate resources. We present a qualitative and quantitative investigation of how programmers learning a new technology employ these strategies to navigate between online resources and evaluate the pertinence of these resources. We performed a diary and interview study with ten programmers learning a new technology, to study how users navigate from the question they have to the resource that satisfies this need. Based on our observations, we propose a resource-seeking model that represents the online resource seeking behaviour of programmers when learning a new technology. The model is comprised of six components that can be divided into two groups: Need-oriented components, i.e. Questions, Preferences, and Beliefs, and Resource-oriented components, i.e. Resources, Cues, and Impression Factors. We identified nine relations between these components and studied how the components are associated. We report on the characteristics of the components and the relationships between them, and discuss the importance of search customization and other implications of our observations for resource creators and search tools.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8

Similar content being viewed by others

Code Availability

Not applicable

References

  • Abdi H et al (2007) Bonferroni and šidák corrections for multiple comparisons. In: Encyclopedia of measurement and statistics, vol 3, pp 103–107

  • Arya DM, Guo JLC, Robillard MP (2020) Information correspondence between types of documentation for APIs. Empir Softw Eng 25(5):4069–4096

    Article  Google Scholar 

  • Arya DM, Nassif M, Robillard MP (2021) A data-centric study of software tutorial design. IEEE Softw

  • Bai GR, Kayani J, Stolee KT (2020) How graduate computing students search when using an unfamiliar programming language. In: Proceedings of the 28th international conference on program comprehension, ICPC, Association for computing machinery, pp 160–171

  • Brandt J, Guo PJ, Lewenstein J, Dontcheva M, Klemmer SR (2009) Two studies of opportunistic programming: interleaving web foraging, learning, and writing code. In: Proceedings of the SIGCHI conference on human factors in computing systems, CHI ’09. Association for Computing Machinery, pp 1589–1598

  • Carroll J (1990) An overview of minimalist instruction. In: Annual Hawaii international conference on system sciences, vol 4. IEEE Computer Society, pp 210–219

  • Chattopadhyay S, Nelson N, Au A, Morales N, Sanchez C, Pandita R, Sarma A (2020) A tale from the trenches: cognitive biases and software development. In: Proceedings of the 42nd ACM/IEEE international conference on software engineering, ICSE ’20. Association for Computing Machinery, New York, pp 654–665

  • Dondio P, Shaheen S (2019) Is Stack Overflow an effective complement to gaining practical knowledge compared to traditional computer science learning?. In: Proceedings of the international conference on education technology and computers (ICETC), pp 132–138

  • Duala-Ekoko E, Robillard MP (2012) Asking and answering questions about unfamiliar apis: an exploratory study. In: 2012 34th International conference on software engineering (ICSE). IEEE, pp 266–276

  • Earle RH, Rosso MA, Alexander KE (2015) User preferences of software documentation genres. In: Proceedings of the 33rd annual international conference on the design of communication, SIGDOC ’15. Association for Computing Machinery

  • Erdem A, Marsella S, Johnson W (1998) Task oriented software understanding. In: Proceedings of international conference on automated software engineering. IEEE Computer Society, p 230

  • Erdos K, Sneed HM (1998) Partial comprehension of complex programs (enough to perform maintenance). In: Proceedings. 6th International workshop on program comprehension. IWPC’98, pp 98–105

  • Escobar-Avila J, Venuti D, Di Penta M, Haiduc S (2019) A survey on online learning preferences for computer science and programming. In: Proceedings of international conference on software engineering: software engineering education and training (ICSE-SEET), pp 170–181

  • Gallardo-Valencia RE, Sim SE (2011) What kinds of development problems can be solved by searching the web?: a field study. In: Proceedings—international conference on software engineering, pp 41–44

  • Jääskeläinen R (2010) Think-aloud protocol. In: Handbook of translation studies, vol 1, pp 371–374

  • Ko AJ, DeLine R, Venolia G (2007) Information needs in collocated software development teams. In: 29th International conference on software engineering, pp 344–353

  • Landis JR, Koch G (1977) The measurement of observer agreement for categorical data. Biometrics 33:159–174

    Article  MATH  Google Scholar 

  • Lawrance J, Bellamy R, Burnett M, Rector K (2008) Using information scent to model the dynamic foraging behavior of programmers in maintenance tasks. In: Proceedings of the SIGCHI conference on human factors in computing systems, CHI ’08. Association for Computing Machinery, pp 1323–1332

  • Lazar J, Feng JH, Hochheiser H Lazar J, Feng J H, Hochheiser H (eds) (2017a) Chapter 11—analyzing qualitative data. Morgan Kaufmann, Boston

  • Lazar J, Feng JH, Hochheiser H (2017b) Chapter 6—diaries. In: Lazar J, Feng J H, Hochheiser H (eds) Research methods in human computer interaction, 2nd edn. Morgan Kaufmann, Boston, pp 135–152

  • Liu MX, Kittur A, Myers BA (2021) To reuse or not to reuse? A framework and system for evaluating summarized knowledge. In: Proceedings of the ACM on human-computer interaction (CSCW1)

  • Lu Y, Hsiao IH (2017) Personalized information seeking assistant (pisa): from programming information seeking to learning. Inf Retr 20:433–455

    Article  Google Scholar 

  • Mack RL, Lewis CH, Carroll JM (1983) Learning to use word processors: problems and prospects. ACM Trans Inf Syst 1(3):254–271

    Article  Google Scholar 

  • Marques A, Bradley NC, Murphy GC (2020) Characterizing task-relevant information in natural language software artifacts. In: IEEE international conference on software maintenance and evolution (ICSME), pp 476–487

  • Mehta CR, Patel NR (2011) IBM SPSS exact tests. IBM Corporation, Armonk

    Google Scholar 

  • Meng M, Steinhardt S, Schubert A (2019) How developers use API documentation: an observation study. Commun Des Q Rev 7:40–49

    Article  Google Scholar 

  • Nadi S, Treude C (2020) Essential sentences for navigating stack overflow answers. In: International conference on software analysis, evolution and reengineering (SANER). IEEE, pp 229–239

  • Piorkowski D, Fleming SD, Scaffidi C, Burnett M, Kwan I, Henley AZ, Macbeth J, Hill C, Horvath A (2015) To fix or to learn? How production bias affects developers’ information foraging during debugging. In: IEEE International conference on software maintenance and evolution (ICSME), pp 11–20

  • Pirolli P, Card S (1999) Information foraging. Psychol Rev 106:643–675

    Article  Google Scholar 

  • Pirolli P, Fu WT (2003) SNIF-ACT: a model of information foraging on the World Wide Web

  • Rao N, Bansal C, Zimmermann T, Awadallah AH, Nagappan N (2019) Analyzing web search behavior for software engineering tasks. arXiv:1912.09519

  • Robillard MP, Marcus A, Treude C, Bavota G, Chaparro O, Ernst N, Gerosa MA, Godfrey M, Lanza M, Linares-vásquez M, Murphy GC, Moreno L, Shepherd D, Wong E (2017) On-demand developer documentation. In: International conference on software maintenance and evolution (ICSME). IEEE, pp 479–483

  • Sadowski C, Stolee KT, Elbaum S (2015) How developers search for code: a case study. In: Proceedings of the 10th joint meeting on foundations of software engineering, ESEC/FSE. Association for Computing Machinery, pp 191–201

  • Sharpe D (2015) Chi-square test is statistically significant: now what? Pract Assess Res Eval 20(1):8

    Google Scholar 

  • Sillito J, Murphy GC, De Volder K (2006) Questions programmers ask during software evolution tasks. In: Proceedings of the SIGSOFT international symposium on foundations of software engineering (FSE). Association for Computing Machinery, pp 23–34

  • Sprent P (2011) Fisher exact test. Springer, Berlin, pp 524–525

    Google Scholar 

  • Srinivasa Ragavan S, Kuttal SK, Hill C, Sarma A, Piorkowski D, Burnett M (2016) Foraging among an overabundance of similar variants. In: Proceedings of the 2016 CHI conference on human factors in computing systems, CHI ’16. Association for Computing Machinery, New York, pp 3509–3521

  • Teevan J, Alvarado C, Ackerman MS, Karger DR (2004) The perfect search engine is not enough: a study of orienteering behavior in directed search. In: Proceedings of the SIGCHI conference on human factors in computing systems, pp 415–422

  • Treude C, Aniche M (2018) Where does google find API documentation?. In: Proceedings of international conference on software engineering. ACM, pp 23–26

  • Xia X, Bao L, Lo D, Kochhar PS, Hassan A, Xing Z (2017) What do developers search for on the web? Empir Softw Eng 22:3149–3185

    Article  Google Scholar 

  • Xie I, Joo S (2012) Factors affecting the selection of search tactics: tasks, knowledge, process, and systems. Inf Process Manag 48(2):254–270

    Article  Google Scholar 

Download references

Funding

This work has been funded by the Natural Sciences and Engineering Research Council (NSERC) Discovery Grant.

Author information

Authors and Affiliations

  1. School of Computer Science, McGill University, Montreal, Quebec, Canada

    Deeksha M. Arya, Jin L. C. Guo & Martin P. Robillard

Authors
  1. Deeksha M. Arya
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Jin L. C. Guo
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Martin P. Robillard
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

Jin L.C. Guo and Martin P. Robillard contributed equally to the research and manuscript.

Corresponding author

Correspondence to Deeksha M. Arya.

Ethics declarations

Ethics Approval

This study is approved by the Research Ethics Board Office at McGill University (file number: 20-07-039).

Conflict of Interests

The authors have no conflicts of interests to declare that are relevant to the content of this article.

Competing Interests

The authors have no competing interests to declare that are relevant to the content of this article.

Additional information

Communicated by: Venera Arnaoudova

Availability of Data and Material

This article is complemented by an appendix containing the documents required to replicate the study as well as our data set to reproduce our results. The appendix has been uploaded as supplementary material. Please refer to the supplementary file README.md for the list of appendix documents and a description of their content. The appendix will be made publicly available upon eventual publication of the article.

Informed Consent

All participants were provided with an informed consent form which needed to be duly signed prior to beginning the study. We proceeded with the study only if we received their signed informed consent. The informed consent form has been uploaded as a supplementary form as “Informed Consent Form.pdf” for reference.

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Arya, D.M., Guo, J.L.C. & Robillard, M.P. How programmers find online learning resources. Empir Software Eng 28, 23 (2023). https://doi.org/10.1007/s10664-022-10246-y

Download citation

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s10664-022-10246-y

Keywords

Search

Navigation