skip to main content
10.1145/3387940.3391485acmconferencesArticle/Chapter ViewAbstractPublication PagesicseConference Proceedingsconference-collections
research-article

Two Decades of Empirical Research on Developers' Information Needs: A Preliminary Analysis

Published: 25 September 2020 Publication History

Abstract

Over the last two decades, developers' daily intake of information has been constantly on the rise and so has the interest of research in investigating the information needs of developers. Knowledge about what information they seek and which sources they rely on is scarce and has to be updated regularly to match the rapid changes in development practices. In this paper, we reflect on the scientific studies published in this field over the last two decades. We present preliminary results of our analysis of a study sample where we particularly focus on the research methods used, the number of recruited participants, and the organisational context in which they emerged. We have investigated a total of 54 studies from 41 publications and found that convenience sampling is the predominant sampling strategy with a prevalence of the industrial organisational context. Moreover, the majority of studies had a reduced sample size and draw participants from a single organisation resulting in high sample homogeneity. Among the studies carried out in industry 51.9% recruited participants from Microsoft.

References

[1]
Marwan Abi-Antoun, Nariman Ammar, and Thomas LaToza. 2010. Questions about Object Structure during Coding Activities. In Proceedings of the 2010 ICSE Workshop on Cooperative and Human Aspects of Software Engineering (CHASE '10). Association for Computing Machinery, New York, NY, USA, 64--71. https://doi.org/10.1145/1833310.1833321
[2]
Bilal Amir and Paul Ralph. 2018. There is No Random Sampling in Software Engineering Research. In Proceedings of the 40th International Conference on Software Engineering: Companion Proceeedings (ICSE '18). Association for Computing Machinery, New York, NY, USA, 344--345. https://doi.org/10.1145/3183440.3195001
[3]
Andrew Begel and Thomas Zimmermann. 2014. Analyze This! 145 Questions for Data Scientists in Software Engineering. In Proceedings of the 36th International Conference on Software Engineering (ICSE 2014). Association for Computing Machinery, New York, NY, USA, 12--23. https://doi.org/10.1145/2568225.2568233
[4]
Silvia Breu, Rahul Premraj, Jonathan Sillito, and Thomas Zimmermann. 2010. Information Needs in Bug Reports: Improving Cooperation between Developers and Users. In Proceedings of the 2010 ACM Conference on Computer Supported Cooperative Work (CSCW '10). Association for Computing Machinery, New York, NY, USA, 301--310. https://doi.org/10.1145/1718918.1718973
[5]
Jim Buckley, Chris Exton, and Judith Good. 2004. Characterizing Programmers' Information-Seeking during Software Evolution. In Proceedings of the 12 International Workshop on Software Technology and Engineering Practice (STEP '04). IEEE Computer Society, USA, 23--29. https://doi.org/10.1109/STEP.2004.7
[6]
R. P. L. Buse and T. Zimmermann. 2012. Information needs for software development analytics. In 2012 34th International Conference on Software Engineering (ICSE). 987--996. https://doi.org/10.1109/ICSE.2012.6227122
[7]
R. M. d. Mello and G. H. Travassos. 2013. Would Sociable Software Engineers Observe Better?. In 2013 ACM / IEEE International Symposium on Empirical Software Engineering and Measurement. 279--282. https://doi.org/10.1109/ESEM.2013.33
[8]
E. Duala-Ekoko and M. P. Robillard. 2012. Asking and answering questions about unfamiliar APIs: An exploratory study. In 2012 34th International Conference on Software Engineering (ICSE). 266--276. https://doi.org/10.1109/ICSE.2012.6227187
[9]
Tore Dybå, Dag I.K. Sjøberg, and Daniela S. Cruzes. 2012. What Works for Whom, Where, When, and Why? On the Role of Context in Empirical Software Engineering. In Proceedings of the ACM-IEEE International Symposium on Empirical Software Engineering and Measurement (ESEM '12). Association for Computing Machinery, New York, NY, USA, 19--28. https://doi.org/10.1145/2372251.2372256
[10]
Robert Feldt, Thomas Zimmermann, Gunnar R. Bergersen, Davide Falessi, Andreas Jedlitschka, Natalia Juristo, Jürgen Münch, Markku Oivo, Per Runeson, Martin Shepperd, and et al. 2018. Four Commentaries on the Use of Students and Professionals in Empirical Software Engineering Experiments. Empirical Softw. Engg. 23, 6 (Dec. 2018), 3801--3820. https://doi.org/10.1007/s10664-018-9655-0
[11]
T. Fritz and G. C. Murphy. 2010. Using information fragments to answer the questions developers ask. In 2010 ACM/IEEE 32nd International Conference on Software Engineering, Vol. 1. 175--184. https://doi.org/10.1145/1806799.1806828
[12]
T. Gorschek, P. Garre, S. Larsson, and C. Wohlin. 2006. A Model for Technology Transfer in Practice. IEEE Software 23, 6 (Nov 2006), 88--95. https://doi.org/10.1109/MS.2006.147
[13]
Daniel Graziotin, Fabian Fagerholm, Xiaofeng Wang, and Pekka Abrahamsson. 2017. On the Unhappiness of Software Developers. Proceedings of the 21st International Conference on Evaluation and Assessment in Software Engineering - EASE'17 (2017). https://doi.org/10.1145/3084226.3084242
[14]
Nicole Haenni, Mircea Lungu, Niko Schwarz, and Oscar Nierstrasz. 2013. Categorizing Developer Information Needs in Software Ecosystems. In Proceedings of the 2013 International Workshop on Ecosystem Architectures (WEA 2013). Association for Computing Machinery, New York, NY, USA, 1--5. https://doi.org/10.1145/2501585.2501586
[15]
Nicole Haenni, Mircea Lungu, Niko Schwarz, and Oscar Nierstrasz. 2014. A Quantitative Analysis of Developer Information Needs in Software Ecosystems. In Proceedings of the 2014 European Conference on Software Architecture Workshops (ECSAW '14). Association for Computing Machinery, New York, NY, USA, Article Article 12, 6 pages. https://doi.org/10.1145/2642803.2642815
[16]
Andreas Höfer and Walter F. Tichy. 2007. Status of Empirical Research in Software Engineering. Springer Berlin Heidelberg, Berlin, Heidelberg, 10--19. https://doi.org/10.1007/978-3-540-71301-2_3
[17]
Martin Ivarsson and Tony Gorschek. 2011. A method for evaluating rigor and industrial relevance of technology evaluations. Empirical Software Engineering 16 (06 2011), 365--395. https://doi.org/10.1007/s10664-010-9146-4
[18]
Xiaoyu Jin, Charu Khatwani, Nan Niu, Michael Wagner, and Juha Savolainen. 2016. Pragmatic Software Reuse in Bioinformatics: How Can Social Network Information Help?. In Software Reuse: Bridging with Social-Awareness, Georgia M. Kapitsaki and Eduardo Santana de Almeida (Eds.). Springer International Publishing, Cham, 247--264.
[19]
J. Josyula, S. Panamgipalli, M. Usman, R. Britto, and N. Bin Ali. 2018. Software Practitioners' Information Needs and Sources: A Survey Study. In 2018 9th International Workshop on Empirical Software Engineering in Practice (IWESEP). 1--6. https://doi.org/10.1109/IWESEP.2018.00009
[20]
Miryung Kim. 2011. An Exploratory Study of Awareness Interests about Software Modifications. In Proceedings of the 4th International Workshop on Cooperative and Human Aspects of Software Engineering (CHASE '11). Association for Computing Machinery, New York, NY, USA, 80--83. https://doi.org/10.1145/1984642.1984662
[21]
Andrew J. Ko, Robert DeLine, and Gina Venolia. 2007. Information Needs in Collocated Software Development Teams. In Proceedings of the 29th International Conference on Software Engineering (ICSE '07). IEEE Computer Society, USA, 344--353. https://doi.org/10.1109/ICSE.2007.45
[22]
Andrew J. Ko, Brad A. Myers, Michael J. Coblenz, and Htet Htet Aung. 2006. An Exploratory Study of How Developers Seek, Relate, and Collect Relevant Information during Software Maintenance Tasks. IEEE Trans. Softw. Eng. 32, 12 (Dec. 2006), 971--987. https://doi.org/10.1109/TSE.2006.116
[23]
Juraj Kubelka, Alexandre Bergel, and Romain Robbes. 2014. Asking and Answering Questions during a Programming Change Task in Pharo Language. In Proceedings of the 5th Workshop on Evaluation and Usability of Programming Languages and Tools (PLATEAU '14). Association for Computing Machinery, New York, NY, USA, 1--11. https://doi.org/10.1145/2688204.2688212
[24]
Juraj Kubelka, Romain Robbes, and Alexandre Bergel. 2019. Live Programming and Software Evolution: Questions during a Programming Change Task. In Proceedings of the 27th International Conference on Program Comprehension (ICPC '19). IEEE Press, 30--41. https://doi.org/10.1109/ICPC.2019.00017
[25]
Thomas D. LaToza. 2008. Answering Common Questions about Code. In Companion of the 30th International Conference on Software Engineering (ICSE Companion '08). Association for Computing Machinery, New York, NY, USA, 983--986. https://doi.org/10.1145/1370175.1370218
[26]
Thomas D. LaToza and Brad A. Myers. 2010. Developers Ask Reachability Questions. In Proceedings of the 32nd ACM/IEEE International Conference on Software Engineering - Volume 1 (ICSE '10). Association for Computing Machinery, New York, NY, USA, 185--194. https://doi.org/10.1145/1806799.1806829
[27]
Thomas D. LaToza and Brad A. Myers. 2010. Hard-to-Answer Questions about Code. In Evaluation and Usability of Programming Languages and Tools (PLATEAU '10). Association for Computing Machinery, New York, NY, USA, Article Article 8, 6 pages. https://doi.org/10.1145/1937117.1937125
[28]
Thomas D. LaToza, Gina Venolia, and Robert DeLine. 2006. Maintaining Mental Models: A Study of Developer Work Habits. In Proceedings of the 28th International Conference on Software Engineering (ICSE '06). Association for Computing Machinery, New York, NY, USA, 492--501. https://doi.org/10.1145/1134285.1134355
[29]
Joseph Lawrance, Rachel Bellamy, Margaret Burnett, and Kyle Rector. 2008. Using Information Scent to Model the Dynamic Foraging Behavior of Programmers in Maintenance Tasks. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '08). Association for Computing Machinery, New York, NY, USA, 1323--1332. https://doi.org/10.1145/1357054.1357261
[30]
L. Layman, M. Diep, M. Nagappan, J. Singer, R. Deline, and G. Venolia. 2013. Debugging Revisited: Toward Understanding the Debugging Needs of Contemporary Software Developers. In 2013 ACM / IEEE International Symposium on Empirical Software Engineering and Measurement. 383--392. https://doi.org/10.1109/ESEM.2013.43
[31]
H. Li, Z. Xing, X. Peng, and W. Zhao. 2013. What help do developers seek, when and how?. In 2013 20th Working Conference on Reverse Engineering (WCRE). 142--151. https://doi.org/10.1109/WCRE.2013.6671289
[32]
Andre Meyer, Earl Barr, Christian Bird, and Thomas Zimmermann. 2019. Today was a Good Day: The Daily Life of Software Developers. IEEE Transactions on Software Engineering PP (03 2019), 1--1. https://doi.org/10.1109/TSE.2019.2904957
[33]
Luca Pascarella, Davide Spadini, Fabio Palomba, Magiel Bruntink, and Alberto Bacchelli. 2018. Information Needs in Contemporary Code Review. Proc. ACM Hum.-Comput. Interact. 2, CSCW, Article Article 135 (Nov. 2018), 27 pages. https://doi.org/10.1145/3274404
[34]
Shaun Phillips, Guenther Ruhe, and Jonathan Sillito. 2012. Information Needs for Integration Decisions in the Release Process of Large-Scale Parallel Development. In Proceedings of the ACM 2012 Conference on Computer Supported Cooperative Work (CSCW '12). Association for Computing Machinery, New York, NY, USA, 1371--1380. https://doi.org/10.1145/2145204.2145408
[35]
Peter L. T. Pirolli. 2007. Information Foraging Theory: Adaptive Interaction with Information (1 ed.). Oxford University Press, Inc., USA.
[36]
Tobias Roehm. 2015. Two User Perspectives in Program Comprehension: End Users and Developer Users. In Proceedings of the 2015 IEEE 23rd International Conference on Program Comprehension (ICPC '15). IEEE Press, 129--139.
[37]
Tobias Roehm, Rebecca Tiarks, Rainer Koschke, and Walid Maalej. 2012. How Do Professional Developers Comprehend Software?. In Proceedings of the 34th International Conference on Software Engineering (ICSE '12). IEEE Press, 255--265.
[38]
Christoffer Rosen and Emad Shihab. 2016. What are mobile developers asking about? A large scale study using stack overflow. Empirical Software Engineering 21, 3 (01 Jun 2016), 1192--1223. https://doi.org/10.1007/s10664-015-9379-3
[39]
Adrian Schröter, Jorge Aranda, Daniela Damian, and Irwin Kwan. 2012. To Talk or Not to Talk: Factors That Influence Communication around Changesets. In Proceedings of the ACM 2012 Conference on Computer Supported Cooperative Work (CSCW '12). Association for Computing Machinery, New York, NY, USA, 1317--1326. https://doi.org/10.1145/2145204.2145401
[40]
Adrian Schröter, Jorge Aranda, Daniela Damian, and Irwin Kwan. 2012. To Talk or Not to Talk: Factors That Influence Communication around Changesets. In Proceedings of the ACM 2012 Conference on Computer Supported Cooperative Work (CSCW '12). Association for Computing Machinery, New York, NY, USA, 1317--1326. https://doi.org/10.1145/2145204.2145401
[41]
Judith Segal, Antony Grinyer, and Helen Sharp. 2005. The Type of Evidence Produced by Empirical Software Engineers. In Proceedings of the 2005 Workshop on Realising Evidence-Based Software Engineering (REBSE '05). Association for Computing Machinery, New York, NY, USA, 1--4. https://doi.org/10.1145/1083174.1083176
[42]
Khaironi Sharif and Jim Buckley. 2009. Observation of Open Source programmers' information seeking. 307--308. https://doi.org/10.1109/ICPC.2009.5090071
[43]
Khaironi Y. Sharif, Michael English, Nour Ali, Chris Exton, J.J. Collins, and Jim Buckley. 2015. An empirically-based characterization and quantification of information seeking through mailing lists during Open Source developers' software evolution. Information and Software Technology 57 (2015), 77--94. https://doi.org/10.1016/j.infsof.2014.09.003
[44]
Jonathan Sillito, Gail C. Murphy, and Kris De Volder. 2008. Asking and Answering Questions during a Programming Change Task. IEEE Trans. Softw. Eng. 34, 4 (July 2008), 434--451. https://doi.org/10.1109/TSE.2008.26
[45]
D. I. K. Sjoeberg, J. E. Hannay, O. Hansen, V. B. Kampenes, A. Karahasanovic, N. Liborg, and A. C. Rekdal. 2005. A survey of controlled experiments in software engineering. IEEE Transactions on Software Engineering 31, 9 (Sep. 2005), 733--753. https://doi.org/10.1109/TSE.2005.97
[46]
Justin Smith, Brittany Johnson, Emerson Murphy-Hill, Bill Chu, and Heather Richter Lipford. 2015. Questions Developers Ask While Diagnosing Potential Security Vulnerabilities with Static Analysis. In Proceedings of the 2015 10th Joint Meeting on Foundations of Software Engineering (ESEC/FSE 2015). Association for Computing Machinery, New York, NY, USA, 248--259. https://doi.org/10.1145/2786805.2786812
[47]
Justin Smith, Brittany Johnson, Emerson Murphy-Hill, Bill Chu, and Heather Richter Lipford. 2015. Questions Developers Ask While Diagnosing Potential Security Vulnerabilities with Static Analysis. In Proceedings of the 2015 10th Joint Meeting on Foundations of Software Engineering (ESEC/FSE 2015). Association for Computing Machinery, New York, NY, USA, 248--259. https://doi.org/10.1145/2786805.2786812
[48]
J. Starke, C. Luce, and J. Sillito. 2009. Working with search results. In 2009 ICSE Workshop on Search-Driven Development-Users, Infrastructure, Tools and Evaluation. 53--56. https://doi.org/10.1109/SUITE.2009.5070023
[49]
A. Sutherland and G. Venolia. 2009. Can peer code reviews be exploited for later information needs?. In 2009 31st International Conference on Software Engineering - Companion Volume. 259--262. https://doi.org/10.1109/ICSE-COMPANION.2009.5070996
[50]
Yida Tao, Yingnong Dang, Tao Xie, Dongmei Zhang, and Sunghun Kim. 2012. How Do Software Engineers Understand Code Changes? An Exploratory Study in Industry. In Proceedings of the ACM SIGSOFT 20th International Symposium on the Foundations of Software Engineering (FSE '12). Association for Computing Machinery, New York, NY, USA, Article Article 51, 11 pages. https://doi.org/10.1145/2393596.2393656
[51]
Christos Tjortjis and Paul Layzell. 2001. Expert Maintainers' Strategies and Needs When Understanding Software: A Case Study Approach. In Proceedings of the Eighth Asia-Pacific on Software Engineering Conference (APSEC '01). IEEE Computer Society, USA, 281.
[52]
Christoph Treude, Ohad Barzilay, and Margaret-Anne Storey. 2011. How Do Programmers Ask and Answer Questions on the Web? (NIER Track). In Proceedings of the 33rd International Conference on Software Engineering (ICSE '11). Association for Computing Machinery, New York, NY, USA, 804--807. https://doi.org/10.1145/1985793.1985907
[53]
C. Treude and M. P. Robillard. 2017. Understanding Stack Overflow Code Fragments. In 2017 IEEE International Conference on Software Maintenance and Evolution (ICSME). 509--513. https://doi.org/10.1109/ICSME.2017.24
[54]
Davi Viana, Tayana Conte, and Cleidson R. B. de Souza. 2014. Knowledge Transfer between Senior and Novice Software Engineers: A Qualitative Analysis. In The 26th International Conference on Software Engineering and Knowledge Engineering, Hyatt Regency, Vancouver, BC, Canada, July 1-3, 2013. 235--240.
[55]
A. Von Mayrhauser and A. M. Vans. 1995. Program comprehension during software maintenance and evolution. Computer 28, 8 (Aug 1995), 44--55. https://doi.org/10.1109/2.402076
[56]
Yuhao Wu, Shaowei Wang, Cor-Paul Bezemer, and Katsuro Inoue. 2019. How do developers utilize source code from stack overflow? Empirical Software Engineering 24, 2 (01 Apr 2019), 637--673. https://doi.org/10.1007/s10664-018-9634-5
[57]
Yunwen Ye. 2006. Supporting Software Development as Knowledge-Intensive and Collaborative Activity. In Proceedings of the 2006 International Workshop on Workshop on Interdisciplinary Software Engineering Research (WISER '06). Association for Computing Machinery, New York, NY, USA, 15--22. https://doi.org/10.1145/1137661.1137666
[58]
Li Zhang, Jia-Hao Tian, Jing Jiang, Yi-Jun Liu, Meng-Yuan Pu, and Tao Yue. 2018. Empirical Research in Software Engineering --- A Literature Survey. Journal of Computer Science and Technology 33 (2018), 876--899.

Cited By

View all
  • (2023)A Field Study of Developer Documentation FormatExtended Abstracts of the 2023 CHI Conference on Human Factors in Computing Systems10.1145/3544549.3585767(1-7)Online publication date: 19-Apr-2023
  • (2023)Under the Bridge: Trolling and the Challenges of Recruiting Software Developers for Empirical Research StudiesProceedings of the 45th International Conference on Software Engineering: New Ideas and Emerging Results10.1109/ICSE-NIER58687.2023.00016(55-59)Online publication date: 17-May-2023
  • (2023)Information needs and presentation in agile software developmentInformation and Software Technology10.1016/j.infsof.2023.107265162:COnline publication date: 1-Oct-2023
  • Show More Cited By

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences
ICSEW'20: Proceedings of the IEEE/ACM 42nd International Conference on Software Engineering Workshops
June 2020
831 pages
ISBN:9781450379632
DOI:10.1145/3387940
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 25 September 2020

Permissions

Request permissions for this article.

Check for updates

Author Tags

  1. empirical studies
  2. information needs
  3. literature review

Qualifiers

  • Research-article
  • Research
  • Refereed limited

Conference

ICSE '20
Sponsor:
ICSE '20: 42nd International Conference on Software Engineering
June 27 - July 19, 2020
Seoul, Republic of Korea

Upcoming Conference

ICSE 2025

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)31
  • Downloads (Last 6 weeks)0
Reflects downloads up to 16 Feb 2025

Other Metrics

Citations

Cited By

View all
  • (2023)A Field Study of Developer Documentation FormatExtended Abstracts of the 2023 CHI Conference on Human Factors in Computing Systems10.1145/3544549.3585767(1-7)Online publication date: 19-Apr-2023
  • (2023)Under the Bridge: Trolling and the Challenges of Recruiting Software Developers for Empirical Research StudiesProceedings of the 45th International Conference on Software Engineering: New Ideas and Emerging Results10.1109/ICSE-NIER58687.2023.00016(55-59)Online publication date: 17-May-2023
  • (2023)Information needs and presentation in agile software developmentInformation and Software Technology10.1016/j.infsof.2023.107265162:COnline publication date: 1-Oct-2023
  • (2022)Recruiting credible participants for field studies in software engineering researchInformation and Software Technology10.1016/j.infsof.2022.107002151:COnline publication date: 1-Nov-2022
  • (2021)Overcoming Knowledge-Sharing Barriers that Affect Software QualityProceedings of the XX Brazilian Symposium on Software Quality10.1145/3493244.3493266(1-9)Online publication date: 8-Nov-2021

View Options

Login options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Figures

Tables

Media

Share

Share

Share this Publication link

Share on social media