Skip to main content
Springer Nature Link
Log in

Teaching Literature Reviewing for Software Engineering Research

  • Chapter
  • First Online:
Handbook on Teaching Empirical Software Engineering

  • 241 Accesses

Abstract

The goal of this chapter is to support teachers in holistically introducing graduate students to literature reviews, with a particular focus on secondary research. It provides an overview of the overall literature review process and the different types of literature review before diving into guidelines for selecting and conducting different types of literature review. The chapter also provides recommendations for evaluating the quality of existing literature reviews and concludes with a summary of our learning goals and how the chapter supports teachers in addressing them.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    https://www2.sigsoft.org/EmpiricalStandards/docs/standards

  2. 2.

    https://www.prisma-statement.org/prisma-2020-flow-diagram

References

  1. Arksey, H., O’Malley, L.: Scoping studies: towards a methodological framework. Int. J. Soc. Res. Methodol. 8(1), 19–32 (2005)

    Article  Google Scholar 

  2. Bajwa, S.S., Wang, X., Duc, A.N., Abrahamsson, P.: “Failures” to be celebrated: an analysis of major pivots of software startups. Empir. Softw. Eng. 22(5), 2373–2408 (2017)

    Article  Google Scholar 

  3. Baltes, S., Ralph, P.: Sampling in software engineering research: A critical review and guidelines. Empir. Softw. Eng. 27(94) (2022)

    Google Scholar 

  4. Borenstein, M., Hedges, L.V., Higgins, J.P., Rothstein, H.R.: Introduction to Meta-Analysis. Wiley, Chichester (2021)

    Book  Google Scholar 

  5. Bullock, R., Tubbs, M.E.: The case meta-analysis method for od. In: Research in Organizational Change and Development, vol 1, pp 171–228. JAI Press, Stamford (1987)

    Google Scholar 

  6. Cartaxo, B., Pinto, G., Soares, S.: Rapid reviews in software engineering. In: Felderer, M., Travassos, G.H. (eds) Contemporary Empirical Methods in Software Engineering, pp 357–384. Springer International Publishing, Cham (2020). https://doi.org/10.1007/978-3-030-32489-6_3

    Chapter  Google Scholar 

  7. Chalmers, I.: The Cochrane collaboration: preparing, maintaining, and disseminating systematic reviews of the effects of health care. Ann. N. Y. Acad. Sci. 703(1), 156–165 (1993)

    Article  Google Scholar 

  8. Cruzes, D.S., Dybå, T.: Research synthesis in software engineering: a tertiary study. Inf. Softw. Technol. 53(5), 440–455 (2011)

    Article  Google Scholar 

  9. Ganann, R., Ciliska, D., Thomas, H.: Expediting systematic reviews: methods and implications of rapid reviews. Implement. Sci. 5(1), 1–10 (2010)

    Article  Google Scholar 

  10. Garousi, V., Felderer, M., Mäntylä, M.V.: Guidelines for including grey literature and conducting multivocal literature reviews in software engineering. Inf. Softw. Technol. 106, 101–121 (2019). https://doi.org/10.1016/j.infsof.2018.09.006

    Article  Google Scholar 

  11. Glass, G.V.: Primary, secondary, and meta-analysis of research. Edu. Res. 5(10), 3–8 (1976)

    Article  Google Scholar 

  12. Gough, D., Oliver, S., Thomas, J.: An Introduction to Systematic Reviews. Sage, London (2017)

    Google Scholar 

  13. Hannay, J.E., Dybå, T., Arisholm, E., Sjøberg, D.I.: The effectiveness of pair programming: a meta-analysis. Inf. Softw. Technol. 51(7), 1110–1122 (2009)

    Article  Google Scholar 

  14. Huang, X., Zhang, H., Zhou, X., Babar, M.A., Yang, S.: Synthesizing qualitative research in software engineering: A critical review. In: Proceedings of the 40th International Conference on Software Engineering, Association for Computing Machinery, New York, NY, ICSE ’18, pp. 1207–1218 (2018). https://doi.org/10.1145/3180155.3180235

  15. Jensen, L.A., Allen, M.N.: Meta-synthesis of qualitative findings. Qualit. Health Res. 6(4), 553–560 (1996)

    Article  Google Scholar 

  16. Jurisch, M.C., Wolf, P., Krcmar, H.: Using the case survey method for synthesizing case study evidence in information systems research. In: Proceedings of the 19th Americas Conference on Information Systems (AMCIS), Association for Information Systems, Chicago, Article 5 (2013)

    Google Scholar 

  17. Kitchenham, B.A., Dyba, T., Jorgensen, M.: Evidence-based software engineering. In: Proceedings of the 26th International Conference on Software Engineering, IEEE Computer Society, Washington, D.C., ICSE ’04, pp. 273–281 (2004)

    Google Scholar 

  18. Larsson, R.: Case survey methodology: quantitative analysis of patterns across case studies. Acad. Manag. J. 36(6), 1515–1546 (1993)

    Article  Google Scholar 

  19. Lucas, W.A.: The case survey method: Aggregating case experience. Technical Report 1515, Rand (1974). https://www.rand.org/pubs/reports/R1515.html

  20. Maybee, J.E.: Hegel’s dialectics. In: Zalta, E.N. (ed.) The Stanford Encyclopedia of Philosophy, Winter 2020 edn. Metaphysics Research Lab, Stanford University, Stanford (2020)

    Google Scholar 

  21. Melegati, J., Wang, X.: Case survey studies in software engineering research. In: Proceedings of the 14th ACM / IEEE International Symposium on Empirical Software Engineering and Measurement (ESEM), Association for Computing Machinery, New York, NY, ESEM ’20 (2020). https://doi.org/10.1145/3382494.3410683

  22. Mohanani, R., Salman, I., Turhan, B., Rodríguez, P., Ralph, P.: Cognitive biases in software engineering: a systematic mapping study. IEEE Trans. Softw. Eng. 46(12), 1318–1339 (2018)

    Article  Google Scholar 

  23. Munn, Z., Peters, M.D., Stern, C., Tufanaru, C., McArthur, A., Aromataris, E.: Systematic review or scoping review? Guidance for authors when choosing between a systematic or scoping review approach. BMC Med. Res. Methodol. 18(1), 1–7 (2018)

    Google Scholar 

  24. Naur, P., Randell, B.: Software Engineering: Report of a Conference Sponsored by the NATO Science Committee, Garmisch, 7–11 Oct. 1968, Brussels, Scientific Affairs Division, NATO (1969)

    Google Scholar 

  25. Petersen, K.: Guidelines for case survey research in software engineering. In: Felderer, M., Travassos, G.H. (eds.) Contemporary Empirical Methods in Software Engineering, pp 63–92. Springer International Publishing, Cham (2020). https://doi.org/10.1007/978-3-030-32489-6_3

    Chapter  Google Scholar 

  26. Petersen, K., Feldt, R., Mujtaba, S., Mattsson, M.: Systematic mapping studies in software engineering. In: Proceedings of the 12th International Conference on Evaluation and Assessment in Software Engineering, BCS Learning & Development, Swindon, GBR, EASE’08, pp. 68–77 (2008)

    Google Scholar 

  27. Petersen, K., Badampudi, D., Shah, S.M.A., Wnuk, K., Gorschek, T., Papatheocharous, E., Axelsson, J., Sentilles, S., Crnkovic, I., Cicchetti, A.: Choosing component origins for software intensive systems: In-house, COTS, OSS or outsourcing?—a case survey. IEEE Trans. Softw. Eng. 44(3), 237–261 (2017)

    Article  Google Scholar 

  28. Purssell, E., McCrae, N.: How to Perform a Systematic Literature Review: A Guide for Healthcare Researchers, Practitioners and Students. Springer International Publishing, Cham (2020). https://doi.org/10.1007/978-3-030-49672-2_2

  29. Rafique, Y., Mišić, V.B.: The effects of test-driven development on external quality and productivity: a meta-analysis. IEEE Trans. Softw. Eng. 39(6), 835–856 (2012)

    Article  Google Scholar 

  30. Ralph, P.: Toward methodological guidelines for process theories and taxonomies in software engineering. IEEE Trans. Softw. Eng. 45(7), 712–735 (2018)

    Article  Google Scholar 

  31. Ralph, P., Baltes, S.: Paving the way for mature secondary research: the seven types of literature review. In: Roychoudhury, A., Cadar, C., Kim, M. (eds.) Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, ESEC/FSE 2022, Singapore, November 14–18, 2022, ACM, pp. 1632–1636 (2022). https://doi.org/10.1145/3540250.3560877

  32. Ralph, P., Baltes, S., Bianculli, D., Dittrich, Y., Felderer, M., Feldt, R., Filieri, A., Furia, C.A., Graziotin, D., He, P., et al.: ACM SIGSOFT empirical standards. arXiv: 201003525 [csSE] (2020)

    Google Scholar 

  33. Ricoeur, P.: Hermeneutics and the Human Sciences: Essays on Language, Action and Interpretation. Cambridge University Press, New York (1981)

    Google Scholar 

  34. Runeson, P., Höst, M.: Guidelines for conducting and reporting case study research in software engineering. Empir. Softw. Eng. 14(2), 131–164 (2009). https://doi.org/10.1007/s10664-008-9102-8

    Article  Google Scholar 

  35. Runeson, P., Höst, M., Rainer, A., Regnell, B.: Case Study Research in Software Engineering - Guidelines and Examples. Wiley, Hoboken (2012). http://eu.wiley.com/WileyCDA/WileyTitle/productCd-1118104358.html

    Book  Google Scholar 

  36. Sadowski, C., Zimmermann, T. (eds.): Rethinking Productivity in Software Engineering. Apress Open/Springer, Berlin (2019). https://doi.org/10.1007/978-1-4842-4221-6

    Google Scholar 

  37. Santos, R.D.S., Ralph, P., Arshad, A., Stol, K.J.: Distributed scrum: a case meta-analysis. ACM Computing Surveys 56(4), 1–37 (2023). https://doi.org/10.1145/3626519

    Article  Google Scholar 

  38. Shepperd, M., Bowes, D., Hall, T.: Researcher bias: the use of machine learning in software defect prediction. IEEE Trans. Softw. Eng. 40(6), 603–616 (2014)

    Article  Google Scholar 

  39. Sollaci, L.B., Pereira, M.G.: The introduction, methods, results, and discussion (IMRAD) structure: a fifty-year survey. J. Med. Library Assoc. 92(3), 364 (2004)

    Google Scholar 

  40. Sterne, J.A.C., Harbord, R.M.: Funnel plots in meta-analysis. Stata J. 4(2), 127–141 (2004). https://doi.org/10.1177/1536867X0400400204

    Article  Google Scholar 

  41. Stol, K.J., Ralph, P., Fitzgerald, B.: Grounded theory in software engineering research: A critical review and guidelines. In: Proceedings of the 38th International Conference on Software Engineering, Association for Computing Machinery, New York, NY, ICSE ’16, pp. 120–131 (2016). https://doi.org/10.1145/2884781.2884833

  42. Walsh, R.: The methods of reflexivity. Humanist. Psychol. 31(4), 51–66 (2003). https://doi.org/10.1080/08873267.2003.9986934

    Article  Google Scholar 

  43. Wohlin, C.: Guidelines for snowballing in systematic literature studies and a replication in software engineering. In: Shepperd, M.J., Hall, T., Myrtveit, I. (eds.) 18th International Conference on Evaluation and Assessment in Software Engineering, EASE ’14, London, England, May 13–14, 2014, ACM, pp. 38:1–38:10 (2014). https://doi.org/10.1145/2601248.2601268

  44. Wohlin, C., Kalinowski, M., Felizardo, K.R., Mendes, E.: Successful combination of database search and snowballing for identification of primary studies in systematic literature studies. Inf. Softw. Technol. 147, 106908 (2022). https://doi.org/10.1016/j.infsof.2022.106908

    Article  Google Scholar 

  45. Yin, R.K.: Case Study Research: Design and Methods, vol. 5. Sage, Thousand Oaks (2009)

    Google Scholar 

  46. Yin, R.K., Heald, K.A.: Using the case survey method to analyze policy studies. Adm. Sci. Quart. 20(3), 371–381 (1975). http://www.jstor.org/stable/2391997

    Article  Google Scholar 

  47. Zhang, H., Huang, X., Zhou, X., Huang, H., Babar, M.A.: Ethnographic research in software engineering: A critical review and checklist. In: Proceedings of the 2019 27th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, Association for Computing Machinery, New York, NY, ESEC/FSE 2019, pp. 659–670 (2019). https://doi.org/10.1145/3338906.3338976

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. University of Bayreuth, Bayreuth, Germany

    Sebastian Baltes

  2. Dalhousie University, Halifax, NS, Canada

    Paul Ralph

Authors
  1. Sebastian Baltes
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Paul Ralph
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Sebastian Baltes .

Editor information

Editors and Affiliations

  1. Blekinge Institute of Technology and fortiss, Karlskrona, Sweden

    Daniel Mendez

  2. University of Groningen, Groningen, The Netherlands

    Paris Avgeriou

  3. Pontifical Catholic University of Rio de Janeiro, Rio de Janeiro, Brazil

    Marcos Kalinowski

  4. Blekinge Institute of Technology, Karlskrona, Sweden

    Nauman Bin Ali

Rights and permissions

Reprints and permissions

Copyright information

© 2024 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this chapter

Check for updates. Verify currency and authenticity via CrossMark

Cite this chapter

Baltes, S., Ralph, P. (2024). Teaching Literature Reviewing for Software Engineering Research. In: Mendez, D., Avgeriou, P., Kalinowski, M., Ali, N.B. (eds) Handbook on Teaching Empirical Software Engineering. Springer, Cham. https://doi.org/10.1007/978-3-031-71769-7_19

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-71769-7_19

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-71768-0

  • Online ISBN: 978-3-031-71769-7

  • eBook Packages: EducationEducation (R0)

Publish with us

Policies and ethics