Skip to main content
Springer Nature Link
Log in

Teaching Systematic Literature Reviews: Strategies and Best Practices

  • Chapter
  • First Online:
Handbook on Teaching Empirical Software Engineering

  • 253 Accesses

Abstract

Systematic literature reviews have become an important method in software engineering in order to obtain new findings and propose innovative ideas for future research. In this chapter, we share strategies and best practices on teaching systematic literature reviews.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Al-Zubidy, A., Carver, J.C.: Identification and prioritization of SLR search tool requirements: an SLR and a survey. Empir. Software Eng. 24(1), 139–169 (2019). https://doi.org/10.1007/s10664-018-9626-5

    Article  Google Scholar 

  2. Al-Zubidy, A., Carver, J.C., Hassler, E.E.: Vision for SLR tooling infrastructure: prioritizing value-added requirements. Inf. Software Technol. 91, 72–81 (2017). https://doi.org/10.1016/j.infsof.2017.06.007

    Article  Google Scholar 

  3. Ampatzoglou, A., Bibi, S., Avgeriou, P., Verbeek, M., Chatzigeorgiou, A.: Identifying, categorizing and mitigating threats to validity in software engineering secondary studies. Inf. Software Technol. 106, 201–230 (2019). https://doi.org/10.1016/j.infsof.2018.10.006

    Article  Google Scholar 

  4. Beecham, S., Baddoo, N., Hall, T., Robinson, H., Sharp, H.: Motivation in Software Engineering: A systematic literature review. Inf. Software Technol. 50(9–10), 860–878 (2008). https://doi.org/10.1016/j.infsof.2007.09.004

    Article  Google Scholar 

  5. Bjørnson, F., Dingsøyr, T.: Knowledge management in software engineering: a systematic review of studied concepts: findings and research methods used. Inf. Software Technol. 50(11), 1055–1068 (2008). https://doi.org/10.1016/j.infsof.2008.03.006

    Article  Google Scholar 

  6. Bowes, D., Hall, T., Beecham, S.: SLuRp: a tool to help large complex systematic literature reviews deliver valid and rigorous results. In: Proceedings of the 2nd International Workshop on Evidential Assessment of Software Technologies (EAST’12), pp. 33–36 (2012). https://doi.org/10.1145/2372233.2372243

  7. Brereton, P. Kitchenham, B., Budgen, D., Turner, M., Khalil, M.: Lessons from applying the systematic literature review process within the software engineering domain. J. Syst. Software 80(1), 571–583 (2007). https://doi.org/10.1016/j.jss.2006.07.009

    Article  Google Scholar 

  8. Budgen, D., Brereton, P., Drummond, S., Williams, N.: Reporting systematic reviews: some lessons from a tertiary study. Inf. Software Technol. 95, 62–74 (2018). https://doi.org/10.1016/j.infsof.2017.10.017

    Article  Google Scholar 

  9. Carver, J.C., Hassler, E.E., Hernandes, E., Kraft, N.A.: Identifying Barriers to the Systematic Literature Review Process. In: 2013 ACM/IEEE International Symposium on Empirical Software Engineering and Measurement (ESEM), pp. 203–212 (2013). https://doi.org/10.1109/ESEM.2013.28

  10. Cooper, H., Hedges, L., Valentine, J.: The Handbook of Research Synthesis and Meta-Analysis, 2nd edn. Russell Sage Foundation (2009)

    Google Scholar 

  11. Costal, D., Farré, C., Franch, X., Quer, C.: How tertiary studies perform quality assessment of secondary studies in software engineering. In: Proceedings of the 24th Iberoamerican Conference on Software Engineering (CIbSE), pp. 56–69 (2021a). https://arxiv.org/abs/2110.03820

  12. Costal, D., Farré, C., Franch, X., Quer, C.: Inclusion and exclusion criteria in software engineering tertiary studies: a systematic mapping and emerging framework. In: Proceedings of the 15th ACM/IEEE International Symposium on Empirical Software Engineering and Measurement (ESEM), pp. 1–6 (2021b). https://doi.org/10.1145/3475716.3484190

  13. Cruzes, D., Dybå, T.: Research synthesis in software engineering. A tertiary study. Inf. Software Technol. 53(5), 440–455 (2011a). https://doi.org/10.1016/j.infsof.2011.01.004

    Article  Google Scholar 

  14. Díaz, O., Contell, J.P.: Developing research questions in conversation with the literature: operationalization y tool support. Empir. Software Eng. 27(7), 174 (2022). https://doi.org/10.1007/s10664-022-10204-8

    Article  Google Scholar 

  15. Díaz, O., Genero, M., Contell, J.P., Piattini, M.: Adding relevance to rigor: assessing the contributions of SLRs in software engineering through citation context analysis. Comput. Sci. Rev. (in press) (2024)

    Google Scholar 

  16. Dixon-Woods, M., Agarwal, S., Jones, D., Young, B., Sutton, A.: Synthesising qualitative and quantitative evidence: a review of possible methods. J. Health Serv. Res. Policy 10(1), 45–53 (2005). https://doi.org/10.1177/135581960501000110

    Article  Google Scholar 

  17. Dybå, T., Dingsøyr, T.: Strength of evidence in systematic reviews in software engineering. In: Proceedings of the Second ACM-IEEE International Symposium on Empirical Software Engineering and Measurement (ESEM), pp. 178–187 (2008a). https://doi.org/10.1145/1414004.1414034

  18. Dybå, T., Dingsøyr, T.: Empirical studies of agile software development: A systematic review. Inf. Software Technol. 50(9–10), 833–859 (2008b). https://doi.org/10.1016/j.infsof.2008.01.006

    Article  Google Scholar 

  19. Fabbri, S., Silva, C., Hernandes, E., Octaviano, F., Di Thommazo, A., Belgamo, A.: Improvements in the StArt tool to better support the systematic review process. In: Proceedings of the 20th International Conference on Evaluation and Assessment in Software Engineering (EASE’16), pp. 1–5 (2016). https://doi.org/10.1145/2915970.2916013

  20. Fernández, A., Insfrán, E., Abrahão, S.: Usability evaluation methods for the web: a systematic mapping study. Inf. Software Technol. 53(8), 789–817 (2011). https://doi.org/10.1016/j.infsof.2011.02.007

    Article  Google Scholar 

  21. Fernández-Sáez, A.M., Genero, M., Chaudron, M.R.V.: Empirical studies concerning the maintenance of UML diagrams and their use in the maintenance of code: A systematic mapping study. Inf. Software Technol. 55(7), 1119–1142 (2013). https://doi.org/10.1016/j.infsof.2012.12.006

    Article  Google Scholar 

  22. Garousi, V., Felderer, M., Mäntylä, M.V.: Guidelines for including grey literature and conducting multivocal literature reviews in software engineering. Inf. Software Technol. 106, 101–121 (2019). https://doi.org/10.1016/j.infsof.2018.09.006

    Article  Google Scholar 

  23. Genero, M., Fernández-Sáez, A.M., Nelson, H.J., Poels, G., Piattini, M.: Research review: A systematic literature review on the quality of UML models. J. Database Manag. 22(3), 46–70 (2011). https://doi.org/10.4018/jdm.2011070103

    Article  Google Scholar 

  24. Genero, M., Piattini, M., Cruz-Lemus, J.A.: Métodos de investigación en informática, 2nd edn. Amazon (2023)

    Google Scholar 

  25. Giuffrida, R., Dittrich, Y.: Empirical studies on the use of social software in global software development – A systematic mapping study. Inf. Software Technol. 55(7), 1143–1164 (2013). https://doi.org/10.1016/j.infsof.2013.01.004

    Article  Google Scholar 

  26. Hall, T., Beecham, S., Bowes, D., Gray, D., Counsell, S.: A Systematic Literature Review on Fault Prediction Performance in Software Engineering. IEEE Trans. Software Eng. 38(6), 1276–1304 (2012). https://doi.org/10.1109/TSE.2011.103

    Article  Google Scholar 

  27. Ivarsson, M., Gorschek, T.: Technology transfer decision support in requirements engineering research: a systematic review of REj. Requir. Eng. 14(3), 155–175 (2009). https://doi.org/10.1007/s00766-009-0080-1

    Article  Google Scholar 

  28. Iwazaki, A.Y., dos Santos, V., Felizardo, K.R., de Souza, E.F., Valentim, N.M.C., Nakagawa, E.Y.: Benefits and Challenges of a Graduate Course: An Experience Teaching Systematic Literature Review. In: IEEE Frontiers in Education Conference, pp. 1–9 (2022). https://doi.org/10.1109/FIE56618.2022.9962446

  29. Kitchenham, B., Charters, S.: Guidelines for performing systematic literature reviews in software engineering. Informe Técnico EBSE-2007-01. Keele University (2007)

    Google Scholar 

  30. Kitchenham, B., Mendes, E., Travassos, G.: Cross versus within-company cost estimation studies: a systematic review. IEEE Trans. Software Eng. 33(5), 316–329 (2007). https://doi.org/10.1109/TSE.2007.1001

    Article  Google Scholar 

  31. Kitchenham, B., Budgen, D., Brereton, P.: Using mapping studies as the basis for further research – A participant-observer case study. Inf. Software Technol. 53(6), 638–651 (2011). https://doi.org/10.1016/j.infsof.2010.12.011

    Article  Google Scholar 

  32. Kitchenham, B., Brereton, P.: A systematic review of systematic review process research in software engineering. Inf. Software Technol. 55(12), 2049–2075 (2013). https://doi.org/10.1016/j.infsof.2013.07.010

    Article  Google Scholar 

  33. Kitchenham, B.A., Budgen, D., Brereton, P.: Evidence-Based Software Engineering and Systematic Reviews. Chapman & Hall/CRC (2015). https://dl.acm.org/doi/10.5555/2994449

  34. Kitchenham, B., Madeyski, L., Budgen, D.: Supplementary Material for SEGRESS: Software Engineering Guidelines for REporting Secondary Studies (2022). [Online]. Available: https://madeyski.e-informatyka.pl/download/SEGRESS22supplement.pdf

  35. Kitchenham, B., Madeyski, L., Budgen, D.: SEGRESS: Software engineering guidelines for REporting secondary studies. IEEE Trans. Software Eng. 49(3), 1273–1298 (2023). https://doi.org/10.1109/TSE.2022.3174092

    Article  Google Scholar 

  36. Kuhrmann, M., Fernández, D.M., Daneva, M.: On the pragmatic design of literature studies in software engineering: an experience-based guideline. Empir. Software Eng. 22(6), 2852–2891 (2017). https://doi.org/10.1007/s10664-016-9492-y

    Article  Google Scholar 

  37. Liu C., Houdek, R.: Teaching Computer Science Graduate Students. Scholarly Literature Review Techniques. In: Proceedings. Frontiers in Education. 36th Annual Conference, pp. 14–19 (2006). https://doi.org/10.1109/FIE.2006.322349

  38. Marshall, C., Brereton, P., Kitchenham, B.: Tools to support systematic reviews in software engineering: a feature analysis. In: Proceedings of the 18th International Conference on Evaluation and Assessment in Software Engineering (EASE’14), pp. 1–10 (2014). https://doi.org/10.1145/2601248.2601270

  39. Marshall, C., Brereton, P: Systematic review toolbox: a catalogue of tools to support systematic reviews. In: Proceedings of the 19th International Conference on Evaluation and Assessment in Software Engineering (EASE’15), pp. 1–6 (2015)

    Google Scholar 

  40. Marshall, C., Brereton, P., Kitchenham, B.: Tools to support systematic reviews in software engineering: a cross-domain survey using semi-structured interviews. In: Proceedings of the 19th International Conference on Evaluation and Assessment in Software Engineering (EASE’15), pp. 1–6 (2015). https://doi.org/10.1145/2745802.2745827

  41. Mendes, E., Wohlin, C., Felizardo, K.R., Kalinowski, M.: When to update systematic literature reviews in software engineering. J. Syst. Software 167, 110607 (2020). https://doi.org/10.1016/j.jss.2020.110607

    Article  Google Scholar 

  42. Molléri, J.S., Benitti, F.B.V.: SESRA: a web-based automated tool to support the systematic literature review process. In: Proceedings of the 19th International Conference on Evaluation and Assessment in Software Engineering (EASE’15), pp. 1–6 (2015). https://doi.org/10.1145/2745802.2745825

  43. Mourão, E., Pimentel, J.F., Murta, L., Kalinowski, M., Mendes, E., Wohlin, C.: On the performance of hybrid search strategies for systematic literature reviews in software engineering. Inf. Software Technol. 123, 106294 (2020). https://doi.org/10.1016/j.infsof.2020.106294

    Article  Google Scholar 

  44. Munir, H., Wnuk, K., Runeson, P.: Open innovation in software engineering: a systematic mapping study. Empir. Software Eng. 21(2), 684–723 (2016). https://doi.org/10.1007/s10664-015-9380-x

    Article  Google Scholar 

  45. Noblit, G., Hare, R.: Meta-Ethnography: Synthesizing Qualitative Studies. Sage (1988)

    Google Scholar 

  46. Oriol, M., Marco, J., Franch, X.: Quality models for web services: A systematic mapping. Inf. Software Technol. 56(10), 1167–1182 (2014). https://doi.org/10.1016/j.infsof.2014.03.012

    Article  Google Scholar 

  47. Parsifal: Parsifal systematic literature review tool (2015) [Online; accessed 1–July-2024]

    Google Scholar 

  48. Paternoster, N., Giardino, C., Unterkalmsteiner, M., Gorschek, T., Abrahamsson, P.: Software development in startup companies: A systematic mapping study. Inf. Software Technol. 56(10), 1200–1218 (2014). https://doi.org/10.1016/j.infsof.2014.04.014

    Article  Google Scholar 

  49. Pedreira, O., García, F., Brisaboa, N., Piattini, M.: Gamification in software engineering – A systematic mapping. Inf. Software Technol. 57, 157–168 (2015). https://doi.org/10.1016/j.infsof.2014.08.007

    Article  Google Scholar 

  50. Petersen, K.: Case study identification with GPT-4 and implications for mapping studies. Inf. Software Technol. 171, 107452 (2024). https://doi.org/10.1016/j.infsof.2024.107452

    Article  Google Scholar 

  51. Petersen, K., Feldt, R., Mujtaba, S., Mattsson, M.: Systematic mapping studies in software engineering. In: Proceedings of the 12th international conference on Evaluation and Assessment in Software Engineering (EASE), pp. 68–77 (2008). https://dl.acm.org/doi/10.5555/2227115.2227123

  52. Petersen, K., Vakkalanka, S., Kuzniarz, L.: Guidelines for conducting systematic mapping studies in software engineering: an update. Inf. Software Technol. 64, 1–18 (2015). https://doi.org/10.1016/j.infsof.2015.03.007

    Article  Google Scholar 

  53. Pino, F., García, F., Piattini, M.: Software process improvement in small and medium software enterprises: a systematic review. Software Q. J. 16(2), 237–261 (2008). https://doi.org/10.1007/s11219-007-9038-z

    Article  Google Scholar 

  54. Pizard, S., Acerenza, F., Otegui, X., Moreno, S., Vallespir, D., Kitchenham, B.: Training students in evidence-based software engineering and systematic reviews: a systematic review and empirical study. Empir. Software Eng. 26(3), 50 (2021). https://doi.org/10.1007/s10664-021-09953-9

    Article  Google Scholar 

  55. Riaz, M., Breaux, T., Williams, L.: How have we evaluated software pattern application? A systematic mapping study of research design practices. Inf. Software Technol. 65, 14–38 (2015). https://doi.org/10.1016/j.infsof.2015.04.002

    Google Scholar 

  56. Sierra, J.M., Vizcaíno, A., Genero, M., Piattini, M.: A systematic mapping study about socio-technical congruence. Inf. Software Technol. 94, 111–129 (2018). https://doi.org/10.1016/j.infsof.2017.10.004

    Article  Google Scholar 

  57. Šmite, U., Wohlin, C., Gorschek, T., Feldt, R.: Empirical evidence in global software engineering: a systematic review. Empir. Software Eng. 15, 91–118 (2010). https://doi.org/10.1007/s10664-009-9123-y

    Article  Google Scholar 

  58. Staples, M., Niazi, M.: Systematic review: systematic review of organizational motivations for adopting CMM-based SPI. Inf. Software Technol. 50(7-8), 605–620 (2008). https://doi.org/10.1016/j.infsof.2007.07.003

    Article  Google Scholar 

  59. Turner, M., Kitchenham, B., Brereton, P., Charters, S., Budgen, D.: Does the technology acceptance model predict actual use? A systematic literature review. Inf. Software Technol. 52(5), 463–479 (2010). https://doi.org/10.1016/j.infsof.2009.11.005

    Article  Google Scholar 

  60. Usman, M., Bin Ali, N., Wohlin, C.: A Quality Assessment Instrument for Systematic Literature Reviews in Software Engineering. e Informatica Software Eng. J. 17(1), 230105 (2023). https://doi.org/10.37190/e-inf

  61. Wieringa, R., Maiden, N., Mead, N., Rolland, C.: Requirements engineering paper classification and evaluation criteria: a proposal and a discussion. Requir. Eng. 11(1), 102–107 (2006). https://doi.org/10.1007/s00766-005-0021-6

    Article  Google Scholar 

  62. Wohlin, C.: Guidelines for snowballing in systematic literature studies and a replication in Software Engineering. In: International Conference on Evaluation and Assessment in Software Engineering (EASE), pp. 321–330 (2014). https://doi.org/10.1145/2601248.2601268

  63. Wohlin, C., Runeson, P., Höst, M., Ohlsson, M. C., Regnell, B., Wesslén, A.: Experimentation in Software Engineering. Springer (2012). https://doi.org/10.1007/978-3-642-29044-2

  64. Wohlin, C., Runeson, P., da Mota Silveira Neto, P.A., Engström, E., do Carmo Machado, I., de Almeida, E.S.: On the reliability of mapping studies in software engineering. J. Syst. Software 86(10), 2594–2610 (2013). https://doi.org/10.1016/j.jss.2013.04.076

  65. Wohlin, C., Mendes, E., Felizardo, K.R., Kalinowski, M.: Guidelines for the search strategy to update systematic literature reviews in software engineering. Inf. Software Technol. 127, 106366 (2020). https://doi.org/10.1016/j.infsof.2020.106366

    Article  Google Scholar 

  66. Wohlin, C., Kalinowski, M., Felizardo, K.R., Mendes, E.: Successful combination of database search and snowballing for identification of primary studies in systematic literature studies. Inf. Software Technol. 147, 106908 (2022). https://doi.org/10.1016/j.infsof.2022.106908

    Article  Google Scholar 

  67. Yang, L., Zhang, H., Shen, H., Huang, X., Zhou, X., Rong, G., Shao, D.: Quality assessment in systematic literature reviews: a software engineering perspective. Inf. Software Technol. 130, 106397 (2021). https://doi.org/10.1016/j.infsof.2020.106397

    Article  Google Scholar 

  68. Zhang, H., Ali Babar, M.: Systematic reviews in software engineering: an empirical investigation. Inf. Software Technol. 55(7), 1341–1354 (2013). https://doi.org/10.1016/j.infsof.2012.09.008

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of Technologies and Information Systems, University of Castilla-La Mancha, Ciudad Real, Spain

    Marcela Genero & Mario Piattini

Authors
  1. Marcela Genero
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Mario Piattini
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Marcela Genero .

Editor information

Editors and Affiliations

  1. Blekinge Institute of Technology and fortiss, Karlskrona, Sweden

    Daniel Mendez

  2. University of Groningen, Groningen, The Netherlands

    Paris Avgeriou

  3. Pontifical Catholic University of Rio de Janeiro, Rio de Janeiro, Brazil

    Marcos Kalinowski

  4. Blekinge Institute of Technology, Karlskrona, Sweden

    Nauman Bin Ali

Rights and permissions

Reprints and permissions

Copyright information

© 2024 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this chapter

Check for updates. Verify currency and authenticity via CrossMark

Cite this chapter

Genero, M., Piattini, M. (2024). Teaching Systematic Literature Reviews: Strategies and Best Practices. In: Mendez, D., Avgeriou, P., Kalinowski, M., Ali, N.B. (eds) Handbook on Teaching Empirical Software Engineering. Springer, Cham. https://doi.org/10.1007/978-3-031-71769-7_20

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-71769-7_20

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-71768-0

  • Online ISBN: 978-3-031-71769-7

  • eBook Packages: EducationEducation (R0)

Publish with us

Policies and ethics