Skip to main content
SpringerLink
Log in

Refining the systematic literature review process—two participant-observer case studies

  • Published:
Empirical Software Engineering Aims and scope Submit manuscript

Abstract

Systematic literature reviews (SLRs) are a major tool for supporting evidence-based software engineering. Adapting the procedures involved in such a review to meet the needs of software engineering and its literature remains an ongoing process. As part of this process of refinement, we undertook two case studies which aimed 1) to compare the use of targeted manual searches with broad automated searches and 2) to compare different methods of reaching a consensus on quality. For Case 1, we compared a tertiary study of systematic literature reviews published between January 1, 2004 and June 30, 2007 which used a manual search of selected journals and conferences and a replication of that study based on a broad automated search. We found that broad automated searches find more studies than manual restricted searches, but they may be of poor quality. Researchers undertaking SLRs may be justified in using targeted manual searches if they intend to omit low quality papers, or they are assessing research trends in research methodologies. For Case 2, we analyzed the process used to evaluate the quality of SLRs. We conclude that if quality evaluation of primary studies is a critical component of a specific SLR, assessments should be based on three independent evaluators incorporating at least two rounds of discussion.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2

Similar content being viewed by others

Notes

  1. This and other related technical reports are available from our website: www.ebse.org.uk

  2. This happened only once.

References

  • Bailey J, Budgen D, Turner M, Kitchenham B, Brereton P, Linkman S (2007) Evidence relating to Object-Oriented software design: A survey, Proceedings of Empirical Software Engineering and Measurement, IEEE Computer Society Press pp. 482–484

  • Barcelos RF, Travassos GH (2006) Evaluation approaches for Software Architectural Documents: A systematic Review, Ibero-American Workshop on Requirements Engineering and Software Environments (IDEAS). La Plata, Argentina

    Google Scholar 

  • Beecham S, Baddoo N, Hall T, Robinson H, Sharp H (2008) Motivation in Software Engineering: A systematic literature review. IST 50:860–878

    Google Scholar 

  • Bellini CG, Pereira RDCDF, Becker JL (2008) Measurement in Software Engineering from the Roadmap to the Crossroads. Int J Softw Eng Knowl 18(1):37–64

    Article  Google Scholar 

  • Bornmann L, Mutz R, Daniel D-D (2010)A reliability-generalization study of journal peer reviews—a multi-level analysis of inter-rater reliability and its determinants. (Submitted)

  • Brereton OP, Kitchenham BA (2007) The Scope of EPIC Case Studies. EPIC technical Report EPIC-2007–04

  • Brereton OP, Kitchenham BA, Budgen D, Turner M, Khalil MA (2007) Lessons from applying the Systematic Literature Review process within the Software Engineering domain. J Syst & Softw 80(4):571–583

    Article  Google Scholar 

  • Brereton P, Kitchenham B, Budgen D, Li Z (2008) Using a Protocol Template for Case Study Planning. Proceedings of EASE 2008, BCS-eWIC

  • Centre for Reviews and Dissemination (2007) What are the criteria for the inclusion of reviews on DARE? Available at http://www.york.ac.uk/inst/crd/faq4.htm (accessed 24 July 2007)

  • Davis A, Dieste O Hickey A, Juristo N, Moreno AM (2006) Effectiveness of Requirements Elicitation Techniques: Empirical Results Derived from a Systematic Review, 14th IEEE International Requirements Engineering Conference (RE’06), pp. 179–188

  • Davis A, Hickey A, Dieste O, Juristo N, Moreno AM (2007) A Quantitative Assessment of Requirements Engineering Publications–1963–2006, LNCS 4542/2007. Requirements Engineering, Foundation for Software Quality, pp 129–143

    Google Scholar 

  • de Boer RC, Farenhorst R (2008) In search of ’Architectural Knowledge’, SHARK '08: Proceedings of the 3 rd international workshop on Sharing and reusing architectural knowledge, May, pp 71–78

  • Dybå T, Dingsøyr T (2008) Empirical studies of agile software development: A systematic review. IST 50:833–859

    Google Scholar 

  • Dybå T, Kitchenham B, Jørgensen M (2005) Evidence-based Software Engineering for Practitioners. IEEE Softw 22(1):58–65

    Article  Google Scholar 

  • Feller J, Finnegan P, Kelly D, MacNamara M (2006) Developing Open Source Software: A Community-Based Analysis of Research, IFIP International Federation for Information Processing, 208/2006, Social Inclusion: Societal and Organizational Implications for Information Systems, pp 261–278

  • Fink A (2005) Conducting Research Literature Reviews. Conducting Research Literature Reviews. From the Internet to Paper. 2nd Edition Sage Publications Ltd.

  • Freire AP, Goularte R, Fortes RPM (2007) Techniques for developing more accessible web applications: a survey towards a process classification, SIGDOC '07: Proceedings of the 25th annual ACM international conference on Design of communication, October, pp 162–169

  • Glass RL, Vessey I, Ramesh V (2002) Research in software engineering: an analysis of the literature. Inf Softw Technol 44:491–506

    Article  Google Scholar 

  • Gómez O, Oktaba H, Piattini M, García F (2008) A Systematic Review Measurement in Software Engineering: State-of-the-Art in Measures ICSOFT 2006, CCIS 10. Lect Notes Comput Sci 5007:165–176

    Google Scholar 

  • Greenhalgh Trisha (2000) How to read a paper: The Basics of Evidence-Based Medicine. BMJ Books

  • Grimstad S, Jorgensen M, Møløkken-Østvold K (2005) The Clients’ Impact on Effort Estimation Accuracy in Software Development Projects, 11th IEEE International Software Metrics Symposium (METRICS'05), pp 3

  • Hannay J, Jørgensen M (2008) The Role of Deliberate Artificial Design Elements in Software Engineering Experiments. IEEE Trans Softw Eng 34(2):242–259

    Article  Google Scholar 

  • Hanssen GK, Bjørnson FO, Westerheim H (2007) Tailoring and Introduction of the Rational Unified Process. EuroSPI 2007, LNCS 4764, pp 7–18

  • Harjumaa L, Markkula J, Oivo M (2008) How does a Measurement Programme Evolve in Software Organizations? PROFES 2008. LNCS 5089:230–243

    Google Scholar 

  • Höfer A, Tichy WF (2007) Status of Empirical Research in Software Engineering, in V. Basili et al., (Eds) Empirical Software Engineering Issues, Springer-Verlag LNCS 4336, pp 10–19

  • Hosbond JH, Nielsen PA (2005) Mobile Systems Development—A literature review, Proceedings of IFIP 8.2 Annual Conference

  • Höst M, Wohlin C, Thelin T (2005) Experimental context classification: incentives and experience of subjects, ICSE’05, Proceedings of the 27th international conference on Software engineering, ACM

  • Jefferies C, Brereton P, Turner M (2008) A Systematic Literature review to investigate Reengineering Existing Systems for Multi-Chanel Access, Conference on Software Maintenance and Reengineering (CSMR). April, Athens, pp 258–262

    Google Scholar 

  • Jørgensen M (2004) A review of studies on expert estimation of software development effort. J Syst Softw 70(1–2):37–60

    Article  Google Scholar 

  • Jørgensen M (2005) Evidence-Based Guidelines for Assessment of Software Development Cost Uncertainty. IEEE Trans Softw Eng 2005:942–954

    Article  Google Scholar 

  • Jørgensen M (2007) Estimation of Software Development Work Effort: Evidence on Expert Judgement and Formal Models. Int J Forecasting 3(3):449–462

    Article  Google Scholar 

  • Jørgensen M, Shepperd M (2007) A Systematic Review of Software Development Cost Estimation Studies. IEEE Trans SE 33(1):33–53

    Article  Google Scholar 

  • Jørgensen M, Dybå T, Kitchenham BA (2005) Teaching Evidence-Based Software Engineering to University Students, 11th IEEE International Software Metrics Symposium (METRICS'05), p. 24

  • Juristo N, Moreno AM, Vegas S, Solari M (2006) In Search of What We Experimentally Know about Unit Testing. IEEE Softw 23(6):72–80

    Article  Google Scholar 

  • Kagdi H, Collard ML, Maletic JI (2006) A survey and taxonomy of approaches for mining software repositories in the context of software evolution. J Softw Maintenance Evol Res Pract 19(2):77–131

    Article  Google Scholar 

  • Kampenes VB, Dybå T, Hannay JE, Sjøberg DIK (2007) A Systematic Review of Effect Size in Software Engineering Experiments. Inf Softw Technol 49(11–12):1073–1086

    Article  Google Scholar 

  • Khan Khalid S, Kunz Regina, Kleijnen Jos, Antes Gerd (2003) Systematic Reviews to Support Evidence-based Medicine, The Royal Society of Medicine Press Ltd

  • Kitchenham BA (2004) Procedures for Undertaking Systematic Reviews, Joint Technical Report, Computer Science Department, Keele University (TR/SE-0401) and National ICT Australia Ltd (0400011 T.1)

  • Kitchenham BA, Charters S (2007) Guidelines for performing Systematic Literature Reviews in Software Engineering Technical Report EBSE/EPIC-2007-01, 2007

  • Kitchenham B, Dybå T, Jørgensen M (2004) Evidence-based Software Engineering. Proceedings of the 26th International Conference on Software Engineering, (ICSE ’04), IEEE Computer Society, Washington DC, USA, pp 273–281

  • Kitchenham B, Mendes E, Travassos GH (2007) A Systematic Review of Cross- vs. Within-Company Cost Estimation Studies. IEEE Trans SE 33(5):316–329

    Article  Google Scholar 

  • Kitchenham BA, Brereton OP, Turner M (2008a) EPIC Case Study 2—Extension of a Tertiary Study. EPIC Technical Report, EPIC-2009-007

  • Kitchenham BA, Brereton OP, Budgen D (2008b) Protocol for extending a Tertiary Study of Systematic Literature Reviews in Software Engineering. EPIC Technical Report, EBSE-2008-006, June

  • Kitchenham BA, Brereton OP, Budgen D, Turner M, Bailey J, Linkman SG (2009a) Systematic Literature reviews in Software Engineering—A Systematic Literature review. Inf Softw Technol 51:7–15

    Article  Google Scholar 

  • Kitchenham BA, Brereton P, Turner M, Niazi M, Linkman S, Pretorius R, Budgen D (2009b) The Impact of Limited Search Procedures for Systematic Literature Reviews—A Participant-Observer Case Study, Proceedings of the Third Symposium on Empirical Software Engineering and Measurement, ESEM’09, pp 336–345

  • Kitchenham B, Brereton P, Budgen D, Li Z (2009c). An Evaluation of Quality Checklist Proposals—A participant-observer case study, in Proceedings of EASE 2009, BCS-eWiC

  • Kitchenham B, Pretorius R, Budgen D, Brereton OP, Turner M, Niazi M, Linkman S (2010a) Systematic Literature Reviews—A Tertiary Study, Information and Software Technology, accepted for publication.

  • Kitchenham BA, Budgen D, Brereton P (2010b) The value of mapping studies—An observer-participant case study. EASE 2010

  • Liebchen GA, Shepperd M (2008) Data sets and Data Quality in Software Engineering. PROMISE '08: Proceedings of the 4th international workshop on Predictor models in software engineering, May 2008, pp 39–44.

  • MacDonell S, Shepperd M (2007) Comparing local and global effort estimation models reflections on a systematic reviews. Proceedings of Empirical Software Engineering and Measurement, IEEE Computer Society Press

  • Mair M, Shepperd M, Jørgensen M (2005) An analysis of data sets used to train and validate cost prediction systems, PROMISE’05 Workshop

  • Martin BR (1996) The use of multiple indicators in the assessment of basic research. Scientometrics 36(3):343–362

    Article  Google Scholar 

  • Martin BR, Irvine J (1983) Assessing Basic Research. Some partial indicators of scientific progress in radio astronomy. Res Policy 12:61–90

    Article  Google Scholar 

  • Mohagheghi P, Conradi R (2007) Quality, productivity and economic benefits of software reuse: a review of industrial studies. Empirical Softw Eng 12:471–516

    Article  Google Scholar 

  • Mohagheghi P, Dehlen V (2008) Where Is the Proof?—A Review of Experiences from Applying MDE in Industry. ECMDA-FA 2008, LNCS 5095, pp. 432–443

  • Neto AD, Subramanyan R, Viera M, Travassos GH Shull F (2008) Improving Evidence about Software Technologies. A Look at model-based testing. IEEE Softw 25(6):242–249

    Google Scholar 

  • Petticrew Mark, Helen Roberts (2005) Systematic Reviews in the Social Sciences: A Practical Guide, Blackwell Publishing

  • Pino FJ, García F, Piattini M (2008) Software process improvement in small and medium enterprises: a review. Softw Qual J 16:237–261

    Article  Google Scholar 

  • Ramesh V, Glass RL, Vessey I (2004) Research in computer science: an empirical study. J Syst Softw 70(1–2):165–176

    Article  Google Scholar 

  • Renger M, Kolfschoten GL, de Vreede G-J (2008) Challenges in Collaborative Modeling: A Literature Review, CIAO! 2008 and EOMAS 2008, LNBIP 10, 2008, pp 61–77

  • Segal J, Grinyer A, Sharp H (2005) The type of evidence produced by empirical software engineers. REBSE’05

  • Shaw M, Clements P (2006) The Golden Age of Software Architecture: A Comprehensive Survey. Technical Report CMU-ISRI-06-101, Software Engineering Institute, Carnegie Mellon University

  • Shepperd M (2007) Software project economics: a roadmap, FOSE’07.

  • Sjøberg DIK, Hannay JE, Hansen O, Kampenes VB, Karahasanovic A, Liborg NK, Rekdal AC (2005) A survey of controlled experiments in software engineering. IEEE Trans SE 31(9):733–753

    Article  Google Scholar 

  • Staples M, Niazi M (2008) Systematic review of organizational motivation for adopting CMM-based SPI. Inf Softw Technol 50(7–8):605–620

    Article  Google Scholar 

  • Tichy W, Lukowicz P, Prechelt L, Heinze E (1995) Experimental Evaluation in Computer Science: A Quantitative Study. J Syst Softw 28(9):9–18

    Article  Google Scholar 

  • Torchiano M, Morisio M (2004) Overlooked Aspects of COTS-Based Development. IEEE Software, pp 88–93

  • Turner M, Kitchenham B, Budgen D, Brereton P (2008) Lessons learnt Undertaking a Large-scale Systematic Literature Review, in Proceedings of EASE 2008, BCS-eWiC

  • Weller AC (2001) Editorial Peer Review. Its Strengths and Weaknesses, Assist Monograph Series, Nre Jersey, USA

    Google Scholar 

  • Wicks MN, Dewar RG (2007) A new research agenda for tool integration, J. Syst Softw 80:1567–1585

    Google Scholar 

  • Yalaho A (2006) A Conceptual Model of ICT-Supported Unified Process of International Outsourcing of Software Production, 10th International Enterprise Distributed Object Computing Conference Workshops (EDOCW’06)

  • Yin Robert K (2003) Case Study Research: Design and Methods, 3 rd Edition, Sage Publications

  • Zhang H, Kitchenham B, Pfahl D (2008) Reflections on 10 years of Software Process Simulation Modeling: A systematic Review. International Workshop on Software Process Simulation Modeling. LNCS 5007:345–356

    Google Scholar 

Download references

Acknowledgment

This study was funded by the UK Engineering and Physical Sciences Research Council project EPIC/E046983/1. We thank the anonymous reviewers for their helpful comments.

Author information

Authors and Affiliations

  1. School of Computing and Mathematics, Keele University, Stoke-on-Trent, ST5 5BG, UK

    Barbara A. Kitchenham, Pearl Brereton, Mark Turner, Mahmood K. Niazi & Stephen Linkman

  2. School of Engineering and Computing Sciences, Durham University, Durham, DH1 3LE, UK

    Rialette Pretorius & David Budgen

Authors
  1. Barbara A. Kitchenham
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Pearl Brereton
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Mark Turner
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Mahmood K. Niazi
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Stephen Linkman
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Rialette Pretorius
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. David Budgen
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Barbara A. Kitchenham.

Additional information

Editor: James Miller

Rights and permissions

Reprints and permissions

About this article

Cite this article

Kitchenham, B.A., Brereton, P., Turner, M. et al. Refining the systematic literature review process—two participant-observer case studies. Empir Software Eng 15, 618–653 (2010). https://doi.org/10.1007/s10664-010-9134-8

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10664-010-9134-8

Keywords

Search

Navigation