research-article

Strength of evidence in systematic reviews in software engineering

Authors:

Torgeir DingsøyrAuthors Info & Claims

ESEM '08: Proceedings of the Second ACM-IEEE international symposium on Empirical software engineering and measurement

Pages 178 - 187

https://doi.org/10.1145/1414004.1414034

Published: 09 October 2008 Publication History

Abstract

Systematic reviews are only as good as the evidence they are based on. It is important, therefore, that users of systematic reviews know how much confidence they can place in the conclusions and recommendations arising from such reviews. In this paper we present an overview of some of the most influential systems for assessing the quality of individual primary studies and for grading the overall strength of a body of evidence. We also present an example of the use of such systems based on a systematic review of empirical studies of agile software development. Our findings suggest that the systems used in other disciplines for grading the strength of evidence for and reporting of systematic reviews, especially those that take account of qualitative and observational studies are of particular relevance for software engineering.

References

[1]

Arisholm, E., Gallis, H., Dybå, T., and Sjøberg, D. (2007) Evaluating Pair Programming with Respect to System Complexity and Programmer Expertise, IEEE Transactions on Software Engineering, 33(2): 65--86

Digital Library

[2]

Bailey, J., Zhang, C., Budget, D., and Turner, M., "Search Engine Overlaps: Do They Agree or Disagree?," Second International Workshop on Realising Evidence-Based Software Engineering (REBSE'07), 2007.

Digital Library

[3]

Bhandari, M., Busse, J. W., Jackowski, D., Montori, V. M., Schünemann, H., Sprague, S., Mears, D., Schemitsch, E. H., Heels-Ansdell, D., and Devereaux, P. J. (2004) Association between Industry Funding and Statistically Significant Pro-Industry Findings in Medical and Surgical Randomized Trials, CMAJ, 170: 477--480

[4]

Biolchini, J., Mian, P. G., Natali, A. C. C., and Travassos, G. H. (2005) Systematic Review in Software Engineering, Univ. Rio de Janeiro, TR, ES 679/05.

[5]

Brehmer, B. (1989) In One Word: Not from Experience, Acta Psychologica, 45(1-3): 223--241.

[6]

Brereton, P., Kitchenham, B. A., Budgen, D., Turner, M., and Khalil, M., "Lessons from Applying the Systematic Literature Review Process within the Software Engineering Domain," Journal of Systems and Software, no. 4, vol. 80, pp. 571--583, 2007.

Digital Library

[7]

Campbell, D. T. and Stanley, J. C. (1963) Experimental and Quasi-Experimental Designs for Research, Boston: Houghton Mifflin Company

[8]

Cook, T. D. and Campbell, D. T. (1979) Quasi-Experimentation: Design & Analysis Issues for Field Settings, Boston: Houghton Mifflin Company

[9]

Cooper, H. (1998) Synthesizing Research (3rd Ed.), Thousand Oaks, CA: Sage.

[10]

Cooper, H. and Hedges, L. V. (Eds.) (1994) Handbook of Research Synthesis, New York: Russell Sage Foundation.

Digital Library

[11]

Davies, A., Dieste, O., Hickey, A., Juristo, N., and Moreno, A. M. (2006) Effectiveness of Requirements Elicitation Techniques: Empirical Results Derived from a Systematic Review, Proceedings 14th IEEE International Requirements Engineering Conference (RE'06), IEEE Computer Society, pp. 179--188.

Digital Library

[12]

Dieste, O. and Padua, A. G. (2007) Developing Search Strategies for Detecting Relevant Experiments for Systematic Reviews, Proceedings of the 1st International Symposium on Empirical Software Engineering and Measurement (ESEM'07), Madrid, Spain, 20-21 Sept., IEEE Computer Society, pp. 215--224.

Digital Library

[13]

Dittrich, Y., John, M., Singer, J., and Tessem, B. (2007) For the Special issue on Qualitative Software Engineering Research, Information and Software Technology, 6(49): 531--539.

Digital Library

[14]

Dixon-Woods, M., Agarwal, S., Jones, D., Young, B., and Sutton, A. (2005) Synthesising Qualitative and Quantitative Evidence: A Review of Possible Methods, J. of Health Services Research & Policy, 10(1): 45--53.

[15]

Dybå, T. and Dingsøyr, T. (2008) Empirical Studies of Agile Software Development: A Systematic Review, Information and Software Technology, 50(9-10): 833--859.

Digital Library

[16]

Dybå, T., Dingsøyr, T., and Hanssen, G. K. (2007) Applying Systematic Reviews to Diverse Study Types: An Experience Report, Proceedings of the 1st International Symposium on Empirical Software Engineering and Measurement (ESEM'07), Madrid, Spain, 20-21 Sept., IEEE Computer Society, pp. 225--234.

Digital Library

[17]

Dybå, T., Kampenes, V. B. and Sjøberg, D. I. K. (2006) A Systematic Review of Statistical Power in Software Engineering Experiments, Information and Software Technology, 48(8):745--755.

[18]

Dybå, T., Kitchenham, B. A., and Jørgensen, M. (2005) Evidence-Based Software Engineering for Practitioners, IEEE Software, 22(1): 58--65.

Digital Library

[19]

Egger, M., Smith, G. D., and Altman, D. G. (2001) Systematic Reviews in Health Care: Meta-analysis in Context (2nd Ed.), London: BMJ Publishing Group.

[20]

GRADE Working Group (2004) Grading Quality of Evidence and Strength of Recommendations," BMJ, 328: 1490.

[21]

Greenhalgh, T. (2006) How to Read a Paper: The Basics of Evidence-Based Medicine (3rd Ed.), London: BMJ Publishing Group.

[22]

Hannay, J., Sjøberg, D. I. K., and Dybå, T. (2007) A Systematic Review of Theory Use in SE Experiments, IEEE Transactions on Software Engineering, 33(2): 87--107.

Digital Library

[23]

Higgins J. P. T. and Green, S. (Eds.) (2008), Cochrane Handbook for Systematic Reviews of Interventions, Version 5.0.0 (updated February 2008), The Cochrane Collaboration, available from www.cochrane-handbook.org.

[24]

Höst, M. and Runeson, P. (2007) Checklists for Software Engineering Case Study Research, Proceedings of the First International Symposium on Empirical Software Engineering and Measurement (ESEM'07), Madrid, Spain, 20-21 Sept., IEEE Computer Society, pp. 479--481

Digital Library

[25]

Jadad, A. R., Moore, R. A., Carroll, D., Jenkinson, C., Reynolds, D. J., Gavaghan, D. J., and McQuay, H. J. (1996) Assessing the Quality of Reports of Randomized Clinical Trials: Is Blinding Necessary?, Controlled Clinical Trials, 17(1): 1--12.

[26]

Jedlitschka, A. and Pfahl, D. (2005) Reporting Guidelines for Controlled Experiments in Software Engineering, Proceedings of the 4th International Symposium on Empirical Software Engineering (ISESE'05), Noosa Heads, Australia, 17-18 Nov, IEEE Computer Society, pp. 95--104.

[27]

Jørgensen, M. and Shepperd, M. (2007) A Systematic Review of Software Development Cost Estimation Studies, IEEE Transactions on Software Engineering, 33(1): 33--53.

[28]

Kampenes, V. B., Dybå, T., Hannay, J. E., and Sjøberg, D. I. K. (2007) A Systematic Review of Effect Size in Software Engineering Experiments, Information and Software Technology, 49(11-12): 1073--1086.

Digital Library

[29]

Kampenes, V. B., Dybå, T., Hannay, J. E., and Sjøberg, D. I. K. (In Press) A Systematic Review of Quasi-Experiments in Software Engineering, Accepted to Information and Software Technology.

Digital Library

[30]

Khan, K. S. ter Riet, G., Glanville, J., Sowden, A. J., and Kleijnen, J. (Eds.) (2001) Undertaking Systematic Review of Research on Effectiveness, CRD's Guidance for those Carrying Out or Commissioning Reviews, CRD Report Number 4 (2nd Ed.), NHS Centre for Reviews and Dissemination, University of York.

[31]

Kitchenham, B. A. (2007) Empirical Paradigm - The Role of Experiments, in V. R. Basili et al. (Eds.), Empirical Software Engineering Issues: Critical Assessment and Future Directions, Proceedings from Int. Workshop, Dagstuhl Castle, June 26-30, 2006, Lecture Notes in Compute Science 4336, Springer, pp. 25--32.

Digital Library

[32]

Kitchenham, B. A. and Charters, S. (2007) Guidelines for performing Systematic Literature Reviews in Software Engineering, Version 2.3, Keele University, EBSE Technical Report, EBSE-2007-01.

[33]

Kitchenham, B. A., Al-Khilidar, H., Babar, M. A., Berry, M., Cox, K., Keung, J., Kurniawati, F., Staples, M., Zhang, H., and Zhu, L. (2008) Evaluating Guidelines for Reporting Empirical Software Engineering Studies, Empirical Software Engineering, 13(1): 97--121.

Digital Library

[34]

Kitchenham, B. A., Dybå, T., and Jørgensen, M. (2004) Evidence-Based Software Engineering, Proceedings of the 26th International Conference on Software Engineering (ICSE 2004), IEEE CS Press, pp. 273--281.

Digital Library

[35]

Kitchenham, B. A., Pfleeger, S. L., Pickard, L. M., Jones, P. W., Hoaglin, D. C., El Emam, K., and Rosenberg, J. (2002) Preliminary Guidelines for Empirical Research in Software Engineering, IEEE Transactions on Software Engineering, 28(8): 721--734.

Digital Library

[36]

Klein, H. K. and Myers, M. D. (1999) A Set of Principles for Conducting and Evaluating Interpretive Field Studies in Information Systems, MIS Quarterly, 23(1): 67--93.

Digital Library

[37]

Laitenberger, O. and Rombach, D. (2003) (Quasi-) Experimental Studies in Industrial Settings, in N. Juristo and A. M. Moreno (Eds.), Lecture Notes on Empirical Software Engineering, Singapore: World Scientific (Series on Software Engineering and Knowledge Engineering 12), pp. 167--227.

Digital Library

[38]

Lee, A. S. (1989) A Scientific Methodology for MIS Case Studies, MIS Quarterly, 13(1): 33--50.

Digital Library

[39]

Mcbreen, P. (2003) Questioning Extreme Programming, Boston, MA, USA: Pearson Education.

Digital Library

[40]

Mendes, E. (2005) A Systematic Review of Web Engineering Research, Proceedings of the 4th International Symposium on Empirical Software Engineering (ISESE'05), Noosa Heads, Australia, 17-18 Nov, IEEE Computer Society, pp. 481--490.

[41]

Moher D., Schultz K. F., Altman D. (2001) The CONSORT Statement: Revised Recommendations for Improving the Quality of Reports of Parallel-Group Randomized Trials, Lancet, 357:1191--1194, April 14.

[42]

Moher D., Cook, D. J., Eastwood, S., Olkin, I., Rennie, D., and Stroup, D. F. (1999) Improving the Quality of Reports of Meta-Analyses of Randomised Controlled Trials: The QUOROM Statement, Lancet, 354: 1896--1900.

[43]

Mulrow, C. and Cook, D. (Eds.) (1998) Systematic Reviews: Synthesis of Best Evidence for Health Care Decisions, Philadelphia: Am. College of Physicians.

[44]

Noblit, G. W. and Hare, R. D. (1988) Meta-Ethnography: Synthesizing Qualitative Studies, Thousand Oaks: Sage.

[45]

Petticrew, M. and Roberts, H. (2006) Systematic Reviews in the Social Sciences: A Practical Guide, Oxford, UK: Blackwell.

[46]

Shadish, W. R., Cook, T. D. and Campbell, D. T. (2002), Experimental and Quasi-Experimental Designs for Generalized Causal Inference, Boston: Houghton Mifflin.

[47]

Sjøberg, D. I. K., Dybå, T., and Jørgensen, M. (2007) The Future of Empirical Methods in Software Engineering Research, 29th International Conference on Software Engineering (ICSE'07), Future of Software Engineering (FOSE'07), Minneapolis, Minnesota, USA, 20-26 May, IEEE Computer Society Press IEEE Computer Society, pp. 358--378.

Digital Library

[48]

Sjøberg, D. I. K., Hannay, J. E., Hansen, O., Kampenes, V. B., Karahasanović, A., Liborg, N.-K., and Rekdal, A. C. (2005) A Survey of Controlled Experiments in Software Engineering, IEEE Transactions on Software Engineering, 31(9):733--753,

Digital Library

[49]

Staples, M. and Niazi, M. (2007) Experiences Using Systematic Review Guidelines, Journal of Systems and Software, 80(9): 1425--1437.

Digital Library

[50]

Stroup, D. F., Berlin, J. A., Morton, S. C., Olkin, I., Williamson, G. D., Rennie, D., Moher, D., Becker, B. J., Sipe, T. A., and Thacker, S. B. (2000) Meta-analysis of Observational Studies in Epidemiology: A Proposal for Reporting, JAMA, 283(15): 2008--2012.

Cited By

Busch NZalewski A(2025)A Systematic Literature Review of Enterprise Architecture Evaluation MethodsACM Computing Surveys10.1145/370658257:5(1-36)Online publication date: 9-Jan-2025
https://dl.acm.org/doi/10.1145/3706582
Sobrín-Hidalgo DManuel Guerrero-Higueras ÁMatellán-Olivera V(2025)Generating Explanations for Autonomous Robots: A Systematic ReviewIEEE Access10.1109/ACCESS.2025.353509713(20413-20426)Online publication date: 2025
https://doi.org/10.1109/ACCESS.2025.3535097
Schmitt V(2025)Disentangling patent quality: using a large language model for a systematic literature reviewScientometrics10.1007/s11192-024-05206-w130:1(267-311)Online publication date: 11-Jan-2025
https://doi.org/10.1007/s11192-024-05206-w
Show More Cited By

Index Terms

Strength of evidence in systematic reviews in software engineering
1. Software and its engineering

Recommendations

Evidence-Based Software Engineering and Systematic Reviews
Using systematic reviews and evidence-based software engineering with masters students
EASE'09: Proceedings of the 13th international conference on Evaluation and Assessment in Software Engineering

Context: The problem of teaching research skills to masters students. In particular, improving their literature reviews, assessing them and providing good feedback.

Objectives: To introduce systematic reviews and evidence-based software engineering (...
The use of systematic reviews in evidence based software engineering: a systematic mapping study
ESEM '14: Proceedings of the 8th ACM/IEEE International Symposium on Empirical Software Engineering and Measurement

Context. A decade ago, Kitchenham, Dybå and Jørgensen argued that software engineering could benefit from an evidence-based research approach similar that that used in medicine, introducing the basis for Evidence Based Software Engineering (EBSE). ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

ESEM '08: Proceedings of the Second ACM-IEEE international symposium on Empirical software engineering and measurement

October 2008

374 pages

ISBN:9781595939715

DOI:10.1145/1414004

General Chair:
Dieter Rombach
University of Kaiserslautern and Fraunhofer Institute for Experimental Software Engineering (Germany)
,
Program Chairs:
Sebastian Elbaum
University of Nebraska-Lincoln (USA)
,
Jürgen Münch
Fraunhofer Institute for Experimental Software Engineering (Germany)

Copyright © 2008 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 09 October 2008

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

ESEM '08

Sponsor:

ESEM '08: 2008 ACM-IEEE International Symposium on Empirical Software Engineering and Measurement

October 9 - 10, 2008

Kaiserslautern, Germany

Acceptance Rates

Overall Acceptance Rate 130 of 594 submissions, 22%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

128
Total Citations
View Citations
2,249
Total Downloads

Downloads (Last 12 months)153
Downloads (Last 6 weeks)18

Reflects downloads up to 01 Mar 2025

Other Metrics

View Author Metrics

Citations

Cited By

Busch NZalewski A(2025)A Systematic Literature Review of Enterprise Architecture Evaluation MethodsACM Computing Surveys10.1145/370658257:5(1-36)Online publication date: 9-Jan-2025
https://dl.acm.org/doi/10.1145/3706582
Sobrín-Hidalgo DManuel Guerrero-Higueras ÁMatellán-Olivera V(2025)Generating Explanations for Autonomous Robots: A Systematic ReviewIEEE Access10.1109/ACCESS.2025.353509713(20413-20426)Online publication date: 2025
https://doi.org/10.1109/ACCESS.2025.3535097
Schmitt V(2025)Disentangling patent quality: using a large language model for a systematic literature reviewScientometrics10.1007/s11192-024-05206-w130:1(267-311)Online publication date: 11-Jan-2025
https://doi.org/10.1007/s11192-024-05206-w
Posso RAltmann J(2025)Trust and Trust-Building Policies to Support Cybersecurity Information Sharing: A Systematic Literature ReviewEconomics of Grids, Clouds, Systems, and Services10.1007/978-3-031-81226-2_19(212-228)Online publication date: 6-Feb-2025
https://doi.org/10.1007/978-3-031-81226-2_19
Santa Barletta VCaivano DPal AScalera MSerrano Martin M(2025)Enabling Quantum Privacy and Security by DesignJournal of Software: Evolution and Process10.1002/smr.7000537:2Online publication date: 6-Feb-2025
https://dl.acm.org/doi/10.1002/smr.70005
Cardoso JCamilo SPicolo J(2024)Inovações tecnológicas no setor elétrico: revisão sistemática e metassínteseRevista de Gestão e Secretariado10.7769/gesec.v15i7.402815:7(e4028)Online publication date: 17-Jul-2024
https://doi.org/10.7769/gesec.v15i7.4028
Sepúlveda SCravero AFonseca GAntonelli L(2024)Systematic Review on Requirements Engineering in Quantum Computing: Insights and Future DirectionsElectronics10.3390/electronics1315298913:15(2989)Online publication date: 29-Jul-2024
https://doi.org/10.3390/electronics13152989
Milella FBandini S(2024)Fostering Artificial Intelligence-based supports for informal caregivers: a systematic review of the literatureIntelligenza Artificiale10.3233/IA-240028(1-21)Online publication date: 2-Jul-2024
https://doi.org/10.3233/IA-240028
Muzammel CSpichkova MHarland J(2024)Cultural influence on RE activities: An extended analysis of state of the artAdjunct Proceedings of the 26th International Conference on Mobile Human-Computer Interaction10.1145/3640471.3680236(1-8)Online publication date: 21-Sep-2024
https://dl.acm.org/doi/10.1145/3640471.3680236
Ouaddi CBenaddi LBouziane EJakimi AChehri ASaadane R(2024) DSL ‐Driven Approaches and Metamodels for Chatbot Development: A Systematic Literature Review Expert Systems10.1111/exsy.13787Online publication date: 13-Nov-2024
https://doi.org/10.1111/exsy.13787
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten