Skip to main content
SpringerLink
Log in

Combining Evidence and Meta-analysis in Software Engineering

  • Chapter
Software Engineering (ISSSE 2010, ISSSE 2009, ISSSE 2011)

Part of the book series: Lecture Notes in Computer Science ((LNPSE,volume 7171))

Abstract

Recently there has been a welcome move to realign software engineering as an evidence-based practice. Many research groups are actively conducting empirical research e.g. to compare different fault prediction models or the value of various architectural patterns. However, this brings some challenges. First, for a particular question, how can we locate all the relevant evidence (primary studies) and make sense of them in an unbiased way. Second, what if some of these primary studies are inconsistent? In which case how do we determine the ‘true’ answer? To address these challenges, software engineers are looking to other disciplines where the systematic review is normal practice (i.e. systematic, objective, transparent means of locating, evaluating and synthesising evidence to reach some evidence-based answer to a particular question). This chapter examines the history of empirical software engineering, overviews different meta-analysis methods and then describe the process of systematic reviews and conclude with some future directions and challenges for researchers.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
eBook
USD 16.99
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 16.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. Altman, D.: Poor-quality medical research – what can journals do? JAMA 287(21), 2765–2767 (2002)

    Article  Google Scholar 

  2. Aronson, J.: A pragmatic view of thematic analysis. The Qualitative Report 2(1) (1994)

    Google Scholar 

  3. Avison, D., Lau, F., Myers, M., Nielsen, P.A.: Action research. CACM 42(1), 94–97 (1999)

    Article  Google Scholar 

  4. Basili, V.R., Shull, F., Lanubile, F.: Building knowledge through families of experiments. IEEE Transactions on Software Engineering 25(4), 456–473 (1999)

    Article  Google Scholar 

  5. Beynon-Davies, P.: Ethnography and information systems development: Ethnography of, for and within is development. Information & Software Technology 39(8), 531–540 (1997)

    Article  Google Scholar 

  6. Blettner, M., Sauerbrei, W., Schlehofer, B., Scheuchenpflug, T., Friedenreich, C.: Traditional reviews, meta-analyses and pooled analyses in epidemiology. Intl. J. of Epidemiology 28(1), 1–9 (1999)

    Article  Google Scholar 

  7. Braun, V., Clarke, V.: Using thematic analysis in psychology. Qualitative Research in Psychology 3(2), 77–101 (2006)

    Article  Google Scholar 

  8. Budgen, D., Charters, S., Turner, M., Brereton, P., Kitchenham, B., Linkman, S.: Investigating the applicability of the evidence-based paradigm to software engineering. In: WISER 2006, pp. 7–13. ACM, Shanghai (2006)

    Google Scholar 

  9. Chalmers, I.: The Cochrane Collaboration: preparing, maintaining, and disseminating systematic reviews of the effects of health care. Annals N.Y. Acad. Sci. 703, 153–163 (1995)

    Google Scholar 

  10. Cochrane Collaboration: The Cochrane Collaboration. Tech. rep., http://www.cochrane.org

  11. Cooper, H.: Scientific guidelines for conducting integrative research reviews. Review of Educational Research 52(summer), 291–302 (1982)

    Article  Google Scholar 

  12. Cooper, H.: Research synthesis and meta-analysis: A step-by-step approach, 4th edn. Sage, Thousand Oaks (2010)

    Google Scholar 

  13. Cruzes, D., Dybå, T.: Research synthesis in software engineering: A tertiary study. Information and Software Technology 53(5), 440–455 (2011)

    Article  Google Scholar 

  14. DARE: About DARE. Tech. rep., Centre for Reviews and Dissemination, York University, UK (2011), http://www.crd.york.ac.uk/cms2web/AboutDare.asp

  15. Dixon-Woods, M., Agarwal, S., Jones, D., Young, B., Sutton, A.: Synthesising qualitative and quantitative evidence: a review of possible methods. Journal of Health Services Research & Policy 10(1), 45–55 (2005)

    Article  Google Scholar 

  16. Dybå, T., Kitchenham, B., Jørgensen, M.: Evidence-based software engineering for practitioners. IEEE Software 22(1), 58–65 (2005)

    Article  Google Scholar 

  17. Egger, M., Schneider, M., Davey-Smith, G.: Meta-analysis spurious precision? Meta-analysis of observational studies. BMJ 316, 140 (1998)

    Article  Google Scholar 

  18. Ellis, P.: The Essential Guide to Effect Sizes: Statistical Power, Meta-Analysis, and the Interpretation of Research Results. Cambridge University Press (2010)

    Google Scholar 

  19. Evans, D.: Hierarchy of evidence: a framework for ranking evidence evaluating healthcare interventions. Journal of Clinical Nursing 12, 77–84 (2003)

    Article  Google Scholar 

  20. Eysenck, H.: Systematic reviews: Meta-analysis and its problems. BMJ 309, 789 (1994)

    Article  Google Scholar 

  21. Glass, G., McGaw, B., Smith, M.: Meta-analysis in social research. Sage Publications, Beverly Hills (1981)

    Google Scholar 

  22. Hayes, W.: Research synthesis in software engineering: a case for meta-analysis. In: 6th IEEE International Softw. Metrics Symp., pp. 143–151. IEEE Computer Society, Boca Raton (1999)

    Google Scholar 

  23. Higgins, J., Green, S. (eds.): Cochrane Handbook for Systematic Reviews of Interventions, Version 5.1.0. The Cochrane Collaboration (2011)

    Google Scholar 

  24. Higgins, J., Thompson, S., Deeks, J., Altman, D.: Measuring inconsistency in meta-analysis. British Medical Journal 327, 557–560 (2003)

    Article  Google Scholar 

  25. Jüni, P., Witschi, A., Bloch, R., Egger, M.: The hazards of scoring the quality of clinical trials for meta-analysis. JAMA 282(11), 1054–1060 (1999)

    Article  Google Scholar 

  26. Katrak, P., Bialocerkowski, A.E., Massy-Westropp, N., Kumar, S., Grimmer, K.A.: A systematic review of the content of critical appraisal tools. BMC Medical Research Methodology (Electronic Resource) 4(1), 22 (2004)

    Article  Google Scholar 

  27. Kitchenham, B., Brereton, P., Budgen, D., Turner, M., Bailey, J., Linkman, S.: Systematic literature reviews in software engineering a systematic literature review. Information & Software Technology 51(1), 7–15 (2009)

    Article  Google Scholar 

  28. Kitchenham, B., Charters, S.: Guidelines for performing systematic literature reviews in software engineering, version 2.3. Tech. Rep. EBSE Technical Report EBSE-2007-01. Keele University, UK (2007)

    Google Scholar 

  29. Kitchenham, B., Mendes, E., Travassos, G.: A systematic review of cross- vs. within-company cost estimation studies. In: 10th Intl. Conf. Empirical Assessment in Soft. Eng, EASE (2006)

    Google Scholar 

  30. Kitchenham, B., Pretorius, R., Budgen, D., Brereton, P., Turner, M., Niazi, M., Linkman, S.: Systematic literature reviews in software engineering a tertiary study. Information and Software Technology 52(8), 792–805 (2010)

    Article  Google Scholar 

  31. Kitchenham, B.A., Pfleeger, S.L., Pickard, L.M., Jones, P.W., Hoaglin, D.C., El Emam, K., Rosenberg, J.: Preliminary guidelines for empirical research in software engineering. IEEE Transactions on Software Engineering 28(8), 721–734 (2002)

    Article  Google Scholar 

  32. Kitchenham, B., Dybå, T., Jørgensen, M.: Evidence-based software engineering. In: 27th IEEE Intl. Softw. Eng. Conf. (ICSE 2004), IEEE Computer Society, Edinburgh (2004)

    Google Scholar 

  33. Luborsky, L., Diguer, L., Seligman, D., Rosenthal, R., Krause, E., Johnson, S., Halperin, G., Bishop, M., Berman, J., Schweizer, E.: The researcher’s own therapy allegiances: A wild card in comparisons of treatment efficacy. Clinical Psychology: Science and Practice 6(1), 95–106 (1999)

    Google Scholar 

  34. MacDonell, S., Shepperd, M., Kitchenham, B., Mendes, E.: How reliable are systematic reviews in empirical software engineering? IEEE Transactions on Software Engineering 36(5), 676–687 (2010)

    Article  Google Scholar 

  35. MacDonell, S., Shepperd, M.: Comparing local and global software effort estimation models reflections on a systematic review. In: 1st Intl. Symp. on Empirical Softw. Eng. & Measurement, Madrid (2007)

    Google Scholar 

  36. Michie, D., Spiegelhalter, D.J., Taylor, C.C.: Machine learning, neural and statistical classification. Ellis Horwood Series in Artificial Intelligence. Ellis Horwood, Chichester (1994)

    MATH  Google Scholar 

  37. Miller, J.: Can results from software engineering experiments be safely combined? In: Briand, L. (ed.) IEEE 6th Intl. Metrics Symp. IEEE Computer Society, Boca Raton (1999)

    Google Scholar 

  38. Naur, P., Randall, B. (eds.): Software Engineering: Report on a Conference by the NATO Science Committee. NATO Scientific Affairs Division, Brussels (1968)

    Google Scholar 

  39. Needleman, I.: A guide to systematic reviews. Journal of Clinical Periodontology 29(suppl. s3), 6–9 (2002)

    Article  Google Scholar 

  40. Oswald, M., Grosjean, S.: Confirmation bias. In: Pohl, R. (ed.) Cognitive Illusions: A Handbook on Fallacies and Biases in Thinking, Judgement and Memory, pp. 79–96. Psychology Press, Hove (2004)

    Google Scholar 

  41. Petticrew, M.: Systematic reviews from astronomy to zoology: Myths and misconceptions. British Medical Journal 322(7278), 98 (2001)

    Article  Google Scholar 

  42. Pickard, L., Kitchenham, B., Jones, P.: Combining empirical results in software engineering. Information & Software Technology 40(14), 811–821 (1998)

    Article  Google Scholar 

  43. Popay, J., Roberts, H., Sowden, A., Petticrew, M., Arai, L., Rodgers, M., Britten, N.: Guidance on the conduct of narrative synthesis in systematic reviews: A product from the ESRC methods programme. Tech. rep., Institute for Health Research, University of Lancaster (version 1, 2006)

    Google Scholar 

  44. Rosenthal, R.: The file drawer problem and tolerance for null results. Psychological Bulletin 86(3), 638–641 (1979)

    Article  Google Scholar 

  45. Sackett, D., Straus, S., Richardson, W., Rosenberg, W., Haynes, R.B.: Evidence-Based Medicine: How to Practice and Teach EBM, 2nd edn. Churchill Livingstone, Edinburgh (2000)

    Google Scholar 

  46. Seaman, C.: Qualitative methods in empirical studies of software engineering. IEEE Transactions on Software Engineering 25(4), 557–572 (1999)

    Article  Google Scholar 

  47. Shadish, W., Cook, T., Campbell, D.: Experimental and quasi-experimental designs for generalized causal inference. Houghton Mifflin, Boston (2002)

    Google Scholar 

  48. Shapiro, S.: Meta analysis/shmeta analysis. American J. of Epidemiology 140, 771–778 (1994)

    Google Scholar 

  49. da Silva, F., Santos, A., Soares, S., Franaa, A., Monteiro, C., Maciel, F.: Six years of systematic literature reviews in software engineering: An updated tertiary study. Information and Software Technology 53(9), 899–913 (2011)

    Article  Google Scholar 

  50. Sommerville, I.: Software Engineering, 5th edn. Addison-Wesley, Wokingham (1996)

    MATH  Google Scholar 

  51. Spencer, L., Ritchie, J., Lewis, J., Dillon, L.: Quality in qualitative evaluation: A framework for assessing research evidence. Tech. rep., The Cabinet Office (2003)

    Google Scholar 

  52. Spoor, P., Airey, M., Bennett, C., Greensill, J., Williams, R.: Use of the capture-recapture technique to evaluate the completeness of systematic literature searches. BMJ 313(7053), 342–343 (1996)

    Article  Google Scholar 

  53. Stroup, D., Berlin, J., Morton, S., Olkin, I., Williamson, G., Rennie, D., Moher, D., Becker, B., Sipe, T., Thacker, S.: Meta-analysis of observational studies in epidemiology: A proposal for reporting. JAMA 283(15), 2008–2012 (2000)

    Article  Google Scholar 

  54. Sutton, A., Higgins, J.: Recent developments in meta-analysis. Statistics in Medicine 27, 625–650 (2008)

    Article  MathSciNet  Google Scholar 

  55. Viller, S., Sommerville, I.: Ethnographically informed analysis for software engineers. International J. of Human-Computer Studies 53(1), 169–196 (2000)

    Article  Google Scholar 

  56. Wohlin, C., Runeson, P., Hst, M., Ohlsson, M., Regnell, B., Wessln, A.: Experimentation in Software Engineering: An Introduction. Kluwer Academic, Norwell (2000)

    Book  MATH  Google Scholar 

  57. Yancey, J.: Ten rules for reading clinical research reports. American J. of Surgery 159, 533–539 (1990)

    Article  Google Scholar 

  58. Yin, R.: Case Study Research - Design and Methods, 3rd edn. SAGE Publications (2003)

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Brunel University, London, UK

    Martin Shepperd

Authors
  1. Martin Shepperd
    View author publications

    You can also search for this author in PubMed Google Scholar

Editor information

Editors and Affiliations

  1. Software Engineering Lab, University of Salerno, Fisciano, SA, Italy

    Andrea De Lucia

  2. Università di Salerno, Via Ponte don Melillo, 84081, Fisciano, SA, Italy

    Filomena Ferrucci

Rights and permissions

Reprints and permissions

Copyright information

© 2013 Springer-Verlag Berlin Heidelberg

About this chapter

Cite this chapter

Shepperd, M. (2013). Combining Evidence and Meta-analysis in Software Engineering. In: De Lucia, A., Ferrucci, F. (eds) Software Engineering. ISSSE ISSSE ISSSE 2010 2009 2011. Lecture Notes in Computer Science, vol 7171. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-36054-1_2

Download citation

  • DOI: https://doi.org/10.1007/978-3-642-36054-1_2

  • Publisher Name: Springer, Berlin, Heidelberg

  • Print ISBN: 978-3-642-36053-4

  • Online ISBN: 978-3-642-36054-1

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation