Skip to main content
SpringerLink
Log in

Towards Multi-Method Research Approach in Empirical Software Engineering

  • Conference paper
Product-Focused Software Process Improvement (PROFES 2009)

Part of the book series: Lecture Notes in Business Information Processing ((LNBIP,volume 32))

Abstract

This paper presents results of a literature analysis on Empirical Research Approaches in Software Engineering (SE). The analysis explores reasons why traditional methods, such as statistical hypothesis testing and experiment replication are weakly utilized in the field of SE. It appears that basic assumptions and preconditions of the traditional methods are contradicting the actual situation in the SE. Furthermore, we have identified main issues that should be considered by the researcher when selecting the research approach. In virtue of reasons for weak utilization of traditional methods we propose stronger use of Multi-Method approach with Pragmatism as the philosophical standpoint.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 39.99
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. Shaw, M.: What makes good research in software engineering? International Journal of Software Tools for Technology Transfer 4(1), 1–7 (2002)

    Article  Google Scholar 

  2. Kitchenham, B.: Guidelines for performing systematic literature reviews in software engineering. Technical report, TR-EBSE-2007-01, UK (2007)

    Google Scholar 

  3. Juristo, N., Moreno, A.: Basics of Software Engineering Experimentation. Kluwer Academic Publishers, Dordrecht (2003)

    MATH  Google Scholar 

  4. Wohlin, C., Runeson, P., Horst, M., Ohlsson, M., Regnell, B., Wesslen, A.: Experimentation in Software Engineering: An Introduction. Kluwer Academic Publishers, Dordrecht (2000)

    Book  MATH  Google Scholar 

  5. Turabian, K.: A Manual for Writers of Research Papers, Theses, and Dissertations. The University of Chicago Press, Chicago (2007)

    Book  Google Scholar 

  6. Fenton, N., Pfleeger, S., Glass, R.: Science and substance: A challenge to software engineers. IEEE Software 4(11), 86–95 (1994)

    Article  Google Scholar 

  7. Basili, V., Shull, F., Lanubile, F.: Building knowledge through families of experiments. IEEE Transactions on Software Engineering 25(4), 456–473 (1999)

    Article  Google Scholar 

  8. Oivo, M.: New opportunities for empirical research. In: Basili, V.R., Rombach, H.D., Schneider, K., Kitchenham, B., Pfahl, D., Selby, R.W. (eds.) Empirical Software Engineering Issues. LNCS, vol. 4336, p. 22. Springer, Heidelberg (2007)

    Chapter  Google Scholar 

  9. Oivo, M., Kuvaja, P., Pulli, P., Similä, J.: Software engineering research strategy: Combining experimental and explorative research (eer). In: Bomarius, F., Iida, H. (eds.) PROFES 2004. LNCS, vol. 3009, pp. 302–317. Springer, Heidelberg (2004)

    Chapter  Google Scholar 

  10. Basili, V.: The role of experimentation in software engineering: Past, current, and future. In: 18th International Conference on Software Engineering, pp. 442–449. IEEE, Berlin (1996)

    Google Scholar 

  11. Basili, V.: Measurement and model building, introduction. In: Basili, V.R., Rombach, H.D., Schneider, K., Kitchenham, B., Pfahl, D., Selby, R.W. (eds.) Empirical Software Engineering Issues. LNCS, vol. 4336, pp. 68–69. Springer, Heidelberg (2007)

    Chapter  Google Scholar 

  12. Vegas, S., Basili, V.: Measurement and model building, discussion and summary. In: Basili, V.R., Rombach, H.D., Schneider, K., Kitchenham, B., Pfahl, D., Selby, R.W. (eds.) Empirical Software Engineering Issues. LNCS, vol. 4336, pp. 115–120. Springer, Heidelberg (2007)

    Chapter  Google Scholar 

  13. Curtis, B.: Measurement and experimentation in software engineering. In: Proceedings of IEEE, pp. 1144–1157. IEEE, Los Alamitos (1980)

    Google Scholar 

  14. Miller, J.: Statistical significance testing – a panacea for software technology experiments? Journal of Systems and Software 2(73), 183–192 (2004)

    Article  Google Scholar 

  15. Zelkowitz, M., Wallance, D.: Experimental validation in software engineering. Information and Software Technology 11(39), 735–743 (1997)

    Article  Google Scholar 

  16. Basili, V., Selby, R., Hutchens, D.: Experimentation in software engineering. IEEE Transactions on Software Engineering 12(7), 733–743 (1986)

    Article  Google Scholar 

  17. Endres, A., Rombach, D.: A Handbook of Software and Systems Engineering: Empirical Observations, Laws and Theories. Pearson Education, Harlow (2003)

    Google Scholar 

  18. Kitchenham, B., Pfleeger, S., Pickard, L., Jones, P., Hoaglin, D., El Emam, K., et al.: Preliminary guidelines for empirical research in software engineering. IEEE Transactions on Software Engineering 8(28), 721–734 (2002)

    Article  Google Scholar 

  19. Cook, T., Campbell, D.: Quasi-Experimentation: Design and Analysis Issues for Field Settings. Houghton Mifflin Company, USA (1979)

    Google Scholar 

  20. Miller, J., Daly, J., Wood, M., Roper, M., Brooks, A.: Statistical power and its subcomponents - missing and misunderstood concepts in empirical software engineering research. Information and Software Technology 4(39), 285–295 (1997)

    Article  Google Scholar 

  21. Dybå, T., Kampenes, V., Sjøberg, D.: A systematic review of statistical power in software engineering experiments. Information and Software Technology 8(48), 745–755 (2006)

    Article  Google Scholar 

  22. Miller, J.: Replicating software engineering experiments: a poisoned chalice or the holy grail. Information and Software Technology 4(47), 233–244 (2005)

    Article  Google Scholar 

  23. Wood, M., Daly, J., Miller, J., Roper, M.: Multi-method research: An empirical investigation of object-oriented technology. Journal of Systems and Software 1(48), 13–26 (1999)

    Article  Google Scholar 

  24. Popper, K.: The Logic of Scientific Discovery. Routledge Classics, New York (1959)

    MATH  Google Scholar 

  25. Ramesh, V., Glass, R., Vessey, I.: Research in computer science: an empirical study. Journal of systems and Software 2(70), 165–176 (2004)

    Article  Google Scholar 

  26. Sjøberg, D., Hannay, J., Hansen, O., By Kampenes, V., Karahasanovic, A., Liborg, N.K., et al.: A survey of controlled experiments in software engineering. IEEE Transactions on Software Engineering 31(9), 733–753 (2005)

    Article  Google Scholar 

  27. Brooks, A., Roper, M., Wood, M., Daly, J., Miller, J.: Replication’s role in software engineering. In: Shull, F., et al. (eds.) Guide to Advanced Empirical Software Engineering, pp. 365–379. Springer, London (2008)

    Chapter  Google Scholar 

  28. Basili, V., Selby, R.: Paradigms for experimentation and empirical studies in software engineering. Reliability Engineering and System Safety 1(32), 171–191 (1991)

    Article  Google Scholar 

  29. Lott, C., Rombach, D.: Repeatable software engineering experiments for comparing defect-detection techniques. Empirical Software Engineering 1(3), 241–277 (1996)

    Article  Google Scholar 

  30. Jedlitschka, A., Ciolkowski, M.: Reporting experiments in software engineering. In: Shull, F., et al. (eds.) Guide to Advanced Empirical Software Engineering, pp. 201–228. Springer, London (2007)

    Google Scholar 

  31. Easterbrook, S., Singer, J., Storey, M.A., Damian, D.: Selecting empirical methods for software engineering research. In: Shull, F., et al. (eds.) Guide to Advanced Empirical Software Engineering, pp. 285–311. Springer, London (2008)

    Chapter  Google Scholar 

  32. Creswell, J.: Research Design: Qualitative, Quantitative, and Mixed Method Approaches. Sage Publications, Inc., London (2008)

    Google Scholar 

  33. Petter, S., Gallivan, M.: Toward a framework for classifying and guiding mixed method research in information systems. In: The 37th Hawaii International Conference on System Sciences, Big Island, HI, USA, pp. 1–10 (2004)

    Google Scholar 

  34. Nunamaker, J., Chen, M., Purdin, T.: Systems development in information systems research. Journal of Management Information Systems 7(3), 89–106 (1991)

    Article  Google Scholar 

  35. Galliers, R.: Research issues in information systems. Journal of Information Technology 2(8), 92–98 (1993)

    Article  Google Scholar 

  36. Sawyer, S.: Studying organizational computing infrastructures: Multi-method approaches. In: Baskerville, R., et al. (eds.) Organizational and Social Perspectives on Information Technology, IFIP TC8 WG8.2 International Working Conference on the Social and Organizational Perspective on Research and Practice in Information Technology, pp. 213–232. Kluwer, Aalborg (2000)

    Google Scholar 

  37. Mingers, J.: Combining is research methods: Towards a pluralist methodology. Information Systems Research 12(3), 240–259 (2001)

    Article  Google Scholar 

  38. Steinfield, C., Huysman, M., David, K., Yang Jang, C., Poot, J., Huis in ’t Veld, M., et al.: New methods for studying global virtual teams: Towards a multi-faceted approach, Wailea Maui, Hawaii, USA. In: The 34th Hawaii International Conference on System Sciences 2001, pp. 1–10 (2001)

    Google Scholar 

  39. Howe, K.: Against the quantitative-qualitative incompatibility thesis. Educational Researcher 17(8), 10–16 (1998)

    Article  MathSciNet  Google Scholar 

  40. Bratthall, L., Jørgensen, M.: Can you trust a single data source exploratory software engineering case study? Empirical Software Engineering 7(1), 9–26 (2002)

    Article  MATH  Google Scholar 

  41. Daly, J., El Emam, K., Miller, J.: An empirical research methodology for software process improvement. In: El Emam, K., et al. (eds.) Elements of Software Process Assessment and Improvement. Wiley-IEEE Computer Society Press, London (1998)

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of Information Processing Science, University of Oulu, Rakentajantie 3, 90014, Finland

    Vladimir Mandić, Jouni Markkula & Markku Oivo

Authors
  1. Vladimir Mandić
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Jouni Markkula
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Markku Oivo
    View author publications

    You can also search for this author in PubMed Google Scholar

Editor information

Editors and Affiliations

  1. Fraunhofer IIESE, Fraunhoferplatz 1, 67663, Kaiserslautern, Germany

    Frank Bomarius

  2. Department of Information Processing Science, University of Oulu, P.O.Box 3000, 90014, Finland

    Markku Oivo

  3. VTT Technical Research Centre of Finland, Tietotie 3, 02150, Espoo, Finland

    Päivi Jaring

  4. Department of Computer Science, University of Helsinki, P.O.Box 68, 00014, Helsinki, Finland

    Pekka Abrahamsson

Rights and permissions

Reprints and permissions

Copyright information

© 2009 Springer-Verlag Berlin Heidelberg

About this paper

Cite this paper

Mandić, V., Markkula, J., Oivo, M. (2009). Towards Multi-Method Research Approach in Empirical Software Engineering. In: Bomarius, F., Oivo, M., Jaring, P., Abrahamsson, P. (eds) Product-Focused Software Process Improvement. PROFES 2009. Lecture Notes in Business Information Processing, vol 32. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-02152-7_9

Download citation

  • DOI: https://doi.org/10.1007/978-3-642-02152-7_9

  • Publisher Name: Springer, Berlin, Heidelberg

  • Print ISBN: 978-3-642-02151-0

  • Online ISBN: 978-3-642-02152-7

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation