Skip to main content
SpringerLink
Log in

Empirical Practice in Software Engineering

  • Chapter
  • First Online:
Perspectives on the Future of Software Engineering

Abstract

Experimental software engineering has been defined as the scientific approach to systematically evaluating software technologies by referring to predefined hypotheses using sound empirical methods.

The purpose of this chapter is to give an overview of the history, current practice, and future of empirical practice in Software Engineering. In particular, based on what we have learned from 20 years of research in empirical software engineering, we describe the empirical approach we are currently using in terms of a scientific approach to applied research and as a means for systematic evaluation.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 84.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 109.00
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info
Hardcover Book
USD 109.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    In this chapter, we use the word study for all kinds of empirical studies. We use experiment as the generic term for controlled and quasi-experiments.

  2. 2.

    Synthesis is the umbrella term covering different strategies for combining empirical evidence [34].

  3. 3.

    We use the term technology to refer to technique, method, and tool, following Basili et al. [44].

References

  1. Basili, V., Selby, R., Hutchens, D.: Experimentation in software engineering. IEEE Trans. Softw. Eng. 12(7), 733–743 (1986)

    Article  Google Scholar 

  2. Wohlin, C., Runeson, P., Höst, M., Ohlsson, M.C., Regnell, B., Wesslén, A.: Experimentation in Software Engineering: An Introduction. Kluwer (2000). ISBN: 0-7923-8682-5

    Google Scholar 

  3. IEEE Computer Society: IEEE: standard glossary of software engineering terminology. IEEE Standard 610.12-1990

    Google Scholar 

  4. Juristo, N., Moreno, A.: Basics of Software Engineering Experimentation. Kluwer, Boston (2001)

    Book  MATH  Google Scholar 

  5. Schulz, W.: Kausalität und Experiment in den Sozialwissenschaften – Methodologie und Forschungstechnik. V. Hase & Koehler Verlag, Mainz (1970)

    Google Scholar 

  6. Rombach, H.D., Basili, V.R., Selby, R.W. (eds.): Experimental Software Engineering Issues: Critical Assessment and Future Directions. International Workshop Dagstuhl Castle, Germany. Lecture Notes in Computer Science, vol. 706. Springer (1992)

    Google Scholar 

  7. http://www.iese.fraunhofer.de. Last visited 14 Feb 2013

  8. Rombach, D.: Fraunhofer: the German model for applied research and technology transfer. In: Proceedings of International Conference on Software Engineering 2000, Limerick, pp. 531–537. (2000)

    Google Scholar 

  9. http://isern.iese.de. Last visited 14 Feb 2013

  10. Fenton, N.E., Pfleeger, S.L., Glass, R.: Science and substance: a challenge to software engineers. IEEE Softw. 11(4), 86–95 (1994)

    Article  Google Scholar 

  11. Pfleeger, S.L.: Design and analysis in software engineering. Part 1: the language of case studies and formal experiments. ACM SIGSOFT Softw. Eng. Notes 19(4), 16–20 (1994)

    Article  Google Scholar 

  12. Kitchenham, B., Linkman, S., Law, D.: DESMET: a methodology for evaluating software engineering methods and tools. Comput. Contr. Eng. J. 8(3), 120–126 (1997)

    Article  Google Scholar 

  13. Singer, J.: (APA) Style guidelines to report experimental results. In: Proceedings of Workshop on Empirical Studies in Software Maintenance, pp. 71–77. Oxford, UK (1999)

    Google Scholar 

  14. American Psychological Association: Publication Manual of the American Psychological Association, 5th edn. American Psychological Association, Washington, DC (2001)

    Google Scholar 

  15. Pfleeger, S.L., Kitchenham, B.A.: Principles of survey research. Part 1: turning lemons into lemonade. ACM SIGSOFT Softw. Eng. Notes 26(6), 16–18 (2001)

    Article  Google Scholar 

  16. Kitchenham, B.A., Pfleeger, S.L., Pickard, L.M., Jones, P.W., Hoaglin, D.C., El Emam, K., Rosenberg, J.: Preliminary guidelines for empirical research in software engineering. IEEE Trans. Softw. Eng. 28(8), 721–734 (2002)

    Article  Google Scholar 

  17. Shaw, M.: Writing good software engineering research papers. In: Proceedings of the 25th International Conference on Software Engineering (ICSE’03), pp. 726–736. IEEE CS, Portland (2003)

    Google Scholar 

  18. Ruhe, G.: Software engineering decision support – a new paradigm for learning software organizations. In: Henninger, S., Maurer, F. (eds.) Advances in LSO. 4th International Workshop, LSO 2002, Chicago, 6 Aug 2002: Revised papers, Lecture Notes in Computer Science, vol. 2640, pp. 104–115. Springer (2003)

    Google Scholar 

  19. Aurum, A., Petersson, H., Wohlin, C.: State-of-the-art: software inspections after 25 years. Softw. Test. Verif. Reliab. 12(3), 133–154 (2002)

    Article  Google Scholar 

  20. Juristo, N., Moreno, A., Vegas, S.: Reviewing 25 years of testing technique experiments. J. Empirical Softw. Eng. 9(1–2), 7–44 (2004)

    Article  Google Scholar 

  21. Jedlitschka, A., Ciolkowski, M.: Towards evidence in software engineering. In: Proceedings of International Symposium on Empirical SE 2004 (ISESE2004), pp. 261–270. Redondo Beach (2004)

    Google Scholar 

  22. Wohlin, C., Petersson, H., Aurum, A.: Combining data from reading experiments in software inspections. In: Juristo, N., Moreno, A. (eds.) Lecture Notes on Empirical Software Engineering, pp. 85–132. World Scientific Publishing, River Edge (2003)

    Google Scholar 

  23. Conradi, R., Wang, A.I. (eds.): Empirical Methods and Studies in Software Engineering – Experiences from ESERNET. Lecture Notes in Computer Science, vol. 2765. Springer, Berlin (2003)

    MATH  Google Scholar 

  24. Conradi, R., Dybå, T., Sjøberg, D., Ulsund, T.: Lessons learned and recommendations from two large Norwegian SPI programmes. In: Oquendo, F. (ed.) 9th European Workshop on Software Process Technology (EWSPT 2003), Helsinki, 1–2 Sept 2003, Lecture Notes in Computer Science, vol. 2786, pp. 32–45. Springer (2003)

    Google Scholar 

  25. Glass, R.L.: Matching methodology to problem domain. Column Pract. Programmer Commun. ACM 47(5), 19–21 (2004)

    Article  Google Scholar 

  26. Turner, R.: Why we need empirical information on best practices. CROSSTALK – J. Defense Softw. Eng. 17(4), 9–11 (2004)

    Google Scholar 

  27. Kitchenham, B.A., Dybå, T., Jørgensen, M.: Evidence-based software engineering. In: Proceedings of 26th International Conference on Software Engineering (ICSE’04), pp. 273–281. Edinburgh (2004)

    Google Scholar 

  28. Kitchenham, B.A.: Procedures for performing systematic reviews. Keele University Technical Report TR/SE-0401; ISSN 1353-7776.1 (2004)

    Google Scholar 

  29. Sjøberg, D., Hannay, J., Hansen, O., By Kampenes, V., Karahasanovic, A., Liborg, N.-K., Rekdal, A.: A survey of controlled experiments in software engineering. Trans. Softw. Eng. 31(9), 733–753 (2005)

    Article  Google Scholar 

  30. Jedlitschka, A., Pfahl, D.: Reporting guidelines for controlled experiments in software engineering. In: Proceedings of International Symposium on Empirical SE 2005 (ISESE2005), Noosa Heads, Australia, Nov 2005, pp. 95–104. IEEE CS (2005)

    Google Scholar 

  31. Basili, V.R., Rombach, D., Schneider, K., Kitchenham, B., Pfahl, D., Selby, R.W. (eds.): Empirical Software Engineering Issues: Critical Assessment and Future Directions. International Workshop Dagstuhl Castle, Germany. Lecture Notes in Computer Science, vol. 4336. Springer (2007)

    Google Scholar 

  32. Dieste, O., Juristo, N.: Systematic review and aggregation of empirical studies on elicitation techniques. IEEE Trans. Softw. Eng. 37(2), 283–304 (2011)

    Article  Google Scholar 

  33. Dybå, T., Dingsøyr, T.: Empirical studies of agile software development: a systematic review. Inf. Softw. Technol. 50(9–10), 833–859 (2008)

    Article  Google Scholar 

  34. Chalmers, I., Hedges, L., Cooper, H.: A brief history of research synthesis. Eval. Health Prof. 25(1), 12–37 (2002)

    Article  Google Scholar 

  35. Ciolkowski, M.: What do we know about perspective-based reading? An approach for quantitative aggregation in software engineering. In: Proceedings of ESEM 2009, Lake Buena Vista, pp. 133–144

    Google Scholar 

  36. Jedlitschka, A., Ciolkowski, M., Pfahl, D.: Reporting controlled experiments in software engineering. In: Shull, F., et al. (eds.) Guide to Advanced Empirical Software Engineering. Springer, New York (2008)

    Google Scholar 

  37. Runeson, P., Höst, M.: Guidelines for conducting and reporting case study research in SE. J. Empirical Softw. Eng. 14(2), 131–164 (2009)

    Article  Google Scholar 

  38. Carver, J.: Towards reporting guidelines for experimental replications: a proposal. In: Proceedings of the 1st International Workshop on Replication in ESE Research (RESER) @ ICSE, Cape Town, 4 May 2010

    Google Scholar 

  39. Ciolkowski, M.: An approach for quantitative aggregation of evidence from controlled experiments in software engineering. Kaiserslautern University, Dissertation, 2011, 231 pp, Fraunhofer Verlag, Stuttgart (2012)

    Google Scholar 

  40. Cruzes, D.S., Dybå, T.: Recommended steps for thematic synthesis in software engineering. In: Proceedings of ESEM 2011, Banff, pp. 275–284

    Google Scholar 

  41. Rombach, D.: Empirical software engineering models: can they become the equivalent of physical laws in traditional engineering? Int. J. Softw. Inf. 5(3), 525–534 (2011)

    Google Scholar 

  42. Linkman, S., Rombach, D.: Experimentation as a vehicle for software technology transfer – a family of software reading techniques. Inf. Softw. Technol. 39(11), 777–780 (1997)

    Article  Google Scholar 

  43. Basili, V.R., Caldiera, G., Rombach, H.D.: Goal question metric paradigm. In: Marciniak, J.J. (ed.) Encyclopedia of Software Engineering, vol. 1, 2nd edn, pp. 528–532. Wiley-Interscience, New York (2001a). doi:10.1002/0471028959. ISBN 10: 0471377376

    Google Scholar 

  44. Basili, V.R., Caldiera, G., Rombach, H.D.: Experience factory. In: Marciniak, J.J. (ed.) Encyclopedia of Software Engineering, vol. 1, pp. 511–519. Wiley-Interscience, New York (2001b). doi:10.1002/0471028959. ISBN 10: 0471377376

    Google Scholar 

  45. Creswell, J.W.: Research Design: Qualitative, Quantitative, and Mixed Methods Approaches, 2nd edn. Sage, Thousand Oaks (2003)

    Google Scholar 

  46. Shadish, W.R., Cook, T.D., Campbell, D.T.: Experimental and Quasi-experimental Design for Generalized Causal Inference. Houghton-Mifflin, Boston (2002)

    Google Scholar 

  47. Yin, R.K.: Case Study Research. Design and Methods, 3rd edn. Sage, Thousand Oaks (2003)

    Google Scholar 

  48. Chen, H.T., Rossi, P.H.: Evaluating with sense: the theory-driven approach. Eval. Rev. 7(3), 283–302 (1983). doi:10.1177/0193841X8300700301

    Article  Google Scholar 

  49. Ciolkowski, M., Laitenberger, O., Biffl, S.: Software reviews: the state of the practice. IEEE Softw. 20(6), 46–51 (2003)

    Article  Google Scholar 

  50. Jedlitschka, A., Ciolkowski, M., Denger, C., Freimut, B., Schlichting, A.: Relevant information sources for successful technology transfer: a survey using inspections as an example. In: Proceedings of the International Symposium on Empirical SE and Measurement 2007 (ESEM2007), pp. 31–40. Madrid (2007)

    Google Scholar 

  51. Jedlitschka, A., Hamann, D., Göhlert, T., Schröder, A.: Adapting PROFES for use in an agile process: an industry experience report. In: Bomarius, F., et al. (ed.) Proceedings of 6th International Conference on Product Focused Software Process Improvement. Profes’2005. Lecture Notes in Computer Science, vol. 3547, pp. 502–516. (2005)

    Google Scholar 

  52. Kläs, M., Nakao, H., Elberzhager, E., Münch, J.: Support planning and controlling of early quality assurance by combining expert judgment and defect data – a case study. J. Empir. Softw. Eng. 15(4), 423–454, Springer (2010)

    Google Scholar 

  53. Kerth, N.: Project Retrospectives: A Handbook for Team Reviews. Dorset House Publishing, New York (2001)

    Google Scholar 

  54. Sjøberg, D.I.K., Dybå, T., Anda, B.C.D., Hannay, J.E.: Building theories in software engineering. In: Shull, F. et al. (eds.) Guide to advanced empirical software engineering. Springer (2008)

    Google Scholar 

  55. Basili, V., Heidrich, J., Lindvall, M., Münch, J., Regardie, M., Rombach, D., Seaman, C., Trendowicz, A.: Linking software development and business strategy through measurement. IEEE Comput. 43(4), 57–65 (2010)

    Article  Google Scholar 

  56. Easterbrook, S.M., Singer, J., Storey, M., Damian, D.: Selecting empirical methods for software engineering research. In: Shull, F., et al. (eds.) Guide to Advanced Empirical Software Engineering. Springer, New York (2008)

    Google Scholar 

  57. Kleinberger, T., Jedlitschka, A., Storf, H., Steinbach-Nordmann, S., Prueckner, S.: An approach to and evaluations of assisted living systems using ambient intelligence for emergency monitoring and prevention. In: Universal Access in HCI. Intelligent and Ubiquitous Interaction Environments. Lecture Notes in Computer Science, vol. 5615, pp. 199–208. (2009)

    Google Scholar 

  58. Venkatesh, V., Bala, H.: Technology acceptance model 3 and a research agenda on interventions. Decis. Sci. 39(2), 273–315 (2008)

    Article  Google Scholar 

  59. Venkatesh, V., Morris, M.G., Davis, G.B., Davis, F.D.: User acceptance of information technology. MIS Q. 27(3), 425–478 (2003)

    Google Scholar 

  60. Nunnenmacher, S., Jung, J., Chehrazi, G., Klaus, A., Lampasona, C., Webel, C., Ciolkowski, M.: A preliminary survey on subjective measurements and personal insights into factors of perceived future project success. In: Proceedings of 5th International Symposium on Empirical SE and Measurement, pp. 396–399, IEEE CS, Los Alamitos (2011)

    Google Scholar 

  61. Harris, P.: Designing and Reporting Experiments in Psychology, 2nd edn. Open University Press, Berkshire (2002)

    Google Scholar 

  62. Wagner, S., Lochmann, K., Heinemann, L., Kläs, M., Trendowicz, A., Plösch, R., Seidl, A., Goeb, A., Streit, J.: The Quamoco product quality modeling and assessment approach. In: Proceedings of 34th International Conference on Software Engineering (ICSE 2012), pp. 1133–1142. ACM/IEEE, Zurich, 2–9 June 2012

    Google Scholar 

  63. Jedlitschka, A.: An empirical model of software managers information needs for software engineering technology selection. Kaiserslautern University Dissertation, 435 pp. Fraunhofer Verlag, Stuttgart (2009)

    Google Scholar 

  64. Trendowicz, A.: Software Cost Estimation, Benchmarking, and Risk Assessment. The Software Decision-Makers’ Guide to Predictable Software Development. The Fraunhofer IESE Series on Software and Systems Engineering. Springer, Berlin (2013)

    Book  Google Scholar 

  65. Lampasona, C., Rostanin, O., Maus, H.: Seamless integration of order processing in MS outlook using SmartOffice: an empirical evaluation. In: Proceedings of International Symposium on Empirical SE and Measurement, pp. 165–168. ACM Press, New York (2012)

    Google Scholar 

  66. Jedlitschka, A.: Evaluating a model of software managers’ information needs: an experiment. In: Proceedings of ACM-IEEE International Symposium on Empirical SE and Measurement (ESEM’10). ACM, Bozen, No. 19, 10 pp (2010)

    Google Scholar 

  67. Jung, J., Höfig, K., Hiller, M., Jedlitschka, A., Domis, D.: Are Ph.D.-Students with Domain Knowledge Appropriate Subjects for Experiments? Kaiserslautern, IESE-Report; 037.12/E (2012)

    Google Scholar 

  68. Rombach, D., Achatz, R.: Research collaborations between academia and industry. In: Proceedings of WS on future of Software Engineering, Minneapolis, pp. 29–36. (2007)

    Google Scholar 

  69. Jedlitschka, A., Pfahl, D.: Experience-based model-driven improvement management with combined data sources from industry and academia. In: Proceedings of International Symposium on Empirical SE ISESE 2003, pp. 154–161. Roman Castles (2003)

    Google Scholar 

  70. Tichy, W.: Empirical software research: an interview with Dag Sjøberg, University of Oslo, Norway. Ubiquity 2011, June, Article 2, 14 pp (2011)

    Google Scholar 

  71. http://www.catalysts.cc/contest/. Last visited 14 Feb 2013

  72. Lincoln, Y., Guba, E.G.: Naturalistic Inquiry. Sage, London/Thousand Oaks/New Delhi (1985)

    Google Scholar 

  73. Pope, C., Mays, N., Popay, J.: Synthesizing Qualitative and Quantitative Health Evidence. A Guide to Methods. Open University Press, Berkshire (2007)

    Google Scholar 

  74. Dixon-Woods, M., Agarwal, S., Jones, D., Young, B., Sutton, A.: Synthesising qualitative and quantitative evidence: a review of possible methods. J. Health Serv. Res. Policy 10, 45–53 (2005)

    Article  Google Scholar 

  75. Miles, M.B., Huberman, A.M.: Qualitative Data Analysis: An Expanded Sourcebook. Sage, Thousand Oaks (1994)

    Google Scholar 

Download references

Acknowledgments

First of all, we thank Dieter Rombach, who provided us with the inspiring environment that Fraunhofer IESE is. In addition, we would like to acknowledge the contributions of current and former colleagues as well as friends within the ISERN community to the evolution of empirical SE at Fraunhofer IESE. Since there are so many we are unable to list all of them, but we are particularly grateful to Vic Basili, Marcus Ciolkowski, Natalia Juristo, and Carolyn Seaman, with whom we closely collaborated and walked the path of empirical SE for the past 10 years.

Author information

Authors and Affiliations

  1. Fraunhofer Institut for Experimental Software Engineering (IESE), Fraunhofer-Platz 1, Kaiserslautern, 67663, Germany

    Andreas Jedlitschka, Liliana Guzmán, Jessica Jung, Constanza Lampasona & Silke Steinbach

Authors
  1. Andreas Jedlitschka
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Liliana Guzmán
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Jessica Jung
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Constanza Lampasona
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Silke Steinbach
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Andreas Jedlitschka .

Editor information

Editors and Affiliations

  1. , Dept. of Computer Science, University of Helsinki, Gustav Hällströmin katu 2b, Helsinki, 00014, Finland

    Jürgen Münch

  2. Institut für Informatik, Universität Hildesheim, Samelsonplatz 1, Hildesheim, 31141, Germany

    Klaus Schmid

Rights and permissions

Reprints and permissions

Copyright information

© 2013 Springer-Verlag Berlin Heidelberg

About this chapter

Cite this chapter

Jedlitschka, A., Guzmán, L., Jung, J., Lampasona, C., Steinbach, S. (2013). Empirical Practice in Software Engineering. In: Münch, J., Schmid, K. (eds) Perspectives on the Future of Software Engineering. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-37395-4_15

Download citation

  • DOI: https://doi.org/10.1007/978-3-642-37395-4_15

  • Published:

  • Publisher Name: Springer, Berlin, Heidelberg

  • Print ISBN: 978-3-642-37394-7

  • Online ISBN: 978-3-642-37395-4

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation