Skip to main content
SpringerLink
Log in

Challenges and Recommendations When Increasing the Realism of Controlled Software Engineering Experiments

  • Chapter
Empirical Methods and Studies in Software Engineering

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 2765))

Abstract

An important goal of most empirical software engineering experiments is the transfer of the research results to industrial applications. To convince industry about the validity and applicability of the results of controlled software engineering experiments, the tasks, subjects and the environments should be as realistic as practically possible. Such experiments are, however, more demanding and expensive than experiments involving students, small tasks and pen-and-paper environments. This chapter describes challenges of increasing the realism of controlled experiments and lessons learned from the experiments that have been conducted at Simula Research Laboratory.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 39.99
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. Basili, V.R., Selby, R.W., Hutchens, D.H.: Experimentation in Software Engineering. IEEE Transactions on Software Engineering SE-12(7), 733–743 (1986)

    Google Scholar 

  2. Rombach, H.D., Basili, V.R., Selby, R.W.: Experimental Software Engineering Issues: Critical Assessment and Future Directions. In: Rombach, H.D., Selby, R.W., Basili, V.R. (eds.) Experimental Software Engineering Issues: LNCS, vol. 706, Springer, Heidelberg (1993)

    Google Scholar 

  3. Basili, V.R.: The Role of Experimentation in Software Engineering: Past, Current, and Future. In: Proceedings of the 18th International Conference on Software Engineering, Berlin, Germany, March 25-29, pp. 442–449 (1996)

    Google Scholar 

  4. Tichy, W.F.: Should Computer Scientists Experiment More? 16 Reasons to Avoid Experimentation. IEEE Computer 31(5), 32–40 (1998)

    MathSciNet  Google Scholar 

  5. Zelkowitz, M.V., Wallace, D.R.: Experimental Models for Validating Technology. IEEE Computer 31(5), 23–31 (1998)

    Google Scholar 

  6. Wohlin, C., Runeson, P., Höst, M., Ohlsson, M.C., Regnell, B., Wesslén, A.: Experimentation in Software Engineering – An Introduction. Kluwer Academic Publishers, Boston (1999)

    Google Scholar 

  7. Juristo, N., Moreno, A.: Basics of Software Engineering Experimentation. Kluwer Academic Publishers, Boston (2001)

    MATH  Google Scholar 

  8. Shadish, W.R., Cook, T.D., Campbell, D.T.: Experimental and Quasi-Experimental Designs for Generalized Causal Inference. Houghton Mifflin Company, Boston (2002)

    Google Scholar 

  9. Montgomery, D.C.: Design and Analysis of Experiments, 4th edn. John Wiley & Sons, New York (1997)

    MATH  Google Scholar 

  10. Harrison, W.: N=1 An Alternative for Software Engineering Research? Beg, Borrow, or Steal: Using Multidisciplinary Approaches in Empirical Software Engineering Research, Workshop, 5 June, 2000 at 22nd Int. Conf. on Softw. Eng. (ICSE), Limerick, Ireland, pp. 39–44 (2000)

    Google Scholar 

  11. Potts, C.: Software-Engineering Research Revisited. IEEE Software 10(5), 19–28 (1993)

    Article  Google Scholar 

  12. Glass, R.L.: The Software-Research Crisis. IEEE Software 11(6), 42–47 (1994)

    Article  Google Scholar 

  13. Sjøberg, D.I.K., Anda, B., Arisholm, E., Dybå, T., Jørgensen, M., Karahasanovic, A., Koren, E.F., Vokac, M.: Conducting Realistic Experiments in Software Engineering, ISESE 2002 (First International Symposium on Empirical Softwarfe Engineering), Nara, Japan, October 3–4, pp. 17–26. IEEE Computer Society, Los Alamitos (2002)

    Google Scholar 

  14. Kitchenham, B.A., Pfleeger, S.L., Pickard, L.M., Jones, P.W., Hoaglin, D.C., El-Emam, K., Rosenberg, J.: Preliminary Guidelines for Empirical Research in Software Engineering. IEEE Transactions on Software Engineering 28(8), 721–734 (2002)

    Article  Google Scholar 

  15. Rogers, E.M.: Diffusion of Innovations, 4th edn. The Free Press, New York (1995)

    Google Scholar 

  16. Arisholm, E., Sjøberg, D.I.K.: A Controlled Experiment in Industry to Evaluate the Effect of Responsibility-Driven Design Principles on Software Changeability for Different Categories of Professional (2003) (in preparation)

    Google Scholar 

  17. Tichy, W.F.: Hints for Reviewing Empirical Work in Software Engineering. Empirical Software Engineering 5(4), 309–312 (2000)

    Article  MathSciNet  Google Scholar 

  18. Basili, V.R., Shull, F., Lanubile, F.: Building Knowledge through Families of Experiments. IEEE Transactions on Software Engineering 25(4), 456–473 (1999)

    Article  Google Scholar 

  19. Arisholm, E., Sjøberg, D.I.K.: A Controlled Experiment in Industry to Evaluate the Effect of Responsibility-Driven Design Principles on Software Changeability for Different Categories of Professionals (2003) (in preparation)

    Google Scholar 

  20. Anda, B., Sjøberg, D.I.K., Jørgensen, M.: Quality and Understandability in Use Case Models. In: Knudsen, J.L. (ed.) ECOOP 2001. LNCS, vol. 2072, pp. 402–428. Springer, Heidelberg (2001)

    Chapter  Google Scholar 

  21. Jørgensen, M.: A Review of Studies on Expert Estimation of Software Development Effort. To appear in Journal of Systems and Software (2004)

    Google Scholar 

  22. Jørgensen, M., Sjøberg: Impact of Software Effort Estimation on Software Work. Journal of Information and Software Technology 43, 939–948 (2001)

    Article  Google Scholar 

  23. Anda, B., Sjøberg, D.I.K.: Towards an Inspection Technique for Use Case Models. In: SEKE 2002 (Fourteenth International Conference on Software Engineering and Knowledge Engineering), Ischia, Italy, July 15-19, pp. 127–134 (2002)

    Google Scholar 

  24. Arisholm, E., Sjøberg, D.I.K., Jørgensen, M.: Assessing the Changeability of two Object-Oriented Design Alternatives – a Controlled Experiment. Empirical Software Engineering 6(3), 231–277 (2001)

    Article  MATH  Google Scholar 

  25. Karahasanovic, A., Sjøberg, D.: Visualizing Impacts of Database Schema Changes – A Controlled Experiment. In: 2001 IEEE Symposium on Visual/Multimedia Approaches to Programming and Software Engineering, Stresa, Italy, September 5-7, pp. 358–365. IEEE Computer Society, Los Alamitos (2001)

    Google Scholar 

  26. Sieber, J.E.: Protecting Research Subjects, Employees and Researchers: Implications for Software Engineering. Empirical Software Engineering 6(4), 329–341 (2001)

    Article  MATH  Google Scholar 

  27. Davis, M.: When is a Volunteer Not a Volunteer? Empirical Software Engineering 6(4), 349–352 (2001)

    Article  Google Scholar 

  28. Fenton, N.: Conducting and Presenting Empirical Software Engineering. Empirical Software Engineering 6(3), 195–200 (2001)

    Article  Google Scholar 

  29. Anda, B.: Comparing Effort Estimates Based on Use Case Points with Expert Estimates. In: Empirical Assessment in Software Engineering (EASE 2002), Keele, UK, April 8-10 (2002)

    Google Scholar 

  30. Jørgensen, M., Sjøberg, D.I.K.: The Impact of Customer Expectation on Software Development Effort Estimates (2003) (submitted for publication)

    Google Scholar 

  31. Vokác, M., Tichy, W., Sjøberg, D.I.K., Arisholm, E., Aldrin, M.: A Controlled Experiment Comparing the Maintainability of Programs Designed with and without Design Patterns – a Replication in a real Programming Environment (2003) (submitted for Publication)

    Google Scholar 

  32. Deligiannis, I.S., Shepperd, M., Webster, S., Roumeliotis, M.: A Review of Experimental Investigations into Object-Oriented Technology. Empirical Software Engineering 7(3), 193–232 (2002)

    Article  MATH  Google Scholar 

  33. Jørgensen, M., Bygdås, S.: An Empirical Study of the Correlation between Development Efficiency and Software Development Tools. Technical Journal of Norwegian Telecom 11, 54–62 (1999)

    Google Scholar 

  34. Jørgensen, M.: An Empirical Study of Software Maintenance Tasks. Journal of Software Maintenance 7, 27–48 (1995)

    Article  Google Scholar 

  35. Arisholm, E., Sjøberg, D.I.K.: Towards a Framework for Empirical Assessment of Changeability Decay. Journal of Systems and Software 53, 3–14 (2000)

    Article  Google Scholar 

  36. Karahasanovic, A., Sjøberg, D.I.K.: Visualising Impacts of Change in Evolving Object-Oriented Systems: An Explorative Study. In: Proceedings of the International Workshop on Graph-Based Tools (GraBaTs 2002), Barcelona, Spain, pp. 22–31 (2002)

    Google Scholar 

  37. Anda, B., Sjøberg, D.I.K.: Applying Use Cases to Design versus Validate Class Diagrams – A Controlled Experiment Using a Professional Modelling Tool. Simula Research Laboratory Technical Report, No. 2003–01 (2003)

    Google Scholar 

  38. Prechelt, L., Unger, B., Tichy, W.F., Brössler, P., Votta, L.G.: A Controlled Experiment in Maintenance Comparing Design Patterns to Simpler Solutions. IEEE Transactions on Software Engineering 27(12), 1134–1144 (2001)

    Article  Google Scholar 

  39. Syversen, E., Anda, B., Sjøberg, D.I.K.: An Evaluation of Applying Use Cases to Construct Design versus Validate Design. In: Hawaii International Conference on System Sciences (HICSS-36), Big Island, Hawaii, January 6-9 (2003)

    Google Scholar 

  40. Briand, L.C., Bunse, C., Daly, J.W.: A Controlled Experiment for Evaluating Quality Guidelines on the Maintainability of Object-Oriented Designs. IEEE Transactions on Software Engineering 27(6), 513–530 (2001)

    Article  Google Scholar 

  41. Karahasanovic, A., Sjøberg, D.I.K., Jørgensen, M.: Data Collection in Software Engineering Experiments. In: Information Resources Management Association Conference, Soft. Eng. Track. 2001, pp. 1027–1028. Idea Group Publishing, Toronto (2001)

    Google Scholar 

  42. Arisholm, E., Sjøberg, D.I.K., Carelius, G.J., Lindsjørn, Y.: A Web-based Support Environment for Software Engineering Experiments. Nordic Journal of Computing 9(4), 231–247 (2002)

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Simula Research Laboratory, P.O. Box 134, NO-1325, Lysaker, Norway

    Dag I. K. Sjøberg, Bente Anda, Erik Arisholm, Tore Dybå, Magne Jørgensen, Amela Karahasanović & Marek Vokáč

Authors
  1. Dag I. K. Sjøberg
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Bente Anda
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Erik Arisholm
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Tore Dybå
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Magne Jørgensen
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Amela Karahasanović
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Marek Vokáč
    View author publications

    You can also search for this author in PubMed Google Scholar

Editor information

Editors and Affiliations

  1. Department of Computer and Information Science (IDI), Norwegian University of Science and Technology (NTNU), Trondheim, Norway

    Reidar Conradi

  2. Department of Computer and Information Science, Norwegian University of Science and Technology, 7491, Trondheim, Norway

    Alf Inge Wang

Rights and permissions

Reprints and permissions

Copyright information

© 2003 Springer-Verlag Berlin Heidelberg

About this chapter

Cite this chapter

Sjøberg, D.I.K. et al. (2003). Challenges and Recommendations When Increasing the Realism of Controlled Software Engineering Experiments. In: Conradi, R., Wang, A.I. (eds) Empirical Methods and Studies in Software Engineering. Lecture Notes in Computer Science, vol 2765. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-540-45143-3_3

Download citation

  • DOI: https://doi.org/10.1007/978-3-540-45143-3_3

  • Publisher Name: Springer, Berlin, Heidelberg

  • Print ISBN: 978-3-540-40672-3

  • Online ISBN: 978-3-540-45143-3

  • eBook Packages: Springer Book Archive

Publish with us

Policies and ethics

Search

Navigation