Skip to main content
SpringerLink
Log in

Empirical Evaluation of Agile Software Development: The Controlled Case Study Approach

  • Conference paper
Product Focused Software Process Improvement (PROFES 2004)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 3009))

Abstract

Agile software development, despite its novelty, is an important domain of research within software engineering discipline. Agile proponents have put forward a great deal of anecdotal evidence to support the application of agile methods in various application domains and industry sectors. Scientifically grounded empirical evidence is, however, still very limited. Most scientific research to date has been conducted on focused practices performed in university settings. In order to generate impact on both the scientific and practical software engineering community, new approaches are needed for performing empirically validated agile software development studies. To meet these needs, this paper presents a controlled case study approach, which has been applied in a study of extreme programming methodology performed in close-to-industry settings. The approach considers the generation of both quantitative and qualitative data. Quantitative data is grounded on three data points (time, size, and defect) and qualitative data on developers’ research diaries and post-mortem sessions.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 39.99
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. Boehm, B., Turner, R.: Balancing Agility and Discipline: A Guide for the Perplexed. Addison-Wesley, Reading (2003)

    Google Scholar 

  2. Beck, K.: Extreme Programming Explained: Embrace Change. Addison Wesley Longman, Inc., Amsterdam (2000)

    Google Scholar 

  3. Cockburn, A.: Agile Software Development. Addison-Wesley, Boston (2002)

    Google Scholar 

  4. Schuh, P.: Recovery, Redemption, and Extreme Programming. IEEE Software 18, 34–41 (2001)

    Article  Google Scholar 

  5. Rasmusson, J.: Introducing XP into Greenfield Projects: Lessons Learned. IEEE Software, 21–28 (2003)

    Google Scholar 

  6. Murru, O., Deias, R., Mugheddu, G.: Assessing XP at a European Internet Company. IEEE Software, 37–43 (2003)

    Google Scholar 

  7. Lindvall, M., Basili, V.R., Boehm, B., Costa, P., Dangle, K., Shull, F., Tesoriero, R., Williams, L., Zelkowitz, M.V.: Empirical Findings in Agile Methods. Presented at XP/Agile Universe 2002, Chicago, IL, USA (2002)

    Google Scholar 

  8. Zelkowitz, M.V., Wallace, D.R.: Experimental Models for Validating Technology. Computer, 23–31 (1998)

    Google Scholar 

  9. Noll, J., Atkinson, D.C.: Comparing Extreme Programming to Traditional Development for Student Projects: A Case Study. In: Marchesi, M., Succi, G. (eds.) XP 2003. LNCS, vol. 2675, Springer, Heidelberg (2003)

    Chapter  Google Scholar 

  10. Rumpe, B.: Quantitative Survey on Extreme Programming Projects. Presented at XP 2002, Alghero, Sardinia, Italy (2002)

    Google Scholar 

  11. Tessem, B.: Experiences in Learning XP Practices: A Qualitative Study. Presented at XP 2003, Genova, Italy (2003)

    Google Scholar 

  12. Abrahamsson, P.: Extreme Programming: First Results from a Controlled Case Study. Presented at 29th Euromicro Conference, Belek-Antalya, Turkey (2003)

    Google Scholar 

  13. Karlström, D.: Introducing Extreme Programming - An Experience Report. Presented at XP 2002, Alghero, Sardinia, Italy (2002)

    Google Scholar 

  14. Maurer, F., Martel, S.: On the Productivity of Agile Software Practices: An Industrial Case Study (2002)

    Google Scholar 

  15. Müller, M.M., Tichy, W.F.: Case Study: Extreme Programming in a University Environment. Presented at 23rd International Conference on Software Engineering, Toronto (2001)

    Google Scholar 

  16. Wood, W.A., Kleb, W.L.: Exploring XP for Scientific Research. IEEE Software, 30–36 (2003)

    Google Scholar 

  17. Janes, B., Russo, P.: An Empirical Analysis of the Discontinuous Use of Pair Programming. Presented at XP 2003, Genova, Italy (2003)

    Google Scholar 

  18. Heiberg, S., Puus, U., Salumaa, P., Seeba, A.: Pair-Programming Effect on Developers Productivity. Presented at XP 2003, Genova, Italy (2003)

    Google Scholar 

  19. Lui, K.M., Chan, K.C.C.: When Does a Pair Outperform Two Individuals? Presented at XP 2003, Genova, Italy (2003)

    Google Scholar 

  20. Müller, M.M., Hagner, O.: Experiment about Test-first programming. Software 149, 131–135 (2002)

    Google Scholar 

  21. Williams, L., Kessler, R.R., Cunningham, W., Jeffries, R.: Strengthening the Case for Pair Programming. IEEE Software 17, 19–25 (2000)

    Article  Google Scholar 

  22. Rostaher, M., Hericko, M.: Tracking Test First Pair Programming - An Experiment. In: Presented at XP/Agile Universe 2002, Chicago, IL, USA (2002)

    Google Scholar 

  23. Basili, V.R., Selby, R.W., Hutchens, D.H.: Experimentation in Software Engineering. IEEE Transactions on Software Engineering SE-12, 733–742 (1986)

    Google Scholar 

  24. Tichy, W.F.: Should Computer Scientists Experiment More? Computer, 32–40 (1998)

    Google Scholar 

  25. Basili, V.R., Lanuble, F.: Building Knowledge through Families of Experiments. IEEE Transactions on Software Engineering 25, 456–473 (1999)

    Article  Google Scholar 

  26. Fenton, N.: Viewpoint Article: Conducting and Presenting Empirical Software Engineering. Empirical Software Engineering 6, 195–200 (2001)

    Article  Google Scholar 

  27. Wohlin, C., Höst, M., Henningsson, K.: Empirical Research Methods in Software Engineering. In: Conradi, R., Wang, A.I. (eds.) ESERNET 2001. LNCS, vol. 2765, pp. 7–23. Springer, Heidelberg (2003)

    Chapter  Google Scholar 

  28. Birk, A., Dingsøyr, T., Stålhane, T.: Postmortem: Never Leave a Project without It. IEEE Software 19, 43–45 (2002)

    Article  Google Scholar 

  29. Dingsøyr, T., Hanssen, G.K.: Extending Agile Methods: Postmortem Reviews as Extended Feedback. Presented at 4th International Workshop on Learning Software Organizations (LSO 2002), Chicago, Illinois, USA (2002)

    Google Scholar 

  30. Collier, B., DeMarco, T., Fearey, P.: A defined process for project post mortem review. IEEE Software 13 (1996)

    Google Scholar 

  31. Wohlin, C., Runeson, P., Höst, M., Ohlsson, M.C., Regnell, B., Wesslén, A.: Experimentation in Software Engineering. Kluwer Academic Publishers, Boston (2000)

    MATH  Google Scholar 

  32. Bratthall, L., Jørgensen, M.: Can you Trust a Single Data Source Exploratory Software Engineering Case Study? Empirical Software Engineering 7, 9–26 (2002)

    Article  MATH  Google Scholar 

  33. Sjøberg, D.I.K., Anda, B., Arisholm, E., Dybå, T., Jørgensen, M., Karahasanovic, A., Vokác: Challenges and Recommendations When Increasing the Realism of Controlled Software Engineering Experiments. In: Conradi, R., Wang, A.I. (eds.) ESERNET 2001. LNCS, vol. 2765, pp. 24–38. Springer, Heidelberg (2003)

    Chapter  Google Scholar 

  34. Yin, R.K.: Case Study Research Design and Methods, 2nd edn. Sage Publications, Thousand Oaks (1994)

    Google Scholar 

  35. Avison, D., Lau, F., Myers, M., Nielsen, P.A.: Action Research. Communications of the ACM 42, 94–97 (1999)

    Article  Google Scholar 

  36. Cunningham, J.B.: Case study principles for different types of cases. Quality and quantity 31, 401–423 (1997)

    Article  Google Scholar 

  37. Oquist, P.: The epistemology of action research. Acta Sociologica 21, 143–163 (1978)

    Google Scholar 

  38. Susman, G.I., Evered, R.D.: An Assessment of the Scientific Merits of Action Research. Administrative Science Quarterly 23, 582–603 (1978)

    Article  Google Scholar 

  39. Potts, C.: Software-Engineering Research Revisited. IEEE Software 10, 19–28 (1993)

    Article  Google Scholar 

  40. Kaplan, A.: The conduct of inquiry: Methodology for behavioral science. Chandler, New York (1964)

    Google Scholar 

  41. Basili, V.R.: Software Development: A Paradigm for the Future. Presented at COMPSAC 1989, Orlando, Florida (1989)

    Google Scholar 

  42. Sandelin, T., Vierimaa, M.: Empirical Studies in ESERNET. In: Conradi, R., Wang, A.I. (eds.) ESERNET 2001. LNCS, vol. 2765, pp. 39–54. Springer, Heidelberg (2003)

    Chapter  Google Scholar 

  43. Eisenhardt, K.: Building Theories from Case Study Research. Academy of Management Review 14, 532–550 (1989)

    Article  Google Scholar 

  44. Eden, C., Huxham, C.: Action Research for the Study of Organizations. In: Clegg, S.R., Hardy, S. (eds.) Studying Organization: Theory & Method, pp. 272–288. SAGE Publications Ltd, London (1999)

    Google Scholar 

  45. Abrahamsson, P., Salo, O., Ronkainen, J., Warsta, J.: Agile Software Development Methods: Review and Analysis. VTT Electronics, Espoo VTT Publications 478 (2002)

    Google Scholar 

  46. Höst, M., Regnell, B., Wohlin, C.: Using Students as Subjects - A Comparative Study of Students and Professionals in Lead-Time Impact Assessment. Empirical Software 5, 201–214 (2000)

    Article  MATH  Google Scholar 

  47. Humphrey, W.S.: A discipline for software engineering. Addison Wesley Longman, Inc., Amsterdam (1995)

    Google Scholar 

  48. Zahran, S.: Software Process Improvement: Practical Guidelines for Business Success. Addison-Wesley, Reading (1998)

    Google Scholar 

  49. Johnson, P.M.: The personal software process: A cautionary case study. IEEE Software 15, 85–88 (1998)

    Article  Google Scholar 

  50. Dingsøyr, T., Moe, N.B., Nytrø, Ø.: Augmenting Experience Reports with Lightweight Postmortem Reviews. Presented at 3rd Int’l Conference on Product Focused Software Improvement (Profes 2001), Kaiserslautern, Germany (2001)

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. VTT Technical Research Centre of Finland, P.O. Box 1100, FIN-90571, Oulu, Finland

    Outi Salo & Pekka Abrahamsson

Authors
  1. Outi Salo
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Pekka Abrahamsson
    View author publications

    You can also search for this author in PubMed Google Scholar

Editor information

Editors and Affiliations

  1. Fraunhofer Institute for Experimental Software Engineering (IESE), Fraunhofer Platz 1, 67663, Kaiserslautern, Germany

    Frank Bomarius

  2. Nara Institute of Science and Technology, Graduate School of Information Science, 8916-5 Takayama-cho, Ikoma-shi, 630-0192, Nara, Japan

    Hajimu Iida

Rights and permissions

Reprints and permissions

Copyright information

© 2004 Springer-Verlag Berlin Heidelberg

About this paper

Cite this paper

Salo, O., Abrahamsson, P. (2004). Empirical Evaluation of Agile Software Development: The Controlled Case Study Approach. In: Bomarius, F., Iida, H. (eds) Product Focused Software Process Improvement. PROFES 2004. Lecture Notes in Computer Science, vol 3009. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-540-24659-6_29

Download citation

  • DOI: https://doi.org/10.1007/978-3-540-24659-6_29

  • Publisher Name: Springer, Berlin, Heidelberg

  • Print ISBN: 978-3-540-21421-2

  • Online ISBN: 978-3-540-24659-6

  • eBook Packages: Springer Book Archive

Publish with us

Policies and ethics

Search

Navigation