Skip to main content
SpringerLink
Log in

A checklist for integrating student empirical studies with research and teaching goals

  • Published:
Empirical Software Engineering Aims and scope Submit manuscript

Abstract

The use of empirical studies with students in software engineering helps researchers gain insight into new or existing techniques and methods. However, due mainly to concerns of external validity, questions have been raised about the value of these types of studies. The authors of this paper draw on their experiences of conducting a large number of empirical studies in university courses in three countries (Italy, Norway, and the United States) to address this important issue. This paper first identifies the requirements that research and pedagogy place on a valid empirical study with students. This information is then used as the basis for a checklist that provides guidance for researchers and educators when planning and conducting studies in university courses. The goal of this checklist is to help ensure that these studies have as much research and pedagogical value as possible. Finally, an example application of the checklist is provided to illustrate its use.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1

Similar content being viewed by others

Notes

  1. http://www.highproductivity.org

References

  • Bagert D, Hilburn TB, Hislop G, Lutz M, McCracken M, Mengel S (1999) Guidelines for Software Engineering Education. SEI Technical Reports. CMU/SEI-99-TR-032

  • Baresi L, Morasca S (2002) An empirical study on the design effort of web applications. Proceedings of 3rd International Conference on Web Information Systems Engineering, Singapore. 345–354

  • Baresi L, Morasca S, Paolini P (2003) Estimating the design effort of web applications. Proceedings of Ninth International Software Metrics Symposium. 62–72

  • Basili VR, Briand LC, Melo WL (1996) A validation of object-oriented design metrics as quality indicators. IEEE Trans Softw Eng 22(10):751–761. doi:10.1109/32.544352

    Article  Google Scholar 

  • Basili VR, Carver JC, Cruzes D, Hochstein LM, Hollingsworth JK, Shull F, Zelkowitz MV (2008) Understanding the high-performance-computing community: a software engineer’s perspective. IEEE Softw 25(4):29–36. doi:10.1109/MS.2008.103

    Article  Google Scholar 

  • Bloom BS (ed) (1956) Handbook I, cognitive domain. Taxonomy of educational objectives: the classification of educational goals. Longman, New York

  • Braught G (2005) Teaching empirical skills and concepts in computer science using random walks. Proceedings of 36th SIGCSE technical symposium on Computer science education, St. Louis, Missouri, USA, ACM Press. 41–45

  • Carver J, Jaccheri L, Morasca S, Shull F (2003) Issues in using students in empirical studies in software engineering education. Proceedings of Ninth International Software Metrics Symposium (METRICS 2003). 239–249

  • CORPORATE (2001) Computing Curricula 2001. J Educ Resour Comput 1(3es):1

    Article  Google Scholar 

  • Daly J (1996) Replication and a multi-method approach to empirical software engineering research. Department of Computer Science. University of Strathclyde. PhD

  • Denning PJ (1992) Educating a new engineer. Commun ACM 35(12):82–97. doi:10.1145/138859.138870

    Article  Google Scholar 

  • Hilburn TB, Humphrey WS (2002) The impending changes in software education. IEEE Softw 19(5):22–24. doi:10.1109/MS.2002.1032848

    Article  Google Scholar 

  • Hochstein L, Nakamura T, Basili VR, Asgari S, Zelkowitz MV, Hollingsworth JK, Shull F, Carver J, Voelp M, Zazworka N, Johnson P (2006) Experiments to understand Hpc time to development. CTWatch Quarterly. November: 24–32

  • Höst M (2002) Introducing empirical software engineering methods in education. Proceedings of 15th Conference on Software Engineering Education and Training, 2002. (CSEE&T 2002). 170–179

  • Höst M, Regnell B, Wohlin C (2000) Using students as subjects-a comparative study of students and professionals in lead-time impact assessment. Empir Softw Eng 5(3):201–214. doi:10.1023/A:1026586415054

    Article  MATH  Google Scholar 

  • Höst M, Wohlin C, Thelin T (2005) Experimental context classification: incentives and experience of subjects. Proceedings of 27th international conference on Software engineering, St. Louis, MO, USA, ACM Press. 470–478

  • Jaccheri L (2001) Software quality and software process improvement course based on interaction with the local software industry. Comput Appl Eng Educ 9(4):265–272. doi:10.1002/cae.10000

    Article  Google Scholar 

  • Jay R (2002) How to build a great team, Financial Times Management

  • Jorgensen M, Teigen KH, Molokken K (2004) Better sure than safe? Over-confidence in judgement based software development effort prediction intervals. J Syst Softw 70(1–2):79–93. doi:10.1016/S0164-1212(02)00160-7

    Article  Google Scholar 

  • Kitchenham BA, Pfleeger SL, Pickard LM, Jones PW, Hoaglin DC, El Emam K, Rosenberg J (2002) Preliminary guidelines for empirical research in software engineering. IEEE Trans Softw Eng 28(8):721–734. doi:10.1109/TSE.2002.1027796

    Article  Google Scholar 

  • McBurney DH (2001) Research methods. Wadsworth Thomson Learning, Stamford

    Google Scholar 

  • Miller J (2005) Replicating software engineering experiments: a poisoned chalice or the Holy Grail. Inf Softw Technol 47(4):233–244. doi:10.1016/j.infsof.2004.08.005

    Article  Google Scholar 

  • Morasca S (2003) A Bayesian approach to software testing evaluation. Proceedings of Software Engineering and Knowledge Engineering, San Francisco Bay, USA. 706–713.

  • Pastel R (2005) Integrating science and research in a Hci design course. St Louis, ACM Press

    Google Scholar 

  • Port D, Klappholz D (2004) Empirical research in the software engineering classroom. Proceedings of 17th Conference on Software Engineering Education and Training, 2004. 132–137

  • Shull F, Lanubile F, Basili VR (2000) Investigating reading techniques for object-oriented framework learning. IEEE Trans Softw Eng 26(11):1101–1118. doi:10.1109/32.881720

    Article  Google Scholar 

  • Shull F, Carver J, Travassos G (2001) An empirical methodology for introducing software processes. Proceedings of The Joint 8th European Software Engineering Conference and 9th ACM SIGSOFT Foundations of Software Engineering, Vienna, Austria. 288–296

  • Shull F, Carver J, Hochstein L, Basili VR (2005) Empirical study design in the area of high performance computing (Hpc). Proceedings of International Symposium on Empirical Software Engineering, Noosa Heads, Australia. 305–314

  • Shull F, Carver J, Vegas S, Juristo N (2008) The role of replications in empirical software engineering. Empir Softw Eng 13(2):211–218. doi:10.1007/s10664-008-9060-1

    Article  Google Scholar 

  • Singer J, Vinson NG (2002) Ethical issues in empirical studies of software engineering. IEEE Trans Softw Eng 28(12):1171–1180. doi:10.1109/TSE.2002.1158289

    Article  Google Scholar 

  • Sjoeberg DIK, Anda B, Arisholm E, Dyba T, Jorgensen M, Karahasanovic A, Koren EF, Vokac M (2002) Conducting realistic experiments in software engineering. Proceedings of 2002 International Symposium on Empirical Software Engineering. 17–26.

  • Sjoeberg DIK, Hannay JE, Hansen O, Kampenes VB, Karahasanovic A, Liborg NK, Rekdal AC (2005) A survey of controlled experiments in software engineering. IEEE Trans Softw Eng 31(9):733–753. doi:10.1109/TSE.2005.97

    Article  Google Scholar 

  • Tichy WF (2000) Hints for reviewing empirical work in software engineering. Empir Softw Eng 5(4):309–312. doi:10.1023/A:1009844119158

    Article  MathSciNet  Google Scholar 

  • Umphress DA, Hendrix TD, Cross JH (2002) Software process in the classroom: the capstone project experience. IEEE Softw 19(5):78–81. doi:10.1109/MS.2002.1032858

    Article  Google Scholar 

  • Vygotsky LS (1978) Mind in society: development of higher psychological processes. Harvard University Press, Cambridge

    Google Scholar 

  • Walia GS, Carver J (2006) Requirements error abstraction and classification: an empirical study. Proceedings of The 5th International Symposium on Empirical Software Engineering, Rio de Janeiro. 336–345

  • Wang AI, Arisholm E (2008) The effect of task order on the maintainability of object-oriented software. Information and Software Technology. Accepted for Publication: Technical Report Available: http://www.idi.ntnu.no/grupper/su/publ/alfw/idi-tr-02-07.pdf

  • Wang AI, Arisholm E, Jaccheri L (2007) Educational approach to an experiment in a software architecture course proceedings of The 20th Conference on Software Engineering Education & Training, 2007. 291–300

  • Way TP (2005) A company-based framework for a software engineering course. Proceedings of 36th SIGCSE technical symposium on Computer science education, St. Louis, Missouri, USA, ACM Press. 132–136

  • Wohlin C, Runeson P, Host M, Ohlsson MC, Regnell B, Wesslen A (2000) Experimentation in software engineering: an introduction, Kluwer Academic Publishers

  • Wood M, Daly J, Miller J, Roper M (1999) Mulit-method research: an empirical investigation of object-oriented technology. J Syst Softw 48(1):13–26. doi:10.1016/S0164-1212(99)00042-4

    Article  Google Scholar 

  • Zelkowitz MV, Wallace DR (1998) Experimental models for validating technology. IEEE Comput 31(5):23–31

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of Computer Science, University of Alabama, Tuscaloosa, AL, USA

    Jeffrey C. Carver

  2. Department of Computer and Information Science, Norwegian University of Science and Technology, Trondheim, Norway

    Letizia Jaccheri

  3. Dipartimento di Scienze della Cultura, Politiche e dell’Informazione, Università degli Studi dell’Insubria, Varese, Italy

    Sandro Morasca

  4. Frauhnofer Center for Experimental Software Engineering, College Park, MD, USA

    Forrest Shull

Authors
  1. Jeffrey C. Carver
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Letizia Jaccheri
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Sandro Morasca
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Forrest Shull
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Jeffrey C. Carver.

Additional information

Editor: Dieter Rombach

Rights and permissions

Reprints and permissions

About this article

Cite this article

Carver, J.C., Jaccheri, L., Morasca, S. et al. A checklist for integrating student empirical studies with research and teaching goals. Empir Software Eng 15, 35–59 (2010). https://doi.org/10.1007/s10664-009-9109-9

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10664-009-9109-9

Keywords

Search

Navigation