research-article

Impact of CS programs on the quality of test cases generation: an empirical study

Authors:

Omar S. Gómez,

Natalia JuristoAuthors Info & Claims

ICSE '16: Proceedings of the 38th International Conference on Software Engineering Companion

Pages 374 - 383

https://doi.org/10.1145/2889160.2889190

Published: 14 May 2016 Publication History

Abstract

Background: Although most Computer Science (CS) programs offered by higher education institutions usually include a software engineering course, some works report a lack of formal training in software testing. Aim: With the aim of studying the possible impact of knowledge acquired from CS programs on software testing, this paper reports an investigation composed of four experiments. The experiments conducted in Spain, Mexico and Ecuador examine the quality of test cases (TC) generated using black-box and white-box methods. The subjects of the experiments were undergraduate and graduate students who were exposed to different levels of CS knowledge. Method: We pool together the data from the four experiments and apply logistic regression to investigate possible relations of the quality of test cases with students' level of exposure to CS knowledge. Results: The quality of test cases generated by students depend significantly on the amount of CS program studied. The odds of generating test cases that reveal failures against those that do not reveal decrease when students are exposed to a low level of CS knowledge. Conclusions: Software testing plays a key role in what is an increasingly complex process of developing and maintaining software products today. The results of our empirical study provide evidence in favor of greater formal training in software testing as part of CS programs.

References

[1]

ACM. Computer Science Curricula 2013: Curriculum Guidelines for Undergraduate Degree Programs in Computer Science, Joint Task Force on Computing Curricula, Association for Computing Machinery (ACM) and IEEE Computer Society. ACM, New York, NY, USA, 2013. 999133.

Digital Library

[2]

A. Agresti. An introduction to categorical data analysis. Wiley series in probability and statistics. Wiley-Interscience, Hoboken (N.J.), 2007.

[3]

V. Basili and B. Perricone. Software errors and complexity: an empirical investigation. Commun. ACM, 27(1):42--52, 1984.

Digital Library

[4]

V. Basili and R. Selby. Comparing the effectiveness of software testing strategies. IEEE Trans. Softw. Eng., 13(12):1278--1296, 1987.

Digital Library

[5]

B. Beizer. Software testing techniques (2nd ed.). Van Nostrand Reinhold Co., New York, NY, USA, 1990.

Digital Library

[6]

J. Brooks, F. P. No silver bullet essence and accidents of software engineering. Computer, 20(4):10--19, april 1987.

Digital Library

[7]

F. Chan and T. Tse. Software testing education and training in hong kong. In Quality Software, 2005. (QSIC 2005). Fifth International Conference on, pages 313--316, Sept 2005.

Digital Library

[8]

Y. Chernak. Validating and improving test-case effectiveness. Software, IEEE, 18(1):81--86, Jan 2001.

Digital Library

[9]

T. Cook and D. Campbell. The design and conduct of quasi-experiments and true experiments in field settings. Rand McNally, Chicago, 1976.

[10]

S. U. Farooq and S. Quadri. An externally replicated experiment to evaluate software testing methods. In Proceedings of the 17th International Conference on Evaluation and Assessment in Software Engineering, EASE '13, pages 72--77, New York, NY, USA, 2013. ACM.

Digital Library

[11]

V. Garousi and A. Mathur. Current state of the software testing education in north american academia and some recommendations for the new educators. In Software Engineering Education and Training (CSEE T), 2010 23rd IEEE Conference on, pages 89--96, March 2010.

Digital Library

[12]

V. Garousi and J. Zhi. A survey of software testing practices in canada. Journal of Systems and Software, 86(5):1354--1376, 2013.

Digital Library

[13]

D. Graham, E. V. Veenendaal, I. Evans, and R. Black. Foundations of Software Testing: ISTQB Certification. Intl Thomson Business Pr, 2008.

Digital Library

[14]

O. S. Gómez, R. A. Aguilar, and J. P. Ucán. Efectividad de técnicas de prueba de software aplicadas por sujetos novicios de pregado. In M. D. Rodríguez, A. I. Martínez, and J. P. García, editors, Encuentro Nacional de Ciencias de la Computación, (ENC2014), Ocotlán de Morelos, Oaxaca, México, noviembre 2014. ISBN:9780990823605.

[15]

B. Hetzel. The Complete Guide to Software Testing. QED Information Sciences, Inc., Wellesley, MA, USA, 2nd edition, 1988.

Digital Library

[16]

D. W. Hosmer and S. Lemeshow. Applied logistic regression. Wiley series in probability and statistics. John Wiley & Sons, Inc. A Wiley-Interscience Publication, New York, Chichester, Weinheim, 2000.

[17]

W. Howden. Functional program testing. IEEE Transactions on Software Engineering, 6:162--169, 1980.

Digital Library

[18]

J. Huang. Measuring the effectiveness of a test case. In Application-Specific Software Engineering Technology, 1998. ASSET-98. Proceedings. 1998 IEEE Workshop on, pages 157--159, Mar 1998.

Digital Library

[19]

IEEE. Ieee guide for software verification and validation plans. IEEE Std 1059-1993, pages i--87, 1994.

[20]

P. C. Jorgensen. Software Testing: A Craftsman's Approach, Third, Edition. AUERBACH, 3 edition, 2008.

Digital Library

[21]

N. Juristo and A. M. Moreno. Basics of Software Engineering Experimentation. Kluwer Academic Publishers, 2001.

Digital Library

[22]

N. Juristo and S. Vegas. Functional testing, structural testing and code reading: What fault type do they each detect? In R. Conradi and A. Wang, editors, Empirical Methods and Studies in Software Engineering, volume 2765 of Lecture Notes in Computer Science, pages 208--232. Springer Berlin -Heidelberg, 2003.

[23]

N. Juristo, S. Vegas, M. Solari, S. Abrahao, and I. Ramos. Comparing the effectiveness of equivalence partitioning, branch testing and code reading by stepwise abstraction applied by subjects. In Software Testing, Verification and Validation (ICST), 2012 IEEE Fifth International Conference on, pages 330--339, April 2012.

Digital Library

[24]

E. Kamsties and C. M. Lott. An empirical evaluation of three defect-detection techniques. In Proceedings of the 5th European Software Engineering Conference, pages 362--383, London, UK, 1995. Springer-Verlag.

Digital Library

[25]

H. J. Lee, L. Naish, and K. Ramamohanarao. The effectiveness of using non redundant test cases with program spectra for bug localization. In Computer Science and Information Technology, 2009. ICCSIT 2009. 2nd IEEE International Conference on, pages 127--134, Aug 2009.

[26]

R. C. Linger, B. I. Witt, and H. D. Mills. Structured Programming; Theory and Practice the Systems Programming Series. Addison-Wesley Longman Publishing Co., Inc., Boston, MA, USA, 1979.

Digital Library

[27]

B. Marick. The craft of software testing: subsystem testing including object-based and object-oriented testing. Prentice-Hall, Inc., Upper Saddle River, NJ, USA, 1995.

Digital Library

[28]

G. J. Myers and C. Sandler. The Art of Software Testing. John Wiley & Sons, 2004.

Digital Library

[29]

N. J. D. Nagelkerke. A note on a general definition of the coefficient of determination. Biometrika, 78(3):691--692, 1991.

[30]

S. Ng, T. Murnane, K. Reed, D. Grant, and T. Chen. A preliminary survey on software testing practices in australia. In Software Engineering Conference, 2004-Proceedings. 2004 Australian, pages 116--125, 2004.

Digital Library

[31]

M. Roper, M. Wood, and J. Miller. An empirical evaluation of defect detection techniques. Information and Software Technology, 39(11):763--775, 1997.

Digital Library

[32]

P. R. Rosenbaum. Observational Studies (Springer Series in Statistics). Springer, 1 edition, January 2002. ISBN-13: 978-0-3879-8967-9.

[33]

E. Scott, A. Zadirov, S. Feinberg, and R. Jayakody. The alignment of software testing skills of is students with industry practices-a south african perspective. Journal of Information Technology Education, 3:161--172, 2004.

[34]

D. I. K. Sjøberg, B. Anda, E. Arisholm, T. Dybå, M. Jørgensen, A. Karahasanovic, E. F. Koren, and M. Vokác. Conducting realistic experiments in software engineering. In ISESE '02: Proceedings of the 2002 International Symposium on Empirical Software Engineering, page 17, Washington, DC, USA, 2002. IEEE Computer Society.

Digital Library

[35]

A. von Mayrhauser, C. Anderson, and R. Mraz. Using a neural network to predict test case effectiveness. In Aerospace Applications Conference, 1995. Proceedings., 1995 IEEE, number 0, pages 77--91 vol. 2, Feb 1995.

[36]

W. Wong, A. Bertolino, V. Debroy, A. Mathur, J. Offutt, and M. Vouk. Teaching software testing: Experiences, lessons learned and the path forward. In Software Engineering Education and Training (CSEE T), 2011 24th IEEE-CS Conference on, pages 530--534, May 2011.

Digital Library

[37]

W. E. Wong. Improving the state of undergraduate software testing education. In 2012 ASEE Annual Conference, San Antonio, Texas, June 2012. ASEE Conferences. https://peer.asee.org/21511.

[38]

C. H. Yan. On effectiveness of equivalent fundamental pairs as test cases for object-oriented software. In Systems, Man, and Cybernetics, 1999. IEEE SMC '99 Conference Proceedings. 1999 IEEE International Conference on, volume 1, pages 933--937 vol. 1, 1999.

[39]

R. Yang, Z. Chen, B. Xu, W. Wong, and J. Zhang. Improve the effectiveness of test case generation on efsm via automatic path feasibility analysis. In High-Assurance Systems Engineering (HASE), 2011 IEEE 13th International Symposium on, pages 17--24, Nov 2011.

Digital Library

Cited By

Scatalon LGarcia RBarbosa E(2020)Teaching Practices of Software Testing in Programming Education2020 IEEE Frontiers in Education Conference (FIE)10.1109/FIE44824.2020.9274256(1-9)Online publication date: 21-Oct-2020
https://dl.acm.org/doi/10.1109/FIE44824.2020.9274256
Ucán Pech JAguilar Vera RGómez O(2017)Software testing education through a collaborative virtual approachTrends and Applications in Software Engineering10.1007/978-3-319-69341-5_21(231-240)Online publication date: 12-Oct-2017
https://doi.org/10.1007/978-3-319-69341-5_21

Index Terms

Impact of CS programs on the quality of test cases generation: an empirical study
1. General and reference
  1. Cross-computing tools and techniques
    1. Metrics
2. Software and its engineering

Recommendations

Creating a High Quality, High Impact CS Teacher Prep Program
SIGCSE 2022: Proceedings of the 53rd ACM Technical Symposium on Computer Science Education - Volume 1

One of the impacts of the CS For All movement has been an increased demand for K-12 teachers who are well-qualified to teach computer science. Unfortunately, the pool of qualified educators is much smaller than the demand, and the pipeline producing ...
Graduate Programs in CS Education: Why 2020 is the Right Time
SIGCSE '20: Proceedings of the 51st ACM Technical Symposium on Computer Science Education

Opportunities for training CS K-12 pre-service and in-service teachers, research in CS Education, and career pathways for PhDs/EdDs in CS education are happening, but often in an uncoordinated way. We advocate that now is the right time for CS and ...
A Static Approach to Prioritizing JUnit Test Cases

Test case prioritization is used in regression testing to schedule the execution order of test cases so as to expose faults earlier in testing. Over the past few years, many test case prioritization techniques have been proposed in the literature. Most ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

ICSE '16: Proceedings of the 38th International Conference on Software Engineering Companion

May 2016

946 pages

ISBN:9781450342056

DOI:10.1145/2889160

General Chair:
Laura Dillon
Michigan State University
,
Program Chairs:
Willem Visser
Stellenbosch University, South Africa
,
Laurie Williams
North Carolina State University

Copyright © 2016 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

ACM: Association for Computing Machinery
SIGSOFT: ACM Special Interest Group on Software Engineering
IEEE-CS\TCSE: TC on Software Engineering
IEEE-CS\DATC: IEEE Computer Society

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 14 May 2016

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

Government of the Republic of Ecuador's Department of Higher Education, Science, Technology and Innovation (Senescyt)

Conference

ICSE '16

Sponsor:

ACM
SIGSOFT
IEEE-CS\TCSE
IEEE-CS\DATC

ICSE '16: 38th International Conference on Software Engineering

May 14 - 22, 2016

Texas, Austin

Acceptance Rates

Overall Acceptance Rate 276 of 1,856 submissions, 15%

Upcoming Conference

ICSE 2025

2025 IEEE/ACM 46th International Conference on Software Engineering

April 26 - May 3, 2025

Ottawa , ON , Canada

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

2
Total Citations
View Citations
228
Total Downloads

Downloads (Last 12 months)9
Downloads (Last 6 weeks)0

Reflects downloads up to 07 Mar 2025

Other Metrics

View Author Metrics

Citations

Cited By

Scatalon LGarcia RBarbosa E(2020)Teaching Practices of Software Testing in Programming Education2020 IEEE Frontiers in Education Conference (FIE)10.1109/FIE44824.2020.9274256(1-9)Online publication date: 21-Oct-2020
https://dl.acm.org/doi/10.1109/FIE44824.2020.9274256
Ucán Pech JAguilar Vera RGómez O(2017)Software testing education through a collaborative virtual approachTrends and Applications in Software Engineering10.1007/978-3-319-69341-5_21(231-240)Online publication date: 12-Oct-2017
https://doi.org/10.1007/978-3-319-69341-5_21

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten