Abstract
Researchers without a technical background in statistics may be tempted to apply analytical techniques in a ritualistic manner. SBSE research is not immune to this problem. We argue that emerging rituals surrounding the use of the Vargha-Delaney effect size statistic may pose serious threats to the scientific validity of the findings. We believe investigations of effect size are important, but more care is required in the application of this statistic. In this paper, we illustrate the problems that can arise, and give guidelines for avoiding them, by applying a ‘transformed’ Vargha-Delaney effect size measurement. We argue that researchers should always consider which transformation is best suited to their problem domain before calculating the Vargha-Delaney statistic.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Ali, S., Briand, L.C., Hemmati, H., Panesar-Walawege, R.K.: A systematic review of the application and empirical investigation of search-based test case generation. IEEE Trans. Softw. Eng. 36(6), 742–762 (2010)
Amdahl, G.M.: Validity of the single processor approach to achieving large scale computing capabilities. In: Proceedings of the Spring Joint Computer Conference, pp. 483–485. ACM, 18–20 April 1967
Arcuri, A., Briand, L.: A hitchhiker’s guide to statistical tests for assessing randomized algorithms in software engineering. Softw. Test. Verif. Reliab. 24, 219–250 (2012)
Department of Defense: US DOD MIL-STD 1472-F: Human engineering standard (1999)
Harman, M., Jia, Y., Zhang, Y.: Achievements, open problems and challenges for search based software testing (keynote). In: 8th IEEE International Conference on Software Testing, Verification and Validation (ICST 2014), Graz, Austria, April 2015
Harman, M., Jones, B.F.: Search based software engineering. Inf. Softw. Technol. 43(14), 833–839 (2001)
Harman, M., Tratt, L.: Pareto optimal search-based refactoring at the design level. In: 9th Annual Conference on Genetic and Evolutionary Computation (GECCO 2007), pp. 1106–1113. ACM Press, London, July 2007
Hsu, H.Y., Orso, A.: Mints: a general framework and tool for supporting test-suite minimization. In: IEEE 31st International Conference on Software Engineering, ICSE 2009, pp. 419–429. IEEE (2009)
Ireson, N., Ciravegna, F., Califf, M.E., Freitag, D., Kushmerick, N., Lavelli, A.: Evaluating machine learning for information extraction. In: Proceedings of the 22nd international conference on Machine learning, pp. 345–352. ACM (2005)
McMinn, P.: How does program structure impact the effectiveness of the crossover operator in evolutionary testing? In: 2010 Second International Symposium on Search Based Software Engineering (SSBSE), pp. 9–18. IEEE (2010)
Moore, G.E., et al.: Cramming more components onto integrated circuits. Proc. IEEE 86(1), 82–85 (1998)
O’Keeffe, M., ÓCinnédie, M.: Search-based refactoring: an empirical study. J. Softw. Maint. 20(5), 345–364 (2008)
Schneider, K., Knauss, E., Houmb, S., Islam, S., Jürjens, J.: Enhancing security requirements engineering by organizational learning. Requirements Eng. 17(1), 35–56 (2012)
Shatnawi, R.: A quantitative investigation of the acceptable risk levels of object-oriented metrics in open-source systems. IEEE Trans. Software Eng. 36(2), 216–225 (2010)
Singh, S., Kahlon, K.: Object oriented software metrics threshold values at quantitative acceptable risk level. CSI Transact. ICT 2(3), 191–205 (2014)
Srinivas, N., Deb, K.: Multi-objective function optimization using non-dominated sorting genetic algorithms. Evol. Comput. 2(3), 221–148 (1995)
Vargha, A., Delaney, H.D.: A critique and improvement of the CL common language effect size statistics of McGraw and Wong. J. Educ. Behav. Stat. 25(2), 101–132 (2000)
Yoo, S., Harman, M.: Regression testing minimisation, selection and prioritisation: a survey. J. Softw. Test. Verif. Reliab. 22(2), 67–120 (2012)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2015 Springer International Publishing Switzerland
About this paper
Cite this paper
Neumann, G., Harman, M., Poulding, S. (2015). Transformed Vargha-Delaney Effect Size. In: Barros, M., Labiche, Y. (eds) Search-Based Software Engineering. SSBSE 2015. Lecture Notes in Computer Science(), vol 9275. Springer, Cham. https://doi.org/10.1007/978-3-319-22183-0_29
Download citation
DOI: https://doi.org/10.1007/978-3-319-22183-0_29
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-22182-3
Online ISBN: 978-3-319-22183-0
eBook Packages: Computer ScienceComputer Science (R0)