research-article

Automated Expected Output Generation: Is this a Problem that has been solved?

Author:

Yogesh SinghAuthors Info & Claims

ACM SIGSOFT Software Engineering Notes, Volume 40, Issue 6

Pages 1 - 5

https://doi.org/10.1145/2830719.2830726

Published: 11 November 2015 Publication History

Abstract

What is the importance of automated test data generation? This question is of concern for the entire software community. We want to know the meaning and usefulness of automated test data generation. Is it related to design of test cases or execution of already designed test cases or both? How can we generate expected output using automated test data generation technique? This is a weak area and limits the applicability of automated testing. The test case creation, execution and evaluation are three distinct activities. The execution and evaluation of test cases can be easily automated, but creation needs expected output that is difficult to automate due to its dependency on functionality of the program. This paper addresses the significance of techniques and tools available for automated test data generation. In this work, we analyze and assess the existing techniques for automated expected output generation and discuss the practical significance of these techniques.

References

[1]

Y. Singh. 2012. Software Testing. Cambridge University Press, USA.

[2]

K. Lakhotia, P. McMinn, and M. Harman. 2009. Automated Test Data Generation for Coverage: Haven't We Solved This Problem Yet?. In Proceedings of the 2009 Testing: Academic and Industrial Conference - Practice and Research Techniques (TAIC-PART '09). IEEE Computer Society, Washington, DC, USA, 95--104. DOI=10.1109/TAICPART.2009.15 http://dx.doi.org/10.1109/TAICPART.2009.15.

Digital Library

[3]

C. V. Ramamoorthy, S. -B. F. Ho, and W. T. Chen. 1976. On the Automated Generation of Program Test Data. IEEE Trans. Softw. Eng. 2, 4 (July 1976), 293--300. DOI=10.1109/TSE.1976.233835 http://dx.doi.org/10.1109/TSE.1976.233835.

Digital Library

[4]

John B. Goodenough. 1975. Toward a theory of test data selection. IEEE Trans. Softw. Eng. 1, 1 (March 1975), 156--173. DOI=10.1109/TSE.1975.6312836 http://dx.doi.org/10.1109/TSE.1975.6312836.

Digital Library

[5]

S. Anand, Edmund K. Burke, Tsong Yueh Chen, John Clark, Myra B. Cohen, Wolfgang Grieskamp, Mark Harman, Mary Jean Harrold, and Phil Mcminn. 2013. An orchestrated survey of methodologies for automated software test case generation. J. Syst. Softw. 86, 8 (August 2013), 1978--2001. DOI=10.1016/j.jss.2013.02.061 http://dx.doi.org/10.1016/j.jss.2013.02.061.

Digital Library

[6]

B. Korel. 1996. Automated test data generation for programs with procedures. In Proceedings of the 1996 ACM SIGSOFT international symposium on Software testing and analysis (ISSTA '96), Steve J. Zeil and Will Tracz (Eds.). ACM, New York, NY, USA, 209--215. DOI=10.1145/229000.226319 http://doi.acm.org/10.1145/229000.226319.

Digital Library

[7]

A.V.K. Shanthi and G. Mohan Kumar. 2011. Automated test cases generation for object oriented software. Indian Journal of Computer Science & Engineering 2, 4, Aug-Sept, 543--546.

[8]

A. Baresel, H. Sthamer, and M. Schmidt. 2002. Fitness function design to improve evolutionary structural testing. In Proceedings of the Genetic and Evolutionary Computation Conference (GECCO 2002), New York, USA, Morgan Kaufmann, 1329--1336.

Digital Library

[9]

B. Jones, H. Sthamer, X. Yang, and D. Eyres. 1995. The automatic generation of software test data sets using adaptive search techniques. In Proceedings of the 3rd International Conference on Software Quality Management, Seville, Spain, 435--444.

[10]

Joachim Wegener, Kerstin Buhr, and Hartmut Pohlheim. 2002. Automatic Test Data Generation For Structural Testing Of Embedded Software Systems By Evolutionary Testing. In Proceedings of the Genetic and Evolutionary Computation Conference (GECCO '02), William B. Langdon, Erick Cantí-Paz, Keith E. Mathias, Rajkumar Roy, David Davis, Riccardo Poli, Karthik Balakrishnan, Vasant G. Honavar, Günter Rudolph, Joachim Wegener, Larry Bull, Mitchell A. Potter, Alan C. Schultz, J. F. Miller, E. Burke, and Natasa Jonoska (Eds.). Morgan Kaufmann Publishers Inc., San Francisco, CA, USA, 1233--1240

Digital Library

[11]

V. Santiago, A.S. Martins do Amaral, N. L. Vijaykumar, Maria de Fatima Mattiello-Francisco, E. Martins, and Odnei Cuesta Lopes. 2006. A Practical Approach for Automated Test Case Generation using Statecharts. In Proceedings of the 30th Annual International Computer Software and Applications Conference - Volume 02 (COMPSAC '06), Vol. 2. IEEE Computer Society, Washington, DC, USA, 183--188. DOI=10.1109/COMPSAC.2006.100 http://dx.doi.org/10.1109/COMPSAC.2006.100.

Digital Library

[12]

R.A. DeMillo and A. J. Offutt. 1991. Constraint-Based Automatic Test Data Generation. IEEE Trans. Softw. Eng. 17, 9 (September 1991), 900--910. DOI=10.1109/32.92910 http://dx.doi.org/10.1109/32.92910.

Digital Library

[13]

M. Balcer, W. Hasling, and T. Ostrand. 1989. Automatic generation of test scripts from formal test specifications. SIGSOFT Softw. Eng. Notes 14, 8 (November 1989), 210--218. DOI=10.1145/75309.75332 http://doi.acm.org/10.1145/75309.75332.

Digital Library

[14]

R. Malhotra, A. Kaur and Y. Singh. 2010. A Regression Test Selection and Prioritization Technique. Journal of Information Processing Systems 6, 2 (June 2010), 235--252.

[15]

Robert M. Poston. 1997. Automating Specification-Based Software Testing (1st ed.). IEEE Computer Society Press, Los Alamitos, CA, USA.

Digital Library

[16]

Robert M. Poston. 1994. Automated testing from object models. Commun. ACM 37, 9 (September 1994), 48--58. DOI=10.1145/182987.184074 http://doi.acm.org/10.1145/182987.184074.

Digital Library

[17]

Q. Xie and A.M. Memon. 2007. Designing and comparing automated test oracles for GUI-based software applications. ACM Trans. Softw. Eng. Methodol. 16, 1, Article 4 (February 2007). DOI=10.1145/1189748.1189752 http://doi.acm.org/10.1145/1189748.1189752.

Digital Library

[18]

Dennis Peters and David L. Parnas. 1994. Generating a test oracle from program documentation: work in progress. In Proceedings of the 1994 ACM SIGSOFT international symposium on Software testing and analysis (ISSTA '94), Thomas Ostrand (Ed.). ACM, New York, NY, USA, 58--65. DOI=10.1145/186258.186508 http://doi.acm.org/10.1145/186258.186508.

Digital Library

[19]

Debra J. Richardson. 1994. TAOS: Testing with Analysis and Oracle Support. In Proceedings of the 1994 ACM SIGSOFT international symposium on Software testing and analysis (ISSTA '94), Thomas Ostrand (Ed.). ACM, New York, NY, USA, 138--153. DOI=10.1145/186258.187158 http://doi.acm.org/10.1145/186258.187158.

Digital Library

[20]

L. K. Dillon and Y. S. Ramakrishna. 1996. Generating oracles from your favorite temporal logic specifications. SIGSOFT Softw. Eng. Notes 21, 6 (October 1996), 106--117. DOI=10.1145/250707.239116 http://doi.acm.org/10.1145/250707.239116.

Digital Library

[21]

P. A. V. Hall. 1998. Towards testing with respect to formal specification. In Second IEE/BCS Conference on Software Engineering IEEE, 159--163.

[22]

P. A. V. Hall. 1991. Relationship between specifications and testing. Inf. Softw. Technol. 33, 1 (February 1991), 47-52. DOI=10.1016/0950-5849(91)90023-5 http://dx.doi.org/10.1016/0950-5849(91)90023-5.

Digital Library

[23]

Debra J. Richardson, Stephanie Leif Aha, and T. Owen O'Malley. 1992. Specification-based test oracles for reactive systems. In Proceedings of the 14th international conference on Software engineering (ICSE '92). ACM, New York, NY, USA, 105--118. DOI=10.1145/143062.143100 http://doi.acm.org/10.1145/143062.143100.

Digital Library

[24]

L. I. Manolache and D. G. Kourie. 2001. Software testing using model programs. Softw. Pract. Exper. 31, 13 (October 2001), 1211--1236. DOI=10.1002/spe.409 http://dx.doi.org/10.1002/spe.409.

Digital Library

[25]

Patrick J. Schroeder and Bogdan Korel. 2000. Black-box test reduction using input-output analysis.SIGSOFT Softw. Eng. Notes 25, 5 (August 2000), 173--177. DOI=10.1145/347636.349042 http://doi.acm.org/10.1145/347636.349042.

Digital Library

[26]

G. Di Lucca, A. Fasolino, and F. Faralli. 2002. Testing Web Applications. In Proceedings of the International Conference on Software Maintenance (ICSM'02) (ICSM '02). IEEE Computer Society, Washington, DC, USA, 310--319.

Digital Library

[27]

K. K. Aggarwal, Yogesh Singh, A. Kaur, and O. P. Sangwan. 2004. A neural net based approach to Test Oracle. SIGSOFT Softw. Eng. Notes 29, 3 (May 2004), 1--6. DOI=10.1145/986710.986725 http://doi.acm.org/10.1145/986710.986725.

Digital Library

[28]

Ye Mao, Feng Boqin, Zhu Li, and Lin Yao. 2006. Neural networks based automated test oracle for software testing. In Proceedings of the 13th international conference on Neural information processing - Volume Part III (ICONIP'06), Irwin King, Laiwan Chan, Jun Wang, and DeLiang Wang (Eds.), Vol. Part III. Springer-Verlag, Berlin, Heidelberg, 498--507.

Digital Library

[29]

M. Vanmali, M. Last, and A. Kandel. 2002. Using a neural network in the software testing process. International Journal of Intelligent Systems 17, 1, 45--62.

[30]

Atif M. Memon, Martha E. Pollack, and Mary Lou Soffa. 2000. Automated test oracles for GUIs. SIGSOFT Softw. Eng. Notes 25, 6 (November 2000), 30--39. DOI=10.1145/357474.355050 http://doi.acm.org/10.1145/357474.355050.

Digital Library

[31]

S.R. Shahamiri, W.M.N.W. Kadir, S.Z. Mohd-Hashim, 2009. A comparative study on automated software test oracle methods, in Proceedings of the 2009 Fourth International Conference on Software Engineering Advances, IEEE Computer Society, Porto, Portugal, 140--145.

Digital Library

[32]

E.T. Barr, M. Harman, P. McMinn, M. Shahbaz, S. Yoo. 2015. The Oracle Problem in Software Testing: A Survey. IEEE Transactions on Software Engineering 41, 5, 507--525.

Digital Library

[33]

J. Su and P. R. Ritter. 1991. Experience in testing the Motif interface. IEEE Software 8, 2 (Mar. 1991), 6--33.

Digital Library

[34]

P. Vogel. 1993. An integrated general purpose automated test environment. In T. Ostrand and E. Weyuker, editors, Proceedings of the International Symposium on Software Testing and Analysis, New York, NY, USA, ACM Press, 61--69.

Digital Library

Cited By

Khari MKumar PBurgos DCrespo R(2018)Optimized test suites for automated testing using different optimization techniquesSoft Computing - A Fusion of Foundations, Methodologies and Applications10.1007/s00500-017-2780-722:24(8341-8352)Online publication date: 1-Dec-2018
https://dl.acm.org/doi/10.1007/s00500-017-2780-7

Recommendations

Symbolic execution and program testing

This paper describes the symbolic execution of programs. Instead of supplying the normal inputs to a program (e.g. numbers) one supplies symbols representing arbitrary values. The execution proceeds as in a normal execution except that values may be ...
Symbolic execution for software testing: three decades later

The challenges---and great promise---of modern symbolic execution techniques, and the tools to help implement them.

Comments

Information & Contributors

Information

Published In

cover image ACM SIGSOFT Software Engineering Notes

ACM SIGSOFT Software Engineering Notes Volume 40, Issue 6

November 2015

77 pages

ISSN:0163-5948

DOI:10.1145/2830719

Editor:
John Georgas

Issue’s Table of Contents

Copyright © 2015 Author.

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 11 November 2015

Published in SIGSOFT Volume 40, Issue 6

Check for updates

Author Tags

Qualifiers

Research-article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

1
Total Citations
View Citations
125
Total Downloads

Downloads (Last 12 months)2
Downloads (Last 6 weeks)0

Reflects downloads up to 03 Mar 2025

Other Metrics

View Author Metrics

Citations

Cited By

Khari MKumar PBurgos DCrespo R(2018)Optimized test suites for automated testing using different optimization techniquesSoft Computing - A Fusion of Foundations, Methodologies and Applications10.1007/s00500-017-2780-722:24(8341-8352)Online publication date: 1-Dec-2018
https://dl.acm.org/doi/10.1007/s00500-017-2780-7

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Issue’s Table of Contents