Empirical evaluation of optimization algorithms when used in goal-oriented automated test data generation techniques

Xiao, Man; El-Attar, Mohamed; Reformat, Marek; Miller, James

doi:10.1007/s10664-006-9026-0

Empirical evaluation of optimization algorithms when used in goal-oriented automated test data generation techniques

Published: 08 November 2006

Volume 12, pages 183–239, (2007)
Cite this article

Empirical Software Engineering Aims and scope Submit manuscript

Man Xiao¹,
Mohamed El-Attar¹,
Marek Reformat¹ &
…
James Miller¹

574 Accesses
48 Citations
Explore all metrics

Abstract

Software testing is an essential process in software development. Software testing is very costly, often consuming half the financial resources assigned to a project. The most laborious part of software testing is the generation of test-data. Currently, this process is principally a manual process. Hence, the automation of test-data generation can significantly cut the total cost of software testing and the software development cycle in general. A number of automated test-data generation approaches have already been explored. This paper highlights the goal-oriented approach as a promising approach to devise automated test-data generators. A range of optimization techniques can be used within these goal-oriented test-data generators, and their respective characteristics, when applied to these situations remain relatively unexplored. Therefore, in this paper, a comparative study about the effectiveness of the most commonly used optimization techniques is conducted.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Sampling in software engineering research: a critical review and guidelines

Article 28 April 2022

Test case selection and prioritization using machine learning: a systematic literature review

Article 14 December 2021

A systematic review of software usability studies

Article 11 December 2017

Notes

Note that a process of selection of parents in Koakutsu et al. (1995) is random, which is different from the GSA we used in experiments presented in the paper.
Source Code is available from the authors upon request.
Due to the lack of space, all coverage plots presented in this paper show the performance of the optimization techniques throughout the first 10,000 SUT iterations. It is important to note that the optimization techniques will keep operating beyond 10,000 SUT iterations, possibly improving their coverage level.
This program uses a slightly simplified version of history, and assumes that the Gregorian Calendar starts on the 15th of October 1582. Hence, this date represents day 1 for the Time Shuttle program.

References

Back T (1996) Evolutionary algorithms in theory and practice. Oxford University Press, New York
Google Scholar
Back T, Hoffmeister F (1991) Extended selection mechanisms in genetic algorithms. Proc. of the Fourth Int. Conf. on Genetic Algorithms 92–99
Back T, Fogel DB, Michalewicz Z (eds) (2000) Evolutionary computations I. Institute of Physics Publishing, Bristol
Baker JE (1985) Adaptive selection methods for genetic algorithms. Proc. of an International Conference on Genetic Algorithms. Erlbaum, Hillsdale
Beizer B (1990) Software testing techniques. Van Nostrand Reinhold, 2nd edn
Bird D, Munoz C (1982) Automatic generation of random Se•checking test cases. IBM Syst J 22(3):229–245
Article Google Scholar
Caruana RA, Schaffer JD (1988) Representation and hidden bias: Gray vs. binary coding for genetic algorithms. In: Proceedings of the Fifth International Conference on Machine Learning, Morgan Kaufmann
Chang J, Richardson D, Sankar S (1996) Structural specification-based testing with ADL. In Proceedings of ISSTA’96 (International Symposium on Software Testing and Analysis), pp 62–70, San Diego
Change KH, Cross JH, Carlisle WH, Brown DB (1991) A framework for intelligent test data generators. J Intell Robot Syst-Theory and Application
Clarke LA (1976) A system to generate test data and symbolically execute programs. IEEE Trans Softw Eng 2(3):215–222
Google Scholar
Cohen J (1988) Statistical power analysis for the behavioral sciences, 2nd edn. Erlbaum, New Jersey
MATH Google Scholar
Corana A, Marchesi M, Martini C, Ridella S (1987) Minimizing multimodal functions of continuous variables with the “simulated annealing” algorithm. ACM Trans Math Softw 13(3):262–280
Article MATH MathSciNet Google Scholar
De Jong, K (1975) The analysis of the behavior of a class of genetic adaptive systems, Ph.D. Dissertation, Dept, of Computer Science, University of Michigan, Ann Arbor
Dowsland KA (1993) Modern heuristic techniques for combinatorial problems. Chapter 2—simulated annealing, McGraw Hill, pp 20–69
Eiben AE, Hinterding R, Michalewicz Z (1999) Parameter control in evolutionary algorithms. IEEE Transactions Computation 3(2):124–139
Article Google Scholar
Goldberg DE (1989) Genetic algorithms in search, optimization, and machine learning. Addison Wesley, New York
MATH Google Scholar
Greffenstette JJ (1986) Optimization of control parameters for genetic algorithms. IEEE Trans Syst Man Cybern SMC-16 1:122–128
Article Google Scholar
Hedges LV, Olkin I (1985) Statistical methods for meta-analysis. Academic, Orlando, FL
MATH Google Scholar
Holland JH (1975) Adaptation in natural and artificial systems. University of Michigan Press
Howden WW (1997) Symbolic testing and the DISSECT symbolic evaluation system. IEEE Trans Softw Eng 3(4):266–278
Google Scholar
Janikow CZ, Michalewicz Z (1991) An experimental comparison of binary and floating point representations in genetic algorithms. In: Belew RK, Booker LB (eds) Proceedings of the Fourth International Conference on Genetic Algorithms, Morgan Kaufmann, pp 31–36
Jones B, Sthamer H, Eyres D (1995) The automatic generation of software test data sets using adaptive search techniques. Proc. of the 3rd Conference on Software Quality Management 2:435–444
Kirkpatrick S, Gelatt CD Jr, Vecchi MP (1983) Optimization by simulated annealing. Science 220(4598):671–680
Article MathSciNet Google Scholar
Koakutsu S, Kang M, Dai WW (1995) Genetic simulated annealing and applications to non-slicing floorplan design. (UCSC-CRL-95-52)
Korel B (1990) A dynamic approach of automated test data generation. Conference on Software Maintenance 311–317
Korel B (1996) Automated test data generation for programs with procedures. In: Ziel SJ (ed) Proceedings of the 1996 International Symposium on Software Testing and Analysis (ISSTA), pp 209–215
McGraw KO, Wong SP (1992) A common language effect-size statistic. Psychol Bull 111:361–365
Article Google Scholar
McMinn P (2004) Search-based software test data generation: a survey. Softw Test Verif Reliab 14(2):105–156
Article Google Scholar
Michael CC, McGraw G, Schatz MA (2001) Generating software test data by evolution. IEEE Trans Softw Eng 27(12):1085–1110
Article Google Scholar
Miki M, Hiroyasu T, Ono K (2002) Simulated annealing with advanced adaptive neighborhood. In: Computational intelligence and applications (Proceedings of the 2nd International Workshop on Intelligent Systems Design and Applications: ISDA-02, 113–118 2002
Miller J (2004) Statistical significance testing—a panacea for software technology experiments? J Syst Sofw 73:183–192
Article Google Scholar
Mills HD, Dyer MD, Linger RC (1987) Cleanroom software engineering. IEEE Softw 4(5):19–25
Google Scholar
Osman AH, Kelly JP (1996) Meta-heuristics: an overview meta-heuristics: Theory & applications. 1–22
Ramamoorthy CV, Ho SF, Chen WT (1976) On the generation of program test data. IEEE Trans Softw Eng 2(4):293–300
Google Scholar
Schaffer JD (1987) Some effects of selection procedures on hyperplane sampling by genetic algorithms. In: Davis L (ed) Genetic algorithms and simulated annealing. Pitman
Schaffer JD, Caruana R, Eshelman L, Das R (1989) A study of control parameters affecting online performance of genetic algorithms for function optimization. In: Proceedings of the 3rd International Conference in Genetic Algorithms, Morgan Kaufmann, San Mateo, California, pp 51–60
Tassey G (2002) The economics impact of inadequate infrastructure for software testing. National Institute of Standards and Technology report
Thevenhod-Fosse P, Waeselynck H (1998) STATEMATE: applied to statistical software testing. Proc. of the 1998 International Symposium on Software Testing and Analysis
Tracy N, Clark J, Mander K (1998) Automated program flaw finding using simulated annealing. Proc. of the International Symposium on Software Testing and Analysis
Voas JM, Morell L, Miller KW (1991) Predicting where faults can hide from testing. IEEE Softw 8(2):41–38
Article Google Scholar
Whitley D (1993) (ed) An executable model of a simple genetic algorithm. Foundations of Genetic Algorithms (FOGA 2). Morgan Kaufmann

Download references

Author information

Authors and Affiliations

Department of Electrical and Computer Engineering, STEAM Laboratory, University of Alberta, 9107-116 Street, Edmonton, AB, T6G 2V4, Canada
Man Xiao, Mohamed El-Attar, Marek Reformat & James Miller

Authors

Man Xiao
View author publications
You can also search for this author in PubMed Google Scholar
Mohamed El-Attar
View author publications
You can also search for this author in PubMed Google Scholar
Marek Reformat
View author publications
You can also search for this author in PubMed Google Scholar
James Miller
View author publications
You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to James Miller.

Appendix

In this section we provide a skeletal outline of the programs used in this study. These outlines provide all of the decision information utilized by the algorithms, but have the sequential code deleted for the sake of brevity. In addition, a brief discussion of the translation of the program into goal-oriented optimization criteria is presented for each program.

1. Hex_dec conversion

... ...
while ((c=getchar())!=‘\n’) //1
... ...
if(c >=‘0’ && c <=‘9’||c >=‘a’ && c<=‘f’||c>=‘A’ && c<=‘F’) //2
... ...
else
... ...
if (i<=MAX) //3
... ...
else printf(“\nMaximum 7 digits of hex number”);
... ...
for (j=0;s[j]!=‘\n’; j++) //4
if (s[j]>=‘0’ && s[j]<=‘9’) //5
... ...
if (s[j]>=‘a’ && s[j]<=‘f’) //6
... ...
if (s[j]>=‘A’ && s[j]<=‘F’) //7
... ...
... ...

There are 7 decisions in Hex_dec. 36 test requirements need to be satisfied to obtain complete condition-decision coverage. Thus, a maximum of 36 objective functions are generated to allow the test generators to calculate the value of objective function ${\Im }$(x). For example, consider the following fragment of code:

if(c>=‘0’ && c<=‘9’||c>=‘a’ && c<=‘f’||c>=‘A’ && c<=‘F’)
{...
}

There are 6 conditions and one decision that need to be evaluated independently. According to the requirement of condition-decision coverage, 14 test requirements should be satisfied. Thus, 14 objective functions are generated as below.

To ensure the decision

If(c>=‘0’ && c<=‘9’||c>=‘a’ && c<=‘f’||c>=‘A’ && c<=‘F’)

takes the value ”true”, the following function is built:

$$ {\text{True}}:{\Im }{\left( x \right)} = \left\{ {\begin{array}{*{20}c} {0}{{\prime 9\prime \geqslant c \geqslant \prime 0\prime ,\prime f\prime \geqslant c \geqslant \prime a,\prime F\prime \geqslant c \geqslant \prime A\prime }}{{{\text{reached}}}} \\ {{\operatorname{minimum} {\left( {{\Im }1,\,{\Im }2,\,{\Im }3} \right)}}}{{{\text{otherwise}}}}{{{\text{reached}}}} \\ {p}{{}}{{{\text{unreached}}}} \\ \end{array} } \right. $$

$$ {\text{where}}\;{\Im }1{\left( x \right)}\left\{ {\begin{array}{*{20}c} {{c - \prime 9\prime }}{{c > \prime 9\prime }} \\ {{\prime 0\prime - c}}{{c < \prime 0\prime }} \\ \end{array} } \right. $$

$$ {\Im }2{\left( x \right)}\left\{ {\begin{array}{*{20}c} {{c - \prime F\prime }}{{c > \prime F\prime }} \\ {{\prime A\prime - c}}{{c < \prime A\prime }} \\ \end{array} } \right. $$

$$ {\Im }3{\left( x \right)}\left\{ {\begin{array}{*{20}c} {{c - \prime f\prime }}{{c > \prime f\prime }} \\ {{\prime a\prime - c}}{{c < \prime a\prime }} \\ \end{array} } \right. $$

To ensure the decision

If(c>=‘0’ && c<=‘9’|| c>=‘a’ && c<=‘f’|| c>=‘A’ && c<=‘F’)

takes value “false”, the following function is built:

$$ {\Im }{\left( x \right)} = \left\{ {\begin{array}{*{20}c} {0}{{{\text{unsatisfied}}{\left( {\prime 9\prime \geqslant c \geqslant \prime 0\prime ,\prime f\prime \geqslant c \geqslant \prime a,\prime F\prime \geqslant c \geqslant \prime A\prime } \right)}}}{{{\text{reached}}}} \\ {{{\Im }1 + {\Im }2 + {\Im }3}}{{{\text{otherwise}}}}{{{\text{reached}}}} \\ {p}{{}}{{{\text{unreached}}}} \\ \end{array} } \right. $$

$$ \begin{array}{*{20}c} {{\text{where}}\;{\Im }1{\left( x \right)} = \left\{ {\begin{array}{*{20}c} {{c - \prime 9\prime }}{{c > \prime 9\prime }} \\ {{\prime 0\prime - c}}{{c < \prime 0\prime }} \\ \end{array} } \right.} \\ {{\Im }2{\left( x \right)} = \left\{ {\begin{array}{*{20}c} {{c - \prime F\prime }}{{c > \prime F\prime }} \\ {{\prime A\prime - c}}{{c < \prime A\prime }} \\ \end{array} } \right.} \\ {{\Im }3{\left( x \right)} = \left\{ {\begin{array}{*{20}c} {{c - \prime f\prime }}{{c > \prime f\prime }} \\ {{\prime a\prime - c}}{{c < \prime a\prime }} \\ \end{array} } \right.} \\ \end{array} $$

In a similar way, functions representing other 12 cases are built:

c>=‘0’

$$ {\text{True}}:{\Im }{\left( x \right)}\left\{ {\begin{array}{*{20}c} {0}{{c \geqslant 0}}{{{\text{reached}}}} \\ {{\prime 0\prime - c}}{{c \leqslant 0}}{{{\text{reached}}}} \\ {p}{{}}{{{\text{unreached}}}} \\ \end{array} } \right. $$

$$ {\text{False}}:{\Im }{\left( x \right)}\left\{ {\begin{array}{*{20}c} {0}{{c < 0}}{{{\text{reached}}}} \\ {{c - \prime 0\prime + 1}}{{c \geqslant 0}}{{{\text{reached}}}} \\ {p}{{}}{{{\text{unreached}}}} \\ \end{array} } \right. $$

c<=‘9’

$$ {\text{True}}:{\Im }{\left( x \right)} = \left\{ {\begin{array}{*{20}c} {0}{{c \leqslant \prime 9\prime }}{{{\text{reached}}}} \\ {{c - \prime 9\prime }}{{c > \prime 9\prime }}{{{\text{reached}}}} \\ {p}{{}}{{{\text{unreached}}}} \\ \end{array} } \right. $$

$$ {\text{False}}:{\Im }{\left( x \right)} = \left\{ {\begin{array}{*{20}c} {0}{{c > \prime 9\prime }}{{{\text{reached}}}} \\ {{\prime 9\prime - c + 1}}{{c \leqslant \prime 9\prime }}{{{\text{reached}}}} \\ {p}{{}}{{{\text{unreached}}}} \\ \end{array} } \right. $$

c>=‘a’

$$ {\text{True}}:{\Im }{\left( x \right)} = \left\{ {\begin{array}{*{20}c} {0}{{c \geqslant \prime a\prime }}{{{\text{reached}}}} \\ {{\prime a\prime - c}}{{c \leqslant \prime a\prime }}{{{\text{reached}}}} \\ {p}{{}}{{{\text{unreached}}}} \\ \end{array} } \right. $$

$$ {\text{False}}:{\Im }{\left( x \right)} = \left\{ {\begin{array}{*{20}c} {0}{{c < \prime a\prime }}{{{\text{reached}}}} \\ {{c - \prime a\prime + 1}}{{c \geqslant \prime a\prime }}{{{\text{reached}}}} \\ {p}{{}}{{{\text{unreached}}}} \\ \end{array} } \right. $$

c<=‘f’

$$ {\text{True}}:{\Im }{\left( x \right)} = \left\{ {\begin{array}{*{20}c} {0}{{c \leqslant \prime f\prime }}{{{\text{reached}}}} \\ {{c - \prime f\prime }}{{c > \prime f\prime }}{{{\text{reached}}}} \\ {p}{{}}{{{\text{unreached}}}} \\ \end{array} } \right. $$

$$ {\text{False}}:{\Im }{\left( x \right)} = \left\{ {\begin{array}{*{20}c} {0}{{c > \prime f\prime }}{{{\text{reached}}}} \\ {{\prime f\prime - c + 1}}{{c \leqslant \prime f\prime }}{{{\text{reached}}}} \\ {p}{{}}{{{\text{unreached}}}} \\ \end{array} } \right. $$

c>=‘A’

$$ {\text{True}}:{\Im }{\left( x \right)} = \left\{ {\begin{array}{*{20}c} {0}{{c \geqslant \prime A\prime }}{{{\text{reached}}}} \\ {{\prime A\prime - c}}{{c \leqslant \prime A\prime }}{{{\text{reached}}}} \\ {p}{{}}{{{\text{unreached}}}} \\ \end{array} } \right. $$

$$ {\text{False}}:{\Im }{\left( x \right)} = \left\{ {\begin{array}{*{20}c} {0}{{c < \prime A\prime }}{{{\text{reached}}}} \\ {{c - \prime A\prime + 1}}{{c \geqslant \prime A\prime }}{{{\text{reached}}}} \\ {p}{{}}{{{\text{unreached}}}} \\ \end{array} } \right. $$

c<=‘F’

$$ {\text{True}}:{\Im }{\left( x \right)} = \left\{ {\begin{array}{*{20}c} {0}{{c \leqslant \prime F\prime }}{{{\text{reached}}}} \\ {{c - \prime F\prime }}{{c > \prime F\prime }}{{{\text{reached}}}} \\ {p}{{}}{{{\text{unreached}}}} \\ \end{array} } \right. $$

$$ {\text{False}}:{\Im }{\left( x \right)} = \left\{ {\begin{array}{*{20}c} {0}{{c > \prime F\prime }}{{{\text{reached}}}} \\ {{\prime F\prime - c + 1}}{{c \leqslant \prime F\prime }}{{{\text{reached}}}} \\ {p}{{}}{{{\text{unreached}}}} \\ \end{array} } \right. $$

2. Timeshuttle

6.2.1 Analysis of the source code

There are 40 decisions in the Timeshuttle program, which are identified below.

int main(void)
{
... ...
if (validInput) //1
{
... ...
}
else
{
... ...
}
... ...
}
void plannedTrip(Month mToday, int dToday, int yToday, Month m, int d, int y)
{
... ...
}
void randomTrip(Month mToday, int dToday, int yToday, Month m, int d, int y)
{
... ...
}
bool isLeapYear(int year)
{
... ...
}
int gregorianDay(Month m, int d, int y)
{
... ...
if (isValidDate(m, d, y)) //2
{
... ...
if (y==YEAR1) //3
{
... ...
}
else
{
... ...
for (int i=YEAR1+1; i<y; i++) //4
{
... ...
}
... ...
}
... ...
}
... ...
}
bool isValidDate(Month m, int d, int y)
{
... ...
if (m<JAN || m>DEC) //5
... ...
else if (d>daysInMonth(m, y) || d<1) //6
... ...
else if (y<YEAR1 || y>YEARMAX) //7
... ...
else if (y==YEAR1) //8
{if (m<OCT) //9
... ...
else if (m==OCT && d<15) //10
... ...
else
... ...
}else
... ...
}
int getYearDay(Month m, int d, int y)
{
... ...
for (Month mo=JAN; mo<m; mo=static_cast<Month>(mo+1)) //11
{
... ...
}
... ...
}
Weekday getWeekday(int gDay)
{
... ...
}
string dayName(Weekday w)
{
string name;
switch(w)
{
case SUN: //12
name = "Sunday"; break;
case MON: //13
name = "Monday"; break;
case TUE: //14
name = "Tuesday"; break;
case WED: //15
name = "Wednesday"; break;
case THU: //16
name = "Thursday"; break;
case FRI: //17
name = "Friday"; break;
case SAT: //18
name = "Saturday"; break;
}
... ...
}
string monthName(Month m)
{
... ...
switch(m)
{
case JAN: //19
name = "January"; break;
case FEB: //20
name = "February"; break;
case MAR: //21
name = "March"; break;
case APR: //22
name = "April"; break;
case MAY: //23
name = "May"; break;
case JUN: //24
name = "June"; break;
case JUL: //25
name = "July"; break;
case AUG: //26
name = "August"; break;
case SEP: //27
name = "September"; break;
case OCT: //28
name = "October"; break;
case NOV: //29
name = "November"; break;
case DEC: //30
name = "December"; break;
}
return name;
}
void getTodaysDate(Month& m, int& d, int& y)
{
... ...
}
void gDay2MDY(int gDay, Month& m, int& d, int& y)
{
... ...
if (gDay<= YEAR1DAYS) //31
{
... ...
if (gDay<= daysInMonth(OCT, y) −14) //32
... ...
else
{
... ...
}
}
else
{
... ...
while (gDay > daysInYear(y)) //33
{
... ...
}
}
while (gDay > daysInMonth(m, y)) //34
{
... ...
}
... ...
}
int daysInMonth(Month m, int y)
{
... ...
switch(m)
{
case JAN: case MAR: case MAY: case JUL: case AUG: case OCT: case DEC: //35
... ...
case APR: case JUN: case SEP: case NOV: //36
... ...
case FEB: //37
... ...
}
... ...
}
int daysInYear(int y)
{
... ...
}
int randInt(int a, int b)
{
... ...
}
void outputMessage(int days, Month mToday, int dToday, int yToday,
Weekday w, Month m, int d, int y)
{
... ...
if (days<0) //38
... ...
else
... ...
if (days<0) //39
... ...
else
... ...
}
bool getInputDate(Month&m, int& d, int& y)
{
... ...
if (!valid) //40
... ...
}

Compared to other programs used in this paper, this program has more functions and more complicated relationship between the input parameters and the variables that appear in the conditions that will be evaluated. Most of the decisions do not include multiple conditions but some nested decisions exist. For example, consider the following fragment:

if (m<JAN || m>DEC) //5
... ...
else if (d>daysInMonth(m, y) || d<1) //6
... ...
else if (y<YEAR1 || y>YEARMAX) //7
... ...
else if (y==YEAR1) //8
{
if (m<OCT) //9
... ...
else if (m==OCT && d<15) //10
... ...
else

In order to obtain a test case to satisfy the decision if (m<OCT), the test case also need to satisfy the condition if (m<JAN||m>DEC) and (y==YEAR1), and not satisfy (d>daysInMonth(m, y)||d<1) and (y<YEAR1||y>YEARMAX). Since the test data generators discussed in this paper does not involve any static analysis of the source code, the test data generators only rely on the instrumented code to guide the search. The generating of objective function of the decision if (m<OCT) is still straightforward. To take the “true” outcome of the decision if (m<OCT), the objective function is generated as below:

$$ {\Im }{\left( {x,\,y,\,z} \right)} = \left\{ {\begin{array}{*{20}c} {0}{{m < oct}}{{{\text{reached}}}} \\ {{m - oct + 1}}{{{\text{otherwise}}}}{{{\text{reached}}}} \\ {p}{{}}{{{\text{unreached}}}} \\ \end{array} } \right. $$

To take the “false” outcome of the decision if (m<OCT), the objective function is as below:

$$ {\Im }{\left( {x,\,y,\,z} \right)} = \left\{ {\begin{array}{*{20}c} {0}{{m \geqslant oct}}{{{\text{reached}}}} \\ {{oct - m}}{{{\text{otherwise}}}}{{{\text{reached}}}} \\ {p}{{}}{{{\text{unreached}}}} \\ \end{array} } \right. $$

where oct=10, and p is a significant value 2147483647.There are in total 84 test requirements that require to be satisfied to obtain full condition-decision coverage in the Timeshuttle program.

3. Perfect number program

... ...
if ((num % 7)==0) //1
... ...
if ((num % 11)==0) //2
... ...
if ((num % 13)==0) //3
... ...
else
... ...
switch(ok)
{
case 1: //4
... ...
case 2: //5
... ...
case 3: //6
... ...
default:
... ...
}
void myint::sum()
{
... ...
if (temp==0) //7
... ...
else
... ...
}
void myint::square()
{
... ...
}
void myint::prime()
{
for (i=1; i<(0.5 * num); i++) //8
{
if (i !=num && num !=1) //9
{
if ((num % i)==0) //10
... ...
}
else
;
}
if (ok>0) //11
... ...
else
... ...
}
void myint::perfect()
{
for (i=2; i<(0.5 * num); i++) //12
{
if (num % i==0) //13
... ...
}
if (perfectsum==num) //14
if num>100 //15
if num>1000 //16
... ...
else
... ...
}
int main()
{
... ...
}

There are 37 test requirements that should be satisfied to obtain complete condition-decision coverage of the perfect number program. The objective functions are generated in the similar way as in the Hex_dec and Timeshuttle programs. For example, for decision 16

if num>1000 //16

$$ {\text{True}}:{\Im }{\left( x \right)} = \left\{ {\begin{array}{*{20}c} {0}{{num > 1000}}{{{\text{reached}}}} \\ {{1000 - num + 1}}{{{\text{otherwise}}}}{{{\text{reached}}}} \\ {p}{{}}{{{\text{unreached}}}} \\ \end{array} } \right. $$

$$ {\text{False}}:{\Im }{\left( x \right)} = \left\{ {\begin{array}{*{20}c} {0}{{num \leqslant 1000}}{{{\text{reached}}}} \\ {{1num - 1000}}{{{\text{otherwise}}}}{{{\text{reached}}}} \\ {p}{{}}{{{\text{unreached}}}} \\ \end{array} } \right. $$

Where p is 2147483647.

4. Triangle classification program

int triangle (int i, int j, int k){
... ...
if ((i<=0) ||(j<=0) ||(k<=0)) //1
... ...
if (i==j) //2
... ...
if (i==k) //3
... ...
if (j==k) //4
... ...
if (tri==0) //5
{
if ((i+j<=k)||(i+k<=j)||(j+k<=i )) //6
... ...
else
... ...
}
if (tri>3) //7
... ...
else
if ((tri==1) && (i+j>k)) //8
... ...
else if ((tri==2) && (i+k>j)) //9...
...
else if ((tri==3) && (j+k>i)) //10
... ...
else
... ...
}
int main(){int a,b,c,t;
... ...
if (t==1) //11
{
... ...
}
else if (t==2) //12
{
... ...
}
else if (t==3){ //13
... ...
}
else if (t==4){ //14
... ...
}
... ...
}

To obtain complete condition-decision coverage, there are 51 test requirements to be satisfied. Consider the following code fragment:

else if ((tri==2) && (i+k>j)) //9

To ensure the condition tri==2 take value ”true”, the following function is built:

$$ {\text{True}}\;{\Im }1 = \left\{ {\begin{array}{*{20}c} {p}{{{\text{unreached}}}} \\ {\begin{aligned}p \times m \\0 \\ \end{aligned} }{\begin{aligned}{\text{reached}}tri \ne 2 \\{\text{reached}}tri = = 2 \\ \end{aligned} } \\ \end{array} } \right. $$

Execution of the instrumented code provides information about the values of tri, i, j and k allows the calculation of $ \mathfrak{J} $(x, y, z). Where p is a significant value and it is the maximum range of input space 2147483647 in the triangle program, m is a value between 0 and 1, which is 0.7 in our experiments.

To ensure the condition tri==2 take value ”false”, the following function is built:

tri==2

$$ {\Im }1 = \left\{ {\begin{array}{*{20}c} {0}{{tri \ne 2}}{{reached}} \\ {{p \times m}}{{tri = 2}}{{reached}} \\ {p}{{}}{{unreached}} \\ \end{array} } \right. $$

In the similar way, functions representing all of the other 4 cases are built:

(i+k>j)

$$ True\;{\Im }2 = \left\{ {\begin{array}{*{20}c} {0}{{i + k > j}}{{reached}} \\ {{1 + abs{\left( {i + k - j} \right)}}}{{i + k \leqslant j}}{{reached}} \\ {p}{{}}{{unreached}} \\ \end{array} } \right. $$

$$ False\;{\Im }2 = \left\{ {\begin{array}{*{20}c} {0}{{i + k \leqslant j}}{{reached}} \\ {{{\left( {i + k - j} \right)}}}{{i + k > j}}{{reached}} \\ {p}{{}}{{unreached}} \\ \end{array} } \right. $$

if(tri==2) && (i+k>j)

$$ True\;{\Im } = \left\{ {\begin{array}{*{20}l} {p \hfill}{{unreached} \hfill} \\ {{{\Im }1 + {\Im }2} \hfill}{{reached} \hfill} \\ \end{array} } \right. $$

$$ False\;{\Im } = \left\{ {\begin{array}{*{20}l} {p \hfill}{{unreached} \hfill} \\ {{minimum{\left( {{\Im }1,{\Im }2} \right)}} \hfill}{{reached} \hfill} \\ \end{array} } \right. $$

5. Rescue program

... ...
if(!(code>9999 && code<100000)) //1
{
... ...
}
else
{
... ...
... ...
if (!(sum%2==0)) //2
{
... ...
}
else
{
... ...
if(rescueDay<1 || rescueDay>7) //3
{
... ...
}
else
{
if(digit4==digit5) //4
{
... ...
}
else if(digit4>digit5) //5
{
... ...
}
else
{
... ...
}
if((rendezvousPt !=2) && (rendezvousPt !=7) && (rendezvousPt !=8)) //6
{
... ...
}
else
{
... ...
switch(rescueDay)
{
case 1: //7
... ...
case 2: //8
... ...
case 3: //9
... ...
case 4: //10
... ...
case 5: //11
... ...
case 6: //12
... ...
case 7: //13
... ...
default:
... ...
}
// end of switch
switch(rendezvousPt)
{
case 2: //14
... ...
case 7: //15
... ...
case 8: //16
... ...
default:
... ...
}
// end of switch
... ...
}
}
}
}
return 0;
}

There are 16 decision branches (which are identified in bold) in Rescue. According to the definition of condition-decision coverage, the test generators need to generate the test set which satisfy will 46 test requirements to obtain a complete coverage. The program is instrumented with additional code that reports the information needed to the test data generator to calculate the value of objective function $ \mathfrak{J} $(x). For example, the first branch in Rescue is

if(!(code>9999 && code<100000))

There are two conditions in it, code>9999 and code<100000. To obtain the complete condition-decision coverage, test data must make each condition take the true and false value, and exercise both the true and false branches of the decision. Thus, 6 test requirements need to be satisfied to obtain complete condition-decision coverage. These 6 test requirements and their corresponding objective functions are shown below.

To ensure the decision

if(!(code>9999 && code<100000))

takes value ”true”, the following function is built:

True :

$$ {\text{True}}:{\Im }{\left( x \right)} = \left\{ {\begin{array}{*{20}c} {0}{\begin{aligned}code \geqslant 100000 \\code \leqslant 9999 \\ \end{aligned} }{{{\text{reached}}}} \\ {{\operatorname{minimum} {\left( {code - 9999,100000 - code} \right)}}}{{9999 < code < 100000}}{{{\text{reached}}}} \\ {p}{{}}{{{\text{unreached}}}} \\ \end{array} } \right. $$

Where p is a significant value 2147483647.

To ensure the decision

if(!(code>9999 && code<100000))

takes value ”false”, the following function is built:

$$ {\text{False}}:{\Im }{\left( x \right)} = \left\{ {\begin{array}{*{20}c} {0}{{9999 < code < 100000}} \\ {{9999 - code + 1}}{{code \leqslant 9999}} \\ {{code - 100000 + 1}}{{code \geqslant 100000}} \\ {p}{{{\text{unreached}}}} \\ \end{array} } \right. $$

In a similar way, functions representing all of the other 5 cases are built:

if(code>9999)

$$ {\text{True}}:{\Im }{\left( x \right)} = \left\{ {\begin{array}{*{20}c} {0}{{code > 9999}}{{{\text{reached}}}} \\ {{9999 - code + 1}}{{code \leqslant 9999}}{{{\text{reached}}}} \\ {p}{{}}{{{\text{unreached}}}} \\ \end{array} } \right. $$

$$ {\text{False}}:{\Im }{\left( x \right)} = \left\{ {\begin{array}{*{20}c} {0}{{code \leqslant 9999}}{{{\text{reached}}}} \\ {{code - 9999}}{{code > 9999}}{{{\text{reached}}}} \\ {p}{{}}{{{\text{unreached}}}} \\ \end{array} } \right. $$

if (code<100000)

$$ {\text{True}}:{\Im }{\left( x \right)} = \left\{ {\begin{array}{*{20}c} {0}{{code < 100000}}{{{\text{reached}}}} \\ {{code - 100000 + 1}}{{code \geqslant 100000}}{{{\text{reached}}}} \\ {p}{{}}{{{\text{unreached}}}} \\ \end{array} } \right. $$

$$ {\text{False}}:{\Im }{\left( x \right)} = \left\{ {\begin{array}{*{20}c} {0}{{code \geqslant 100000}}{{{\text{reached}}}} \\ {{100000 - code}}{{code < 100000}}{{{\text{reached}}}} \\ {p}{{}}{{{\text{unreached}}}} \\ \end{array} } \right. $$

For the fourth branch if(digit4==digit5), the way to construct the objective function is slightly different from the first branch. Two test requirements must be satisfied; the objective functions are shown below.

$$ {\text{True}}:{\Im }{\left( x \right)} = \left\{ {\begin{array}{*{20}c} {0}{{digit4 = digit5}} \\ {\begin{aligned}\operatorname{ abs} {\left( {digit4 - digit5} \right)} + p' \\p \\ \end{aligned} }{\begin{aligned}digit4 \ne digit5 \\{\text{unreached}} \\ \end{aligned} } \\ \end{array} } \right. $$

$$ {\text{False}}:{\Im }{\left( x \right)} = \left\{ {\begin{array}{*{20}c} {{p'}}{{digit4 = digit5}} \\ {\begin{aligned}0 \\p \\ \end{aligned} }{\begin{aligned}digit4 \ne digit5 \\{\text{unreached}} \\ \end{aligned} } \\ \end{array} } \right. $$

p and p′ are two significant values, where p′ << p. In the experiments, p=2147483647 and p=0.7p′. In the experiment on Rescue, 46 objective functions are generated in a similar way as discussed above.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Xiao, M., El-Attar, M., Reformat, M. et al. Empirical evaluation of optimization algorithms when used in goal-oriented automated test data generation techniques. Empir Software Eng 12, 183–239 (2007). https://doi.org/10.1007/s10664-006-9026-0

Download citation

Published: 08 November 2006
Issue Date: April 2007
DOI: https://doi.org/10.1007/s10664-006-9026-0

Keywords

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Empirical evaluation of optimization algorithms when used in goal-oriented automated test data generation techniques

Abstract

Access this article

Similar content being viewed by others

Sampling in software engineering research: a critical review and guidelines

Test case selection and prioritization using machine learning: a systematic literature review

A systematic review of software usability studies

Notes

References

Author information

Authors and Affiliations

Corresponding author

Appendix

Rights and permissions

About this article

Cite this article

Keywords

Navigation

Empirical evaluation of optimization algorithms when used in goal-oriented automated test data generation techniques

Abstract

Access this article

Similar content being viewed by others

Sampling in software engineering research: a critical review and guidelines

Test case selection and prioritization using machine learning: a systematic literature review

A systematic review of software usability studies

Notes

References

Author information

Authors and Affiliations

Corresponding author

Appendix

Appendix

Rights and permissions

About this article

Cite this article

Share this article

Keywords

Search

Navigation