Abstract
This paper investigates a measurement approach to support the implementation of Simulated Annealing (SA) applied to test generation. SA, like other metaheuristics, is a generic technique that must be tuned to the testing problem under consideration. Finding an adequate setting of SA parameters, that will offer good performance for the target problem, is known to be difficult. Our measurement approach is intended to guide the implementation choices to be made. It builds upon advanced research on how to characterize search problems and the dynamics of metaheuristic techniques applied to them. Central to this research is the concept of landscape. Existing measures of landscape have mainly been applied to combinatorial problems considered in complexity theory. We show that some of these measures can be useful for testing problems as well. The diameter and autocorrelation are retained to study the adequacy of alternative settings of SA parameters. A new measure, the Generation Rate of Better Solutions (GRBS), is introduced to monitor convergence of the search process and implement stopping criteria. The measurement approach is experimented on various case studies, and allows us to successfully revisit a problem issued from our previous work on testing control systems.
Article PDF
Similar content being viewed by others
References
Abdellatif-Kaddour O (2003) Property-oriented testing of control systems: stepwise construction of test scenarios generated by simulated annealing search. Doctoral Dissertation, Polytechnic National Institut of Toulouse, France, LAAS Report n° 03573, (In French)
Abdellatif-Kaddour O, Thevenod-Fosse P, Waeselynck H (2003a) Property-oriented testing: a strategy for exploring dangerous scenarios. Proc. ACM Symposium on Applied Computing (SAC'2003), Melbourne, USA, pp 1128–1134
Abdellatif-Kaddour O, Thevenod-Fosse P, Waeselynck H (2003b) An empirical investigation of simulated annealing applied to property-oriented testing. Proc. ACS/IEEE International Conference on Computer Systems and Applications (AICCSA'03), Tunis, Tunisia
Abrial J-R, Börger E, Langmaar H (eds) (1996) Formal methods for industrial applications: specifying and programming the steam boiler control. Springer, Berlin Heidelberg New York
Adamopoulos K, Harman M, Hierons RM (2004) How to overcome the equivalent mutant problem and achieve tailored selective mutation using co-evolution. Proc. Genetic and Evolutionary Computation Conference (GECCO 2004), LNCS 3103, Springer, Berlin Heidelberg New York, pp 1338–1349
Angel E, Zissimopoulos V (1998) Autocorrelation coefficient for the graph bipartitioning problem. Theor Comp Sci 191:229–243
Angel E, Zissimopoulos V (2000) On the classification of NP-complete problems in terms of their correlation coefficient. Discrete App Math 99(1–3):261–277
Belaidouni M (2001) Metaheuristics and search landscapes. Doctoral Dissertation, University of Angers, France (In French)
Belaidouni M, Hao JK (2000) Landscapes of the maximal constraint satisfaction problem. Proc. 4th European Conference on Artificial Evolution (EA'99), LNCS 1829, Springer, Berlin Heidelberg New York, pp 244–255
Belaidouni M, Hao JK (2002) SAT, Local search dynamics and density of states. Proc. 5th European Conference on Artificial Evolution, LNCS 2310, Springer, Berlin Heidelberg New York, pp 192–204
Clarke J, Dolado JJ, Harman M, Hierons R, Jones B, Lumkin M, Mitchell B, Mancoridis S, Rees K, Roper M, Shepperd M (2003) Reformulating software engineering as a search problem. IEE Proc Softw 150(3):161–175
Connolly DT (1990) An improved annealing scheme for the QAP. Eur J Oper Res 46(1):93–100
Eremeev AV, Reeves CR (2003) On confidence intervals for the number of local optima. Proc. EvoWorkshops 2003, LNCS 2611, Springer, Berlin Heidelberg New York, pp 224–235
Frank J, Cheeseman P, Stutz J (1997) When gravity fails: local search topology. J Artif Intell Res 7:249–281
Gross HG, Jones B, Eyres DE (2000) Structural performance measure of evolutionary testing applied to worst-case timing of real-time systems. IEE Proc Softw 147(2):161–175
Harman M, Jones BF (2001) Search-based software engineering. Inf Softw Technol 43(14):833–839
Holland J (1975) Adaptation in natural and artificial systems. University of Michigan
Hordijk W (1996) A measure of landscapes. Evol Comput 4(4):335–360
Jones T, Forrest S (1995) Fitness distance correlation as a measure of problem difficulty for genetic algorithms. Proc Int Conf on Genetic Algorithms (ICGA'03), pp 184–192
Jones BF, Sthamer H-H, Eyres DE (1996) Automatic structural testing using genetic algorithms. Softw Eng J 11(5):299–306
Kirkpatrick S, Gellat CD, Vecchi MP (1983) Optimization by simulated annealing. Science 220(4598):671–680
Lammermann F, Wegener J (2005) Test-goal-specific termination criteria for evolutionary white-box testing by means of software measures. Proc. 6th Metaheuristics International Conference (MIC'2005)
Lundy M, Mees AI (1986) Convergence of an annealing algorithm. Math Program 34(1):111–124
Maniezzo V, Dorigo M, Colorni A (1995) Algodesk: an experimental comparison of eight evolutionary heuristics applied to the quadratic assignment problem. Eur J Oper Res 81(1):188–204
McMinn P (2004) Search-based software test data generation: a survey. Softw Test Verif Reliab 14(2):105–156
Merz P, Freisleben B (2000) Fitness landscape analysis and memetic algorithms for the quadratic assignment problem. IEEE Trans Evol Comput 4(4):337–352
Michael CC, McGraw G, Schatz MA (2001) Generating software test data by evolution. IEEE Trans Softw Eng 27(12):1085–1110
Nugent CE, Vollman TE, Ruml J (1968) An experimental comparison of techniques for the assignment of facilities to locations. Oper Res 16:150–173
O'Sullivan M, Vössner S, Wegener J (1998) Testing temporal correctness of real-time systems—a new approach using genetic algorithms and cluster analysis. Proc. 6th European Conference on Software Testing, Analysis & Review (EuroSTAR 1998)
Pargas RP, Harrold M-J, Peck RR (1999) Test data generation using genetic algorithms. Softw Test Verif Reliab 9(4):263–282
Rayward-Smith VJ, Osman IH, Reeves CR, Smith GD (1996) Modern heuristic search methods. Wiley
Rosé H, Ebeling W, Asselmeyer T (1996) The density of states—a measure of the difficulty of optimisation problems. Proc. Parallel Problem Solving from Nature (PPSN IV), LNCS 1141, Springer, Berlin Heidelberg New York, pp 208–217
Schultz AC, Grefenstette JJ, De Jong KA (1995) Learning to break things: adaptative testing of intelligent controllers. Handbook on evolutionary computation, chapter G3.5. IOP and Oxford University Press
Stadler PF, Schnabl W (1992) The landscape of the traveling salesman problem. Phys Lett, A 161(4):337–344
Taillard ED (1991) Robust tabu search for the quadratic assignment problem. Parallel Comput 17(4&5):443–455
Tracey N (2000) A search-based automated test data generation framework for safety-critical software. PhD Dissertation, University of York, UK
Tracey N, Clark J, Mander K, McDermid J (1998a) An automated framework for structural test-data generation. Proc. 13th IEEE Conference on Automated Software Engineering (ASE), Hawaii, USA, pp 285–288
Tracey N, Clark J, Mander K (1998b) Automated program flaw finding using simulated annealing. Proc. ACM Int. Symp. on Software Testing and Analysis (ISSTA’98), Clearwater Beach, Florida, USA, pp 73–81
Tracey N, Clark J, Mander K, McDermid J (2000) Automated test-data generation for exception conditions. Softw Pract Exp 30(1):61–79
Vanneschi L, Tomassini M, Collard P, Clergue M (2003) Fitness distance correlation in structural mutation genetic programming. Proc. Europ. Conf. on Genetic Programming (EuroGP'03), LNCS 2610, Springer, Berlin Heidelberg New York, pp 455–464
Wegener J, Buehler O (2004) Evaluation of different fitness functions for the evolutionary testing of an autonomous parking system. Proc. Genetic and Evolutionary Computation Conference (GECCO-2004), LNCS 3103, Springer, Berlin Heidelberg New York, pp 1400–1412
Wegener J, Sthamer HH, Jones BF, Eyres DE (1997) Testing real-time systems using genetic algorithms. Softw Qual J 6(2):127–135
Wegener J, Buhr K, Pohlheim H (2002) Automatic test data generation for structural testing of embedded software systems by evolutionary testing. Proc. Genetic and Evolutionary Computation Conference (GECCO-2002), New York, USA, pp 1233–1240
Weinberger E (1990) Correlated and uncorrelated landscapes and how to tell the difference. Biol Cybern 63:325–336
Yokoo M (1997) Why adding more constraints makes a problem easier for hill-climbing algorithms: analyzing landscapes of CSPs. Proc. Int. Conf. on Principles and Practice of Constraint Programming (CP'97), LNCS 1330, Springer, Berlin Heidelberg New York, pp 356–370
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Waeselynck, H., Thévenod-Fosse, P. & Abdellatif-Kaddour, O. Simulated annealing applied to test generation: landscape characterization and stopping criteria. Empir Software Eng 12, 35–63 (2007). https://doi.org/10.1007/s10664-006-7551-5
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10664-006-7551-5