research-article

Evolutionary algorithm for prioritized pairwise test data generation

Authors:

Peter M. Kruse,

Francisco Chicano,

Enrique AlbaAuthors Info & Claims

GECCO '12: Proceedings of the 14th annual conference on Genetic and evolutionary computation

Pages 1213 - 1220

https://doi.org/10.1145/2330163.2330331

Published: 07 July 2012 Publication History

Abstract

Combinatorial Interaction Testing (CIT) is a technique used to discover faults caused by parameter interactions in highly configurable systems. These systems tend to be large and exhaustive testing is generally impractical. Indeed, when the resources are limited, prioritization of test cases is a must. Important test cases are assigned a high priority and should be executed earlier. On the one hand, the prioritization of test cases may reveal faults in early stages of the testing phase. But, on the other hand the generation of minimal test suites that fulfill the demanded coverage criteria is an NP-hard problem. Therefore, search based approaches are required to find the (near) optimal test suites. In this work we present a novel evolutionary algorithm to deal with this problem. The experimental analysis compares five techniques on a set of benchmarks. It reveals that the evolutionary approach is clearly the best in our comparison. The presented algorithm can be integrated into a professional tool for CIT.

References

[1]

S. Amland. Risk-based testing: Risk analysis fundamentals and metrics for software testing including a financial application case study. Journal of Systems and Software, 53(3):287--295, 2000.

Digital Library

[2]

A. Baresel, D. W. Binkley, M. Harman, and B. Korel. Evolutionary testing in the presence of loop-assigned flags: A testability transformation approach. In ISSTA 2004, pages 108--118, 2004.

Digital Library

[3]

Bryce and Memon. Test suite prioritization by interaction coverage. In DOSTA'07, pages 1--7, New York, 2007.

Digital Library

[4]

R. C. Bryce and C. J. Colbourn. Prioritized interaction testing for pair-wise coverage with seeding and constraints. Information and Software Technology, 48(10):960--970, 2006.

[5]

R. C. Bryce and C. J. Colbourn. One-test-at-a-time heuristic search for interaction test suites. In GECCO '07, pages 1082--1089, New York, 2007.

Digital Library

[6]

D. Cohen and Shi. Interaction testing of highly-configurable systems in the presence of constraints. In ISSTA '07, New York,2007. ACM. 594071.

Digital Library

[7]

M. Cohen, J. Snyder, and G. Rothermel. Testing across configurations: implications for combinatorial testing. SIGSOFT Softw. Eng. Notes, 31:1--9, November 2006.

Digital Library

[8]

E. D1az, R. Blanco, and J. Tuya. Tabu search for automated loop coverage in software testing. In (ICKEDS'06), pages 229--234, Porto, 2006.

[9]

S. Elbaum, A. Malishevsky, and G. Rothermel. Test case prioritization: a family of empirical studies. IEEE Transactions on Software Engineering, 28(2):159 --182, feb 2002.

Digital Library

[10]

S. Elbaum, G. Rothermel, S. Kanduri, and A. G. Malishevsky. Selecting a cost-effective test case prioritization technique. Software Quality Control, 12:185--210, September 2004.

Digital Library

[11]

M. Grindal, J. Offutt, and S. F. Andler. Combination testing strategies: A survey. Software Testing, Verification, and Reliability, 15:167--199, 2005.

[12]

M. Grochtmann and K. Grimm. Classification trees for partition testing. Software Testing, Verification and Reliability, 3(2):63--82, 1993.

[13]

M. Harman. The current state and future of search based software engineering. In (ICSE/FOSE '07), pages 342--357, Minneapolis, May 2007.

Digital Library

[14]

M. Harman and B. F. Jones. Search-based software engineering. Information & Software Technology, 43(14):833--839, December 2001.

[15]

Y. X. Jones B, Sthamer H and E. D. The automatic generation of software test data sets using adaptive search techniques. In 3rd International Conference on Software Quality Management, pages 435--444, 1995.

[16]

B. Korel. Automated software test data generation. IEEE Trans. Softw. Eng., 16(8):870--879, 1990.

Digital Library

[17]

R. Kuhn, Y. Lei, and R. Kacker. Practical combinatorial testing: Beyond pairwise. IT Professional, 10:19--23, May 2008.

Digital Library

[18]

C. Y. Lee. Representation of switching circuits by binary-decision programs. Bell System Technical Journal, 38:985:999, July 1959.

[19]

E. Lehmann and J. Wegener. Test case design by means of the cte xl. In EuroSTAR 2000, Kopenhagen, Denmark, December 2000.

[20]

P. McMinn. Search-based software test data generation: a survey. Software Testing, Verification and Reliability, 14(2):105--156, June 2004.

Digital Library

[21]

W. Miller and D. L. Spooner. Automatic generation of floating-point test data. IEEE Trans. Software Eng., 2(3):223--226, 1976.

Digital Library

[22]

T. N. A search-based automated test-data generation framework for safety critical software. Master's thesis, PhD Thesis, University of York, 2000.

[23]

C. Nie and H. Leung. A survey of combinatorial testing. ACM Comput. Surv, 43(2):11, 2011.

Digital Library

[24]

T. J. Ostrand and M. J. Balcer. The category-partition method for specifying and generating fuctional tests. Commun. ACM, 31:676--686, June 1988.

Digital Library

[25]

X. Qu, M. Cohen, and K. Woolf. Combinatorial interaction regression testing: A study of test case generation and prioritization. In ICSM 2007, pages 255--264, oct. 2007.

[26]

E. Salecker, R. Reicherdt, and S. Glesner. Calculating prioritized interaction test sets with constraints using binary decision diagrams. ICSTW '11, pages 278--285, Washington, 2011.

Digital Library

[27]

G. H. Walton, J. H. Poore, and C. J. Trammell. Statistical testing of software based on a usage model. Softw. Pract. Exper., 25:97--108, January 1995.

Digital Library

[28]

Y. Zhan and J. A. Clark. The state problem for test generation in simulink. In GECCO'06:, pages 1941--1948. ACM Press, 2006.

Digital Library

Cited By

Sajib M(2024)Determining Factors and Correlation of Factors Influential for Student Engagement2024 15th International Conference on Computing Communication and Networking Technologies (ICCCNT)10.1109/ICCCNT61001.2024.10724770(1-6)Online publication date: 24-Jun-2024
https://doi.org/10.1109/ICCCNT61001.2024.10724770
Arrieta AValle PAgirre JSagardui G(2023)Some Seeds Are Strong: Seeding Strategies for Search-based Test Case SelectionACM Transactions on Software Engineering and Methodology10.1145/353218232:1(1-47)Online publication date: 13-Feb-2023
https://dl.acm.org/doi/10.1145/3532182
Singh ASinghrova ABhatia RRattan D(2023)A Systematic Literature Review on Test Case Prioritization TechniquesAgile Software Development10.1002/9781119896838.ch7(101-159)Online publication date: 8-Feb-2023
https://doi.org/10.1002/9781119896838.ch7
Show More Cited By

Index Terms

Evolutionary algorithm for prioritized pairwise test data generation
1. Computing methodologies
  1. Artificial intelligence
    1. Search methodologies
      1. Heuristic function construction
2. Software and its engineering
  1. Software creation and management
    1. Software verification and validation
      1. Software defect analysis
        Software testing and debugging

Recommendations

Prioritizing random combinatorial test suites
SAC '17: Proceedings of the Symposium on Applied Computing

The behaviour of a system under test can be influenced by several factors, such as system configurations, user inputs, and so on. It has also been observed that many failures are caused by only a small number of factors. Combinatorial testing aims at ...
The density algorithm for pairwise interaction testing: Research Articles

There are many published algorithms for generating interaction test suites for software testing, exemplified by AETG, IPO, TCG, TConfig, simulated annealing and other heuristic search, and combinatorial design techniques. Among these, greedy one-test-at-...
An orchestrated survey of methodologies for automated software test case generation

Test case generation is among the most labour-intensive tasks in software testing. It also has a strong impact on the effectiveness and efficiency of software testing. For these reasons, it has been one of the most active research topics in software ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

GECCO '12: Proceedings of the 14th annual conference on Genetic and evolutionary computation

July 2012

1396 pages

ISBN:9781450311779

DOI:10.1145/2330163

Editor:
Terence Soule
University of Idaho, USA
,
General Chair:
Jason H. Moore
Dartmouth College, USA

Copyright © 2012 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGEVO: ACM Special Interest Group on Genetic and Evolutionary Computation

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 07 July 2012

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

GECCO '12

Sponsor:

SIGEVO

GECCO '12: Genetic and Evolutionary Computation Conference

July 7 - 11, 2012

Pennsylvania, Philadelphia, USA

Acceptance Rates

Overall Acceptance Rate 1,669 of 4,410 submissions, 38%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

23
Total Citations
View Citations
251
Total Downloads

Downloads (Last 12 months)4
Downloads (Last 6 weeks)0

Reflects downloads up to 07 Mar 2025

Other Metrics

View Author Metrics

Citations

Cited By

Sajib M(2024)Determining Factors and Correlation of Factors Influential for Student Engagement2024 15th International Conference on Computing Communication and Networking Technologies (ICCCNT)10.1109/ICCCNT61001.2024.10724770(1-6)Online publication date: 24-Jun-2024
https://doi.org/10.1109/ICCCNT61001.2024.10724770
Arrieta AValle PAgirre JSagardui G(2023)Some Seeds Are Strong: Seeding Strategies for Search-based Test Case SelectionACM Transactions on Software Engineering and Methodology10.1145/353218232:1(1-47)Online publication date: 13-Feb-2023
https://dl.acm.org/doi/10.1145/3532182
Singh ASinghrova ABhatia RRattan D(2023)A Systematic Literature Review on Test Case Prioritization TechniquesAgile Software Development10.1002/9781119896838.ch7(101-159)Online publication date: 8-Feb-2023
https://doi.org/10.1002/9781119896838.ch7
Abdul Manan MAbang Jawawi DAhmad J(2021)A Systematic Literature Review on Test Case Prioritization in Combinatorial TestingProceedings of the 5th International Conference on Algorithms, Computing and Systems10.1145/3490700.3490710(55-61)Online publication date: 24-Sep-2021
https://dl.acm.org/doi/10.1145/3490700.3490710
Arrieta AAgirre JSagardui GCoello Coello C(2020)Seeding strategies for multi-objective test case selectionProceedings of the 2020 Genetic and Evolutionary Computation Conference10.1145/3377930.3389810(1222-1231)Online publication date: 25-Jun-2020
https://dl.acm.org/doi/10.1145/3377930.3389810
(2018)An automation approach for architecture discovery in software design using genetic algorithmInternational Journal of Computational Science and Engineering10.5555/3292834.329283817:4(390-397)Online publication date: 1-Jan-2018
https://dl.acm.org/doi/10.5555/3292834.3292838
Satish PNikhil PRangarajan K(2018)A Test Prioritization Algorithm That Cares for "Don't Care" Values and Higher Order Combinatorial CoverageACM SIGSOFT Software Engineering Notes10.1145/3149485.314951042:4(1-9)Online publication date: 11-Jan-2018
https://dl.acm.org/doi/10.1145/3149485.3149510
Sabbaghi AKeyvanpour M(2017)A novel approach for combinatorial test case generation using multi objective optimization2017 7th International Conference on Computer and Knowledge Engineering (ICCKE)10.1109/ICCKE.2017.8167914(411-418)Online publication date: Oct-2017
https://doi.org/10.1109/ICCKE.2017.8167914
Aggarwal MSabharwal S(2017)Combinatorial Test Set Prioritization Using Data Flow TechniquesArabian Journal for Science and Engineering10.1007/s13369-017-2631-y43:2(483-497)Online publication date: 15-Jun-2017
https://doi.org/10.1007/s13369-017-2631-y
Sánchez ASegura SParejo JRuiz-Cortés A(2017)Variability testing in the wildSoftware and Systems Modeling (SoSyM)10.1007/s10270-015-0459-z16:1(173-194)Online publication date: 1-Feb-2017
https://dl.acm.org/doi/10.1007/s10270-015-0459-z
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten