skip to main content
10.1145/2001420.2001450acmconferencesArticle/Chapter ViewAbstractPublication PagesisstaConference Proceedingsconference-collections
research-article

Feedback driven adaptive combinatorial testing

Published: 17 July 2011 Publication History

Abstract

The configuration spaces of modern software systems are too large to test exhaustively. Combinatorial interaction testing (CIT) approaches, such as covering arrays, systematically sample the configuration space and test only the selected configurations. The basic justification for CIT approaches is that they can cost-effectively exercise all system behaviors caused by the settings of t or fewer options. We conjecture, however, that in practice many such behaviors are not actually tested because of masking effects -- failures that perturb execution so as to prevent some behaviors from being exercised. In this work we present a feedback-driven, adaptive, combinatorial testing approach aimed at detecting and working around masking effects. At each iteration we detect potential masking effects, heuristically isolate their likely causes, and then generate new covering arrays that allow previously masked combinations to be tested in the subsequent iteration. We empirically assess the effectiveness of the proposed approach on two large widely used open source software systems. Our results suggest that masking effects do exist and that our approach provides a promising and efficient way to work around them.

References

[1]
Advanced Combinatorial Testing System (ACTS), 2010. http://csrc.nist.gov/groups/SNS/acts/documents/comparison-report.html.
[2]
R. Brownlie, J. Prowse, and M. S. Phadke. Robust testing of AT&T PMX/StarMAIL using OATS. AT&T Technical Journal, 71(3):41--7, 1992.
[3]
R. C. Bryce and C. J. Colbourn. Prioritized interaction testing for pair-wise coverage with seeding and constraints. Journal of Information and Software Technology, 48(10):960--970, 2006.
[4]
R. C. Bryce, A. Rajan, and M. P. Heimdahl. Interaction testing in model-based development: Effect on model-coverage. In Asia Pacific Software Engineering Conference, pages 259--268, 2006.
[5]
K. Burr and W. Young. Combinatorial test techniques: Table-based automation, test generation and code coverage. In Proceedings of the International Conference on Software Testing Analysis & Review, 1998.
[6]
A. Calvagna and A. Gargantini. A logic-based approach to combinatorial testing with constraints. In Tests and Proofs, Lecture Notes in Computer Science, 4966, pages 66--83, 2008.
[7]
D. M. Cohen, S. R. Dalal, M. L. Fredman, and G. C. Patton. The AETG system: an approach to testing based on combinatorial design. IEEE Transactions on Software Engineering, 23(7):437--44, 1997.
[8]
M. B. Cohen, M. B. Dwyer, and J. Shi. Constructing interaction test suites for highly-configurable systems in the presence of constraints: A greedy approach. IEEE Transactions on Software Engineering, 34(5):633--650, 2008.
[9]
S. R. Dalal, A. Jain, N. Karunanithi, J. M. Leaton, C. M. Lott, G. C. Patton, and B. M. Horowitz. Model-based testing in practice. In Proceedings of the International Conference on Software Engineering, pages 285--294, 1999.
[10]
S. Fouché, M. B. Cohen, and A. Porter. Incremental covering array failure characterization in large configuration spaces. In Proceedings of the International Symposium on Software Testing and Analysis, pages 177--188, New York, NY, USA, 2009. ACM.
[11]
D. Kuhn, D. R. Wallace, and A. M. Gallo. Software fault interactions and implications for software testing. IEEE Transactions on Software Engingeering, 30(6):418--421, 2004.
[12]
T. M. Mitchell. Machine Learning. McGraw Hill, 1997.
[13]
MySQL, 2006. http://www.mysql.com.
[14]
T. J. Ostrand and M. J. Balcer. The category-partition method for specifying and generating functional tests. Communications of the ACM, 31:678--686, 1988.
[15]
X. Qu, M. B. Cohen, and G. Rothermel. Configuration-aware regression testing: An empirical study of sampling and prioritization. In International Symposium on Software Testing and Analysis, pages 75--85, July 2008.
[16]
X. Qu, M. B. Cohen, and K. M. Woolf. Combinatorial interaction regression testing: A study of test case generation and prioritization. In International Conference on Software Maintenance, pages 255--264, October 2007.
[17]
I. H. Witten and E. Frank. Data Mining: Practical Machine Learning Tools and Techniques with Java Implementations. Morgan Kaufmann, 1999.
[18]
C. Yilmaz, M. B. Cohen, and A. Porter. Covering arrays for efficient fault characterization in complex configuration spaces. IEEE Transactions on Software Engineering, 31(1):20--34, Jan 2006.
[19]
X. Yuan, M. Cohen, and A. M. Memon. Covering array sampling of input event sequences for automated GUI testing. In International Conference on Automated Software Engineering, pages 405--408, 2007.

Cited By

View all

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences
ISSTA '11: Proceedings of the 2011 International Symposium on Software Testing and Analysis
July 2011
394 pages
ISBN:9781450305624
DOI:10.1145/2001420
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

In-Cooperation

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 17 July 2011

Permissions

Request permissions for this article.

Check for updates

Author Tags

  1. adaptive testing
  2. combinatorial testing
  3. covering arrays
  4. software quality assurance

Qualifiers

  • Research-article

Funding Sources

Conference

ISSTA '11
Sponsor:

Acceptance Rates

Overall Acceptance Rate 42 of 144 submissions, 29%

Upcoming Conference

ISSTA '25

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)18
  • Downloads (Last 6 weeks)2
Reflects downloads up to 14 Feb 2025

Other Metrics

Citations

Cited By

View all
  • (2022)Flexible Combinatorial Interaction TestingIEEE Transactions on Software Engineering10.1109/TSE.2020.301031748:3(1030-1066)Online publication date: 1-Mar-2022
  • (2022)CIT-dailyJournal of Systems and Software10.1016/j.jss.2022.111353190:COnline publication date: 1-Aug-2022
  • (2021)On the Testing of Statistical SoftwareJournal of Statistical Theory and Practice10.1007/s42519-021-00208-w15:4Online publication date: 25-Aug-2021
  • (2020)Understanding and discovering software configuration dependencies in cloud and datacenter systemsProceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering10.1145/3368089.3409727(362-374)Online publication date: 8-Nov-2020
  • (2020)An Interleaving Approach to Combinatorial Testing and Failure-Inducing Interaction IdentificationIEEE Transactions on Software Engineering10.1109/TSE.2018.286577246:6(584-615)Online publication date: 1-Jun-2020
  • (2020)Identifying Failure-Causing Schemas in the Presence of Multiple FaultsIEEE Transactions on Software Engineering10.1109/TSE.2018.284425946:2(141-162)Online publication date: 1-Feb-2020
  • (2020)Combinatorial Testing of Browsers’ Support for MultimediaIEEE Transactions on Reliability10.1109/TR.2020.297515069:4(1323-1340)Online publication date: Dec-2020
  • (2020)Coverage Guided Multiple Base Choice Testing2020 IEEE 20th International Conference on Software Quality, Reliability and Security Companion (QRS-C)10.1109/QRS-C51114.2020.00020(53-60)Online publication date: Dec-2020
  • (2020)A Comparison Infrastructure for Fault Characterization Algorithms2020 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW)10.1109/ICSTW50294.2020.00042(201-210)Online publication date: Oct-2020
  • (2020)How does combinatorial testing perform in the real world: an empirical studyEmpirical Software Engineering10.1007/s10664-019-09799-2Online publication date: 20-Apr-2020
  • Show More Cited By

View Options

Login options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Figures

Tables

Media

Share

Share

Share this Publication link

Share on social media