skip to main content
10.1145/1007512.1007519acmconferencesArticle/Chapter ViewAbstractPublication PagesisstaConference Proceedingsconference-collections
Article

Covering arrays for efficient fault characterization in complex configuration spaces

Published: 01 July 2004 Publication History

Abstract

Testing systems with large configurations spaces that change often is a challenging problem. The cost and complexity of QA explodes because often there isn't just one system, but a multitude of related systems. Bugs may appear in certain configurations, but not in others.The Skoll system and process has been developed to test these types of systems through distributed, continuous quality assurance, leveraging user resources around-the-world, around-the-clock. It has been shown to be effective in automatically characterizing configurations in which failures manifest. The derived information helps developers quickly narrow down the cause of failures which then improves turn around time for fixes. However, this method does not scale well. It requires one to exhaustively test each configuration in the configuration space.In this paper we examine an alternative approach. The idea is to systematically sample the configuration space, test only the selected configurations, and conduct fault characterization on the resulting data. The sampling approach we use is based on calculating a mathematical object called a covering array. We empirically assess the effect of using covering array derived test schedules on the resulting fault characterizations and provide guidelines to practitioners for their use.

References

[1]
L. Breiman, J. Freidman, R. Olshen, and C. Stone. Classification and Regression Trees. Wadsworth, Monterey, CA, 1984.
[2]
R. Brownlie, J. Prowse, and M. S. Padke. Robust testing of AT&T PMX/StarMAIL using OATS. AT&T Technical Journal, 71(3):41--7, 1992.
[3]
K. Burr and W. Young. Combinatorial test techniques: Table-based automation, test generation and code coverage. In Proc. of the Intl. Conf. on Software Testing Analysis & Review, 1998.
[4]
M. Chateauneuf and D. Kreher. On the state of strength-three covering arrays. Journal of Combinatorial Designs, 10(4):217--238, 2002.
[5]
D. M. Cohen, S. R. Dalal, M. L. Fredman, and G. C. Patton. The AETG system: an approach to testing based on combinatorial design. IEEE Transactions on Software Engineering, 23(7):437--44, 1997.
[6]
M. B. Cohen, C. J. Colbourn, P. B. Gibbons, and W. B. Mugridge. Constructing test suites for interaction testing. In Proc. of the Intl. Conf. on Software Engineering, (ICSE '03), pages 38--44, 2003.
[7]
S. R. Dalal, A. Jain, N. Karunanithi, J. M. Leaton, C. M. Lott, G. C. Patton, and B. M. Horowitz. Model-based testing in practice. In Proc. of the Intl. Conf. on Software Engineering, (ICSE), pages 285--294, 1999.
[8]
I. S. Dunietz, W. K. Ehrlich, B. D. Szablak, C. L. Mallows, and A. Iannino. Applying design of experiments to software testing. In Proc. of the Intl. Conf. on Software Engineering, (ICSE '97), pages 205--215, 1997.
[9]
D. Kuhn and M. Reilly. An investigation of the applicability of design of experiments to software testing. Proc. 27th Annual NASA Goddard/IEEE Software Engineering Workshop, pages 91--95, 2002.
[10]
R. Mandl. Orthogonal Latin squares: an application of experiment design to compiler testing. Communications of the ACM, 28(10):1054--1058, 1985.
[11]
A. Memon, A. Porter, C. Yilmaz, A. Nagarajan, D. C. Schmidt, and B. Natarajan. Skoll: Distributed continuous quality assurance. To be appear in Proc. of the Intl. Conf. on Software Engineering, (ICSE '04), 2004.
[12]
N. Sloane. Covering arrays and intersecting codes. Journal of Combinatorial Designs, 1(1):51--63, 1993.
[13]
I. H. Witten and E. Frank. Data Mining: Practical Machine Learning Tools and Techniques with Java Implementations. Morgan Kaufmann, 1999.

Cited By

View all
  • (2019)Charting a Course Through Uncertain Environments: SEA Uses Past Problems to Avoid Future Failures2019 IEEE 30th International Symposium on Software Reliability Engineering (ISSRE)10.1109/ISSRE.2019.00011(1-12)Online publication date: Oct-2019
  • (2019)Practical Fault Localization with Combinatorial Test Design2019 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW)10.1109/ICSTW.2019.00063(268-271)Online publication date: Apr-2019
  • (2018)Test-Algebra-Based Fault Location Analysis for the Concurrent Combinatorial TestingIEEE Transactions on Reliability10.1109/TR.2018.283344967:3(802-831)Online publication date: Sep-2018
  • Show More Cited By

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences
ISSTA '04: Proceedings of the 2004 ACM SIGSOFT international symposium on Software testing and analysis
July 2004
294 pages
ISBN:1581138202
DOI:10.1145/1007512
  • cover image ACM SIGSOFT Software Engineering Notes
    ACM SIGSOFT Software Engineering Notes  Volume 29, Issue 4
    July 2004
    284 pages
    ISSN:0163-5948
    DOI:10.1145/1013886
    Issue’s Table of Contents
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 01 July 2004

Permissions

Request permissions for this article.

Check for updates

Author Tags

  1. covering arrays
  2. distributed continuous quality assurance
  3. fault characterization
  4. software testing

Qualifiers

  • Article

Conference

ISSTA04
Sponsor:

Acceptance Rates

Overall Acceptance Rate 58 of 213 submissions, 27%

Upcoming Conference

ISSTA '25

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)9
  • Downloads (Last 6 weeks)1
Reflects downloads up to 14 Feb 2025

Other Metrics

Citations

Cited By

View all
  • (2019)Charting a Course Through Uncertain Environments: SEA Uses Past Problems to Avoid Future Failures2019 IEEE 30th International Symposium on Software Reliability Engineering (ISSRE)10.1109/ISSRE.2019.00011(1-12)Online publication date: Oct-2019
  • (2019)Practical Fault Localization with Combinatorial Test Design2019 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW)10.1109/ICSTW.2019.00063(268-271)Online publication date: Apr-2019
  • (2018)Test-Algebra-Based Fault Location Analysis for the Concurrent Combinatorial TestingIEEE Transactions on Reliability10.1109/TR.2018.283344967:3(802-831)Online publication date: Sep-2018
  • (2018)A tuned version of genetic algorithm for efficient test suite generation in interactive t-way testing strategyInformation and Software Technology10.1016/j.infsof.2017.10.00794:C(165-185)Online publication date: 1-Feb-2018
  • (2017)IntroductionCombinatorial Testing in Cloud Computing10.1007/978-981-10-4481-6_1(1-13)Online publication date: 26-Oct-2017
  • (2016)CoLUA: Automatically Predicting Configuration Bug Reports and Extracting Configuration Options2016 IEEE 27th International Symposium on Software Reliability Engineering (ISSRE)10.1109/ISSRE.2016.29(150-161)Online publication date: Oct-2016
  • (2015)An empirical study on configuration-related issues: investigating undeclared and unused identifiersACM SIGPLAN Notices10.1145/2936314.281420651:3(35-44)Online publication date: 26-Oct-2015
  • (2015)An empirical study on configuration-related issues: investigating undeclared and unused identifiersProceedings of the 2015 ACM SIGPLAN International Conference on Generative Programming: Concepts and Experiences10.1145/2814204.2814206(35-44)Online publication date: 26-Oct-2015
  • (2015)Detecting latent cross-platform API violationsProceedings of the 2015 IEEE 26th International Symposium on Software Reliability Engineering (ISSRE)10.1109/ISSRE.2015.7381841(484-495)Online publication date: 2-Nov-2015
  • (2015)Achievement of minimized combinatorial test suite for configuration-aware software functional testing using the Cuckoo Search algorithmInformation and Software Technology10.1016/j.infsof.2015.05.00566:C(13-29)Online publication date: 1-Oct-2015
  • Show More Cited By

View Options

Login options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Figures

Tables

Media

Share

Share

Share this Publication link

Share on social media