Article

Random testing of formal software models and induced coverage

Authors:

Dejan Desovski,

Bojan CukicAuthors Info & Claims

RT '06: Proceedings of the 1st international workshop on Random testing

Pages 20 - 27

https://doi.org/10.1145/1145735.1145739

Published: 17 July 2006 Publication History

Abstract

This paper presents a methodology for random testing of software models. Random testing tools can be used very effectively early in the modeling process, e.g., while writing formal requirements specification for a given system. In this phase users cannot know whether a correct operational model is being built or whether the properties that the model must satisfy are correctly identified and stated. So it is very useful to have tools to quickly identify errors in the operational model or in the properties, and make appropriate corrections. Using Lurch, our random testing tool for finite-state models, we evaluated the effectiveness of random model testing by detecting manually seeded errors in an SCR specification of a real-world personnel access control system. Having detected over 80% of seeded errors quickly, our results appear to be very encouraging. We further defined and measured test coverage metrics with the goal of understanding why some of the mutants were not detected. Coverage measures allowed us to understand the pitfalls of random testing of formal models, thus providing opportunities for future improvement.

References

[1]

Requirements Specification for Personnel Access Control System. National Security Agency, 2003.]]

[2]

P. Anderson, T. Reps, and T. Teitelbaum. Design and Implementation of a Fine-Grained Software Inspection Tool. IEEE Transactions on Software Engineering, 29(8), 2003.]]

Digital Library

[3]

J. H. Andrews, L. C. Briand, and Y. Labiche. Is Mutation an Appropriate Tool for Testing Experiments? In ICSE '05: Proceedings of the 27th International Conference on Software Engineering, 2005.]]

Digital Library

[4]

S. Antoy and R. Hamlet. Automatically Checking an Implementation against Its Formal Specification. IEEE Transactions on Software Engineering, 26(1), 2000.]]

Digital Library

[5]

M. Archer, C. Heitmeyer, and E. Riccobene. Proving Invariants of I/O Automata with TAME. Automated Software Engineering, 9(3), 2002.]]

Digital Library

[6]

R. Bharadwaj and S. Sims. Salsa: Combining Constraint Solvers with BDDs for Automatic Invariant Checking. In Proc. Tools and Algorithms for the Construction and Analysis of Systems, 2000.]]

Digital Library

[7]

A. Cimatti, E. Clarke, E. Giunchiglia, F. Giunchiglia, M. Pistore, M. Roveri, R. Sebastiani, and A. Tacchella. NuSMV Version 2: An OpenSource Tool for Symbolic Model Checking. In Proc. International Conference on Computer-Aided Verification, 2002.]]

Digital Library

[8]

E. Clarke, O. Grumberg, and D. Peled. Model Checking. MIT Press, 1999.]]

Digital Library

[9]

J. Cobleigh, L. Clarke, and L. Osterweil. FLAVERS: a Finite-State Verification Technique for Software Systems. IBM Systems Journal, 41(1), 2002.]]

Digital Library

[10]

A. Gargantini and C. Heitmeyer. Using Model Checking to Generate Tests from Requirements Specifications. In Proc. Joint 7th European Software Engineering Conference and 7th ACM Sigsoft International Symposium on Foundations of Software Engineering, 1999.]]

Digital Library

[11]

P. Godefroid. Software Model Checking: the Verisoft Approach. Formal Methods in System Design, 26(2), 2005.]]

Digital Library

[12]

P. Godefroid, N. Klarlund, and K. Sen. DART: Directed automated random testing. In PLDI '05: Proceedings of the 2005 ACM SIGPLAN conference on Programming Language Design and Implementation, 2005.]]

Digital Library

[13]

A. Groce and W. Visser. Heuristic Model Checking for Java Programs. In SPIN Workshop on Model Checking of Software, 2002.]]

Digital Library

[14]

R. Hamlet. Random Testing. In J. Maciniak, editor, Encyclopedia of Software Engineering. Wiley, 1994.]]

[15]

C. Heitmeyer, M. Archer, R. Bharadwaj, and R. Jeffords. Tools for Constructing Requirements Specifications: The SCR Toolset at the Age of Ten. International Journal of Computer Systems Science and Engineering, 20(1), 2005.]]

[16]

C. Heitmeyer, R. Jeffords, and B. Labaw. Automated Consistency Checking of Requirements Specifications. ACM Transactions on Software Engineering and Methodology, 5(3), 1996.]]

Digital Library

[17]

G. Holzmann. On-the-Fly, LTL Model Checking with SPIN. spinroot. com/spin/whatispin.html.]]

[18]

G. Holzmann. The SPIN Model Checker. Addison-Wesley, 2003.]]

[19]

E. I. Leonard and C. L. Heitmeyer. Program Synthesis from Formal Requirements Specifications Using APTS. Higher-Order and Symbolic Computation, (16), 2003.]]

Digital Library

[20]

K. McMillan. The SMV Model Checker. www-cad.eecs.berkeley.edu/~kenmcmil.]]

[21]

R. Motwani and P. Raghavan. Randomized Algorithms. Cambridge University Press, 1995.]]

Digital Library

[22]

D. Owen and T. Menzies. Lurch: A Lightweight Alternative to Model Checking. In Proc. 15th International Conference on Software Engineering and Knowledge Engineering, 2003.]]

[23]

D. J. Richardson, S. L. Aha, and T. O. O'Malley. Specification-based test oracles for reactive systems. In ICSE '92: Proceedings of the 14th International Conference on Software Engineering, 1992.]]

Digital Library

[24]

S. Sims, R. Cleaveland, K. Butts, and S. Ranville. Automated Validation of Software Models. In Proc. 16th International Conference on Automated Software Engineering, 2001.]]

Digital Library

[25]

E. Weyuker, T. Goradia, and A. Singh. Automatically Generating Test Data from a Boolean Specification. IEEE Transactions on Software Engineering, 20(5), 1994.]]

Digital Library

[26]

J. A. Whittaker and J. H. Poore. Markov Analysis of Software Specifications. ACM Transactions on Software Engineering and Methodology, 2(1), 1993.]]

Digital Library

Cited By

Krishnan PVenkatesh RBokil PMuske TSuman V(2012)Effectiveness of Random Testing of Embedded SystemsProceedings of the 2012 45th Hawaii International Conference on System Sciences10.1109/HICSS.2012.233(5556-5563)Online publication date: 4-Jan-2012
https://dl.acm.org/doi/10.1109/HICSS.2012.233
AboElFotoh MDean TMayor RNg JCouturier CKark AMartin P(2009)An empirical evaluation of a language-based security testing techniqueProceedings of the 2009 Conference of the Center for Advanced Studies on Collaborative Research10.1145/1723028.1723043(112-121)Online publication date: 2-Nov-2009
https://dl.acm.org/doi/10.1145/1723028.1723043
Zhang SDean TKnight S(2007)Lightweight State Based Mutation Testing for SecurityTesting: Academic and Industrial Conference Practice and Research Techniques - MUTATION (TAICPART-MUTATION 2007)10.1109/TAIC.PART.2007.31(223-232)Online publication date: Sep-2007
https://doi.org/10.1109/TAIC.PART.2007.31

Recommendations

Is branch coverage a good measure of testing effectiveness?
Empirical Software Engineering and Verification

Most approaches to testing use branch coverage to decide on the quality of a given test suite. The intuition is that covering branches relates directly to uncovering faults. The empirical study reported here applied random testing to 14 Eiffel classes ...
Comparison of adaptive random testing and random testing under various testing and debugging scenarios

Adaptive random testing is an enhancement of random testing. Previous studies on adaptive random testing assumed that once a failure is detected, testing is terminated and debugging is conducted immediately. It has been shown that adaptive random ...
Prioritizing random combinatorial test suites
SAC '17: Proceedings of the Symposium on Applied Computing

The behaviour of a system under test can be influenced by several factors, such as system configurations, user inputs, and so on. It has also been observed that many failures are caused by only a small number of factors. Combinatorial testing aims at ...

Comments

Information & Contributors

Information

Published In

cover image ACM Other conferences

RT '06: Proceedings of the 1st international workshop on Random testing

July 2006

84 pages

ISBN:159593457X

DOI:10.1145/1145735

Program Chairs:
Johannes Mayer
Ulm University, Germany
,
Robert Merkel
Swinburne University of Technology, Australia

Copyright © 2006 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 17 July 2006

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Article

Conference

ISSTA06

ISSTA06: International Symposium on Software Testing and Analysis 2006

July 20, 2006

Maine, Portland

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

3
Total Citations
View Citations
513
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 01 Mar 2025

Other Metrics

View Author Metrics

Citations

Cited By

Krishnan PVenkatesh RBokil PMuske TSuman V(2012)Effectiveness of Random Testing of Embedded SystemsProceedings of the 2012 45th Hawaii International Conference on System Sciences10.1109/HICSS.2012.233(5556-5563)Online publication date: 4-Jan-2012
https://dl.acm.org/doi/10.1109/HICSS.2012.233
AboElFotoh MDean TMayor RNg JCouturier CKark AMartin P(2009)An empirical evaluation of a language-based security testing techniqueProceedings of the 2009 Conference of the Center for Advanced Studies on Collaborative Research10.1145/1723028.1723043(112-121)Online publication date: 2-Nov-2009
https://dl.acm.org/doi/10.1145/1723028.1723043
Zhang SDean TKnight S(2007)Lightweight State Based Mutation Testing for SecurityTesting: Academic and Industrial Conference Practice and Research Techniques - MUTATION (TAICPART-MUTATION 2007)10.1109/TAIC.PART.2007.31(223-232)Online publication date: Sep-2007
https://doi.org/10.1109/TAIC.PART.2007.31

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten