research-article

Achieving Effective Test Suites for Reactive Systems using Specification Mining and Test Suite Reduction Techniques

Authors:

Padmanabhan Krishnan,

R. VenkateshAuthors Info & Claims

ACM SIGSOFT Software Engineering Notes, Volume 40, Issue 1

Pages 1 - 8

https://doi.org/10.1145/2693208.2693226

Published: 06 February 2015 Publication History

Abstract

Failures in reactive embedded systems are often unacceptable. Moreover, effective testing of such systems to detect potential critical failures is a difficult task.

We present an automated black box test suite generation technique for reactive systems. The technique is based on dynamic mining of specifications, in form of a finite state machine (FSM), from initial runs. The set of test cases thus produced contain several redundant test cases, many of which are eliminated by a simple greedy test suite reduction algorithm to give the final test suite. The effectiveness of tests generated by our technique was evaluated using five case studies from the reactive embedded domain. Results indicate that a test suite generated by our technique is promising in terms of effectiveness and scalability. While the test suite reduction algorithm removes redundant test cases, the change in effectiveness of test suites due to this reduction is examined in the experimentation.

We present our specification mining based test suite generation technique, the test suite reduction technique and results on industrial case studies.

References

[1]

GCOV -- A Structural Code Coverage Measurement Tool.

[2]

F. Aarts, B. Jonsson, and J. Uijen. Generating Models of Infinite-State Communication Protocols using Regular Inference with Abstraction. In Proceedings of the 22nd IFIP WG 6.1 International Conference on Testing Software and Systems, pages 188--204. Springer-Verlag, 2010.

Digital Library

[3]

P. Bokil, P. Darke, U. Shrotri, and R. Venkatesh. Automatic Test Data Generation for C Programs. In Proceedings of the Third IEEE International Conference on Secure Software Integration and Reliability Improvement (SSIRI), pages 359--368. IEEE Computer Society, 2009.

Digital Library

[4]

A. Cimatti, A. Griggio, A. Micheli, I. Narasamdya, and M. Roveri. Kratos benchmarks. URL: https://es.fbk.eu/tools/kratos/index.php?n=Main.Benchmarks.

[5]

P. Cousot and R. Cousot. Verification of Embedded Software: Problems and Perspectives. In Proceedings of the 1st International Workshop on Embedded Software (EMSOFT), LNCS 2211, pages 97--113. Springer-Verlag, 2001.

Digital Library

[6]

V. Dallmeier, N. Knopp, C. Mallon, S. Hack, and A. Zeller. Generating Test Cases for Specification Mining. In Proceedings of the 19th International Symposium on Software Testing and Analysis (ISSTA), pages 85--96. ACM, 2010.

Digital Library

[7]

V. Dallmeier, C. Lindig, A. Wasylkowski, and A. Zeller. Mining object behavior with ADABU. In Proceedings of the 2006 International Workshop on Dynamic Systems Analysis, WODA '06, pages 17--24, New York, NY, USA, 2006. ACM.

Digital Library

[8]

G. de Caso, V. Braberman, D. Garbervetsky, and S. Uchitel. Program Abstractions for Behaviour Validation. In Proceedings of the 33rd International Conference on Software Engineering, pages 381--390. ACM, 2011.

Digital Library

[9]

M. Gabel and Z. Su. Symbolic mining of temporal specifications. In Proceedings of the 30th International Conference on Software Engineering, pages 51--60. ACM, 2008.

Digital Library

[10]

C. Ghezzi, A. Mocci, and M. Monga. Synthesizing intensional behavior models by graph transformation. In Proceedings of the 31st International Conference on Software Engineering, ICSE '09, pages 430--440, Washington, DC, USA, 2009. IEEE Computer Society.

Digital Library

[11]

P. Godefroid and A. Taly. Automated synthesis of symbolic instruction encodings from I/O samples. In Proceedings of the 33rd ACM SIGPLAN Conference on Programming Language Design and Implementation, pages 441--452. ACM, 2012.

Digital Library

[12]

D. Hamlet. When Only Random Testing Will Do. In Proceedings of the 1st International Workshop on Random testing, pages 1--9. ACM, 2006.

Digital Library

[13]

G. J. Holzmann and M. H. Smith. A Practical Method for Verifying Event-driven Software. In Proceedings of the 21st International Conference on Software Engineering (ICSE), pages 597--607. ACM, 1999.

Digital Library

[14]

A. Jääskeläinen, A. Kervinen, M. Katara, A. Valmari, and H. Virtanen. Synthesizing Test Models from Test Cases. In Proceedings of the 4th International Haifa Verification Conference on Hardware and Software: Verification and Testing, pages 179--193. Springer-Verlag, 2009.

Digital Library

[15]

H. Kelly J., V. Dan S., C. John J., and R. Leanna K. A Practical Tutorial on Modified Condition/Decision Coverage. Technical report, NASA Langley Center, 2001.

Digital Library

[16]

P. Krishnan, R. Venkatesh, P. Bokil, T. Muske, and V. Suman. Effectiveness of random testing of embedded systems. In Proceedings of the 45th Hawaii International Conference on System Sciences (HICSS), pages 5556--5563. IEEE Computer Society, 2012.

Digital Library

[17]

K. Lakhotia, P. McMinn, and M. Harman. Automated Test Data Generation for Coverage: Haven't We Solved This Problem Yet? In Proceedings of the 2009 Testing: Academic and Industrial Conference -- Practice and Research Techniques, pages 95--104. IEEE Computer Society, 2009.

Digital Library

[18]

C. Le Goues and W. Weimer. Measuring code quality to improve specification mining. IEEE Transactions in Software Engineering, 38(1):175--190, Jan. 2012.

Digital Library

[19]

D. Lo and S.-C. Khoo. Smartic: Towards building an accurate, robust and scalable specification miner. In Proceedings of the 14th ACM SIGSOFT International Symposium on Foundations of Software Engineering, SIGSOFT '06/FSE-14, pages 265--275, New York, NY, USA, 2006. ACM.

Digital Library

[20]

D. Lo, L. Mariani, and M. Santoro. Learning extended FSA from software: An empirical assessment. Journal in System Software, 85(9):2063--2076, Sept. 2012.

Digital Library

[21]

G. J. Myers and C. Sandler. The Art of Software Testing. John Wiley & Sons, 2004.

Digital Library

[22]

M. Naik, H. Yang, G. Castelnuovo, and M. Sagiv. Abstractions from tests. In Proceedings of the 39th Annual ACM SIGPLAN-SIGACT Symposium on Principles of Programming Languages, pages 373--386. ACM, 2012.

Digital Library

[23]

A. J. Offutt. A Practical System for Mutation Testing: Help for the Common Programmer. In Proceedings of the 1994 International Conference on Test, pages 824--830. IEEE Computer Society, 1994.

Digital Library

[24]

J. Peleska, A. Honisch, F. Lapschies, H. Löding, H. Schmid, P. Smuda, E. Vorobev, and C. Zahlten. A real-world benchmark model for testing concurrent real-time systems in the automotive domain. In Proceedings of the 23rd IFIP WG 6.1 International Conference on Testing Software and Systems(ICTSS), pages 146--161. Springer-Verlag, 2011.

Digital Library

[25]

V. research survey. http://blog.vdcresearch.com/embedded sw/2011/06/2011-embedded-engineer-survey-results-programming-languagesused-to-develop-software.html. 2011.

[26]

G. Rothermel, M. J. Harrold, J. von Ronne, and C. Hong. Empirical studies of test-suite reduction. Journal of Software Testing, Verification, and Reliability, 12:219--249, 2002.

[27]

S. Shoham, E. Yahav, S. Fink, and M. Pistoia. Static Specification Mining using Automata-based Abstractions. In Proceedings of the 2007 International Symposium on Software Testing and Analysis, pages 174--184. ACM, 2007.

Digital Library

[28]

R. R. Suman and R. Mall. State model extraction of a software component by observing its behavior. SIGSOFT Software Engineering Notes, 34:1--7, January 2009.

Digital Library

[29]

T. E. Vos, F. F. Lindlar, B. Wilmes, A. Windisch, A. I. Baars, P. M. Kruse, H. Gross, and J. Wegener. Evolutionary functional black-box testing in an industrial setting. Software Quality Control, 21(2):259--288, June 2013.

Digital Library

[30]

A. Zeller. Specifications for Free. In Proceedings of the Third International Conference on NASA Formal methods, pages 2--12. Springer-Verlag, 2011.

Digital Library

[31]

S. Zhang, D. Saff, Y. Bu, and M. D. Ernst. Combined Static and Dynamic Automated Test Generation. In Proceedings of the 2011 International Symposium on Software Testing and Analysis(ISSTA), pages 353--363. ACM, 2011.

Digital Library

Cited By

Subashini BMala D(2022)An Effective Approach to Test Suite Reduction and Fault Detection Using Data Mining TechniquesResearch Anthology on Agile Software, Software Development, and Testing10.4018/978-1-6684-3702-5.ch056(1109-1138)Online publication date: 2022
https://doi.org/10.4018/978-1-6684-3702-5.ch056
Xue YLi Y(2020)Multi-objective Integer Programming Approaches for Solving the Multi-criteria Test-suite Minimization ProblemACM Transactions on Software Engineering and Methodology10.1145/339203129:3(1-50)Online publication date: 1-Jun-2020
https://dl.acm.org/doi/10.1145/3392031
Lin WZeng HGao HMiao HWang XQi L(2018)Test Sequence Reduction of Wireless Protocol Conformance Testing to Internet of ThingsSecurity and Communication Networks10.1155/2018/37236912018Online publication date: 1-Nov-2018
https://dl.acm.org/doi/10.1155/2018/3723691
Show More Cited By

Index Terms

Achieving Effective Test Suites for Reactive Systems using Specification Mining and Test Suite Reduction Techniques
1. Software and its engineering
  1. Software creation and management
    1. Software verification and validation
      1. Software defect analysis
        Software testing and debugging

Recommendations

Improving Fault Detection Capability by Selectively Retaining Test Cases during Test Suite Reduction

Software testing is a critical part of software development. As new test cases are generated over time due to software modifications, test suite sizes may grow significantly. Because of time and resource constraints for testing, test suite minimization ...
Empirically evaluating Greedy-based test suite reduction methods at different levels of test suite complexity

Test suite reduction is an important approach that decreases the cost of regression testing. A test suite reduction technique operates based on the relationship between the test cases in the regression test suite and the test requirements in the program ...
Improving the effectiveness of test suite reduction for user-session-based testing of web applications

Context: Test suite reduction is the problem of creating and executing a set of test cases that are smaller in size but equivalent in effectiveness to an original test suite. However, reduced suites can still be large and executing all the tests in a ...

Comments

Information & Contributors

Information

Published In

cover image ACM SIGSOFT Software Engineering Notes

ACM SIGSOFT Software Engineering Notes Volume 40, Issue 1

January 2015

237 pages

ISSN:0163-5948

DOI:10.1145/2693208

Editor:
Michael Wing

Issue’s Table of Contents

Copyright © 2015 Authors.

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 06 February 2015

Published in SIGSOFT Volume 40, Issue 1

Check for updates

Author Tags

Qualifiers

Research-article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

7
Total Citations
View Citations
130
Total Downloads

Downloads (Last 12 months)2
Downloads (Last 6 weeks)0

Reflects downloads up to 08 Mar 2025

Other Metrics

View Author Metrics

Citations

Cited By

Subashini BMala D(2022)An Effective Approach to Test Suite Reduction and Fault Detection Using Data Mining TechniquesResearch Anthology on Agile Software, Software Development, and Testing10.4018/978-1-6684-3702-5.ch056(1109-1138)Online publication date: 2022
https://doi.org/10.4018/978-1-6684-3702-5.ch056
Xue YLi Y(2020)Multi-objective Integer Programming Approaches for Solving the Multi-criteria Test-suite Minimization ProblemACM Transactions on Software Engineering and Methodology10.1145/339203129:3(1-50)Online publication date: 1-Jun-2020
https://dl.acm.org/doi/10.1145/3392031
Lin WZeng HGao HMiao HWang XQi L(2018)Test Sequence Reduction of Wireless Protocol Conformance Testing to Internet of ThingsSecurity and Communication Networks10.1155/2018/37236912018Online publication date: 1-Nov-2018
https://dl.acm.org/doi/10.1155/2018/3723691
Rehman Khan SLee SJavaid NAbdul W(2018)A Systematic Review on Test Suite Reduction: Approaches, Experiment’s Quality Evaluation, and GuidelinesIEEE Access10.1109/ACCESS.2018.28096006(11816-11841)Online publication date: 2018
https://doi.org/10.1109/ACCESS.2018.2809600
Mala DSubashini B(2017)An Effective Approach to Test Suite Reduction and Fault Detection Using Data Mining TechniquesInternational Journal of Open Source Software and Processes10.4018/IJOSSP.20171001018:4(1-31)Online publication date: 1-Oct-2017
https://dl.acm.org/doi/10.4018/IJOSSP.2017100101
(2017)Test suite reduction based on traceability matrix with association rule mining techniqueInternational Journal of Information Systems and Change Management10.1504/IJISCM.2017.0898489:3(205-237)Online publication date: 1-Jan-2017
https://dl.acm.org/doi/10.1504/IJISCM.2017.089848
Vöst SWagner S(2016)Trace-based test selection to support continuous integration in the automotive industryProceedings of the International Workshop on Continuous Software Evolution and Delivery10.1145/2896941.2896951(34-40)Online publication date: 14-May-2016
https://dl.acm.org/doi/10.1145/2896941.2896951

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Issue’s Table of Contents