Skip to main content
SpringerLink
Log in

On the Reuse of TLM Mutation Analysis at RTL

  • Published:
Journal of Electronic Testing Aims and scope Submit manuscript

Abstract

Mutation analysis has gained consensus during the last decades as being an efficient technique for measuring the quality of SW testbench. More recently, it has been efficiently applied for validating testbenches of embedded system models implemented in hardware description language (HDL) at different abstraction levels (i.e., RTL, TLM). This article analyzes how mutation analysis performed at TLM can be reused at RTL and, in particular, how such a reuse can help designers in (i) optimizing the time spent for simulation at RTL, and (ii) improving the RTL testbench quality. Two alternatives of TLM mutation analysis reuse are presented and investigated for proposing an efficient methodology of RTL mutation analysis. Through experimental results, the proposed methodology is compared to the standard RTL mutation analysis to confirm its efficiency in terms of both simulation time and reached mutation coverage.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6

Similar content being viewed by others

References

  1. Abramovici M, Breuer M, Friedman A (1990) Digital systems testing and testable design. Computer Science Press, New York

    Google Scholar 

  2. Agrawal H, DeMillo RA, Hathaway B, Hsu W, Hsu W, Krauser EW, Martin RJ, Mathur AP, Spafford E (1989) Design of mutant operators for the C programming language. Purdue University, West Lafayette, Indiana, techreport SERC-TR-41-P

  3. Alexander RT, Bieman JM, Ghosh S, Bixia J (2002) Mutation of Java objects. In: Proc. of IEEE ISSRE, pp 341–351

  4. Belli F, Budnik C-J, Wong W-E (2006) Basic operations for generating behavioral mutants. In: Proc. of IEEE ISSRE, pp 10–18

  5. Bombieri N, Fummi F, Pravadelli G (2006) On the evaluation of transactor-based verification for reusing TLM assertions and testbenches at RTL. In: Proc. of ACM/IEEE conference on design, automation and test in Europe, DATE, pp 1007–1012

  6. Bombieri N, Fummi F, Pravadelli G (2008) A mutation model for the SystemC TLM 2.0 communication interfaces. In: Proc. of ACM/IEEE conference on design, automation and test in Europe, DATE, pp 396–401

  7. Bombieri N, Fummi F, Pravadelli G (2009) On the mutation analysis of SystemC TLM-2.0 standard. In: Proceedings of IEEE international workshop on microprocessor test and verification, MTV, pp 32–37

  8. Bombieri N, Fummi F, Pravadelli G, Hampton M, Letombe F (2009) Functional qualification of TLM verification. In: Proc. of the ACM/IEEE conference on design, automation and test in Europe, DATE, pp 190–195

  9. Bradbury JS, Cordy JR, Dingel J (2006) Mutation operators for concurrent Java (J2SE 5.0). In: Proc. of IEEE ISSRE workshops, pp 11–20

  10. Budd TA, Sayward FG (1977) Users guide to the Pilot mutation system. Yale University, New Haven, Connecticut, Technical report 114

  11. Budd TA, DeMillo RA, Lipton R, Sayward F (1980) Theoretical and empirical studies on using program mutation to test the functional correctness of programs. In: Proceedings of the 7th ACM SIGPLAN-SIGACT symposium on principles of programming languages. ACM, pp 220–233

  12. Cai L, Gajski DD (2003) Transaction level modeling: an overview. In: ACM/IEEE CODES+ISSS, pp 19–24

  13. Catapult C Synthesis (2010). Mentor graphics. http://www.mentor.com/esl/catapult/

  14. Choi BJ, DeMillo RA, Krauser EW, Martin RJ, Mathur AP, Offutt AJ, Pan H, Spafford EH (1989) The Mothra tool set. In: Proceedings of the 22nd annual Hawaii international conference on system sciences (HICSS), pp 275–284

  15. Coussy P, Gajski DD, Meredith M, Takach A (2009) An introduction to high-level synthesis. IEEE Des Test Comput 13(24):8–17

    Article  Google Scholar 

  16. Cynthesizer - TLM Synthesis (2008). Forte Design Systems. http://www.forteds.com/products/tlmsynthesis.asp

  17. DeMillo RA, Lipton RJ, Sayward FG (1978) Hints on test data selection: help for the practicing programmer. Computer 11(4):34–41

    Article  Google Scholar 

  18. Do H, Rothermel G (2006) On the use of mutation faults in empirical assessments of test case prioritization techniques. IEEE Trans Softw Eng 32(9):733–752

    Article  Google Scholar 

  19. Guarnieri V, Bombieri N, Pravadelli G, Fummi F, Hantson H, Raik J, Jenihhin M, Ubar R (2011) Mutation analysis for SystemC designs at TLM. In: Proc. of IEEE Latin-American test workshop (LATW), pp 27–30

  20. Guderlei R, Just R, Schneckenburger C, Schweiggert F (2008) Benchmarking testing strategies with tools from mutation analysis. In: International conference on software testing verification and validation workshop. IEEE, pp 360–364

  21. Hamlet RG (1977) Testing programs with the aid of a compiler. IEEE Treans Softw Eng 3(4):279–290

    Article  MathSciNet  MATH  Google Scholar 

  22. Hantson H, Raik J, Jenihhin M, Chepurov A, Ubar R, di Guglielmo G, Fummi F (2010) Mutation analysis with high-level decision diagrams. In: Test workshop (LATW), 2010 11th Latin American, pp 1–6

  23. Hyunsook D, Rothermel G (2006) On the use of mutation faults in empirical assessments of test case prioritization techniques. IEEE Trans Softw Eng 32(9):733–752

    Article  Google Scholar 

  24. Irvine SA et al (2007) Jumble Java byte code to measure the effectiveness of unit tests. In: Mutation testing workshop, pp 169–175

  25. Lipton R (1971) Fault diagnosis of computer programs. Carnegie Mellon University, Student report

  26. Lisherness P, Cheng K-T (Tim) (2010) SCEMIT: a SystemC error and mutation injection tool. In: Proc. of ACM/IEEE design automation conference (DAC), pp 228–233

  27. Lyu M-R, Zubin H, Sze SKS, Xia C (2003) An empirical study on testing and fault tolerance for software reliability engineering. In: Proc. of IEEE ISSRE, pp 119–130

  28. Ma Y-S, Offutt AJ, Kwon YR (2005) MuJava: an automated class mutation system: research articles. Softw Test Verif Reliab 15:97–133

    Article  Google Scholar 

  29. Offutt AJ, King KN (1987) A Fortran 77 interpreter for mutation analysis. In: Papers of the symposium on interpreters and interpretive techniques. SIGPLAN ’87, pp 177–188

  30. Offutt AJ, Rothermel G, Zapf C (1993) An experimental evaluation of selective mutation. In: Proceedings of the 15th international conference on software engineering (ICSE’93), Baltimore, Maryland, pp 100–107

  31. Offutt AJ, Lee A, Rothermel G, Untch R, Zapf C (1996) An experimental determination of sufficient mutant operators. ACM Trans Softw Eng Methodol 5(2):99–118

    Article  Google Scholar 

  32. Offutt AJ, Voas J (1996) Subsumption of condition coverage techniques by mutation testing. Department of Information and Software Systems Engineering, George Mason University, Technical report ISSE-TR-96-01

  33. Sen A (2009) Mutation operators for concurrent SystemC designs. In: Proc. of IEEE international workshop on microprocessor test and verification. MTV, pp 27–31

  34. Sen A, Abadir MS (2010) Coverage metrics for verification of concurrent SystemC designs using mutation testing. In: Proc. of IEEE international high-level design, validation, and test workshop, pp 75–81

  35. Tuya J, Suarez-Cabal MJ, De La Riva C (2006) SQLMutation: a tool to generate mutants of SQL database queries. In: Mutation testing workshop, pp 39–43

Download references

Acknowledgements

The work has been supported by FP7 DIAMOND project, CEBE Center of Excellence and Estonian SF grants 8478 and 9429.

Author information

Authors and Affiliations

  1. Dipartimento di Informatica, Università di Verona, Strada Le Grazie 15, 37134, Verona, Italy

    Valerio Guarnieri, Giuseppe Di Guglielmo, Nicola Bombieri, Graziano Pravadelli & Franco Fummi

  2. Department of Computer Engineering, Tallinn University of Technology, Raja 15, 12618, Tallinn, Estonia

    Hanno Hantson, Jaan Raik, Maksim Jenihhin & Raimund Ubar

Authors
  1. Valerio Guarnieri
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Giuseppe Di Guglielmo
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Nicola Bombieri
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Graziano Pravadelli
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Franco Fummi
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Hanno Hantson
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Jaan Raik
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. Maksim Jenihhin
    View author publications

    You can also search for this author in PubMed Google Scholar

  9. Raimund Ubar
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Jaan Raik.

Additional information

Responsible Editor: L. M. Bolzani Pöhls

Rights and permissions

Reprints and permissions

About this article

Cite this article

Guarnieri, V., Di Guglielmo, G., Bombieri, N. et al. On the Reuse of TLM Mutation Analysis at RTL. J Electron Test 28, 435–448 (2012). https://doi.org/10.1007/s10836-012-5303-6

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10836-012-5303-6

Keywords

Search

Navigation