Skip to main content
SpringerLink
Log in

Differential Testing of Simulation-Based Virtual Machine Generators

Automatic Detection of VM Generator Semantic Gaps Between Simulation and Generated VMs

  • Conference paper
  • First Online:
Reuse and Software Quality (ICSR 2022)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 13297))

Included in the following conference series:

Abstract

Testing and debugging language Virtual Machines (VMs) is a laborious task without the proper tooling. This complexity is aggravated when the VM targets multiple architectures. To solve this problem, simulation-based VM generator frameworks allow one to write test cases on the simulation, but those test cases do not ensure the correctness of the generated artifact due to the semantic gaps between the simulated VM and generated VMs.

We propose Test Transmutation to extend simulation-based VM generator frameworks to support test case generation. It validates the generated VM by also running test cases generated from existing simulation test cases. Results of the generated test cases are compared against the simulation test cases using differential testing. Moreover, test cases are automatically mutated with non-semantic-preserving mutations.

Test Transmutation detects bugs that are representative of typical VM modifications. We demonstrate its effectiveness by applying it to a set of real test cases of the Pharo VM. It allowed us to find several issues that were unknown to the VM development team. Our approach shows promising results to test simulation-based VM generator frameworks.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 44.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 59.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    www.pypy.org.

  2. 2.

    https://www.qemu.org.

References

  1. Abdi, M., Rocha, H., Demeyer, S.: test amplification in the Pharo Smalltalk ecosystem. In: International Workshop on Smalltalk Technologies (IWST), August 2019. shorturl.at/floF4

  2. Aichernig, B.K., et al.: Model-based mutation testing of an industrial measurement device. In: Seidl, M., Tillmann, N. (eds.) TAP 2014. LNCS, vol. 8570, pp. 1–19. Springer, Cham (2014). https://doi.org/10.1007/978-3-319-09099-3_1

    Chapter  Google Scholar 

  3. Aichernig, B.K., Brandl, H., Jöbstl, E., Krenn, W.: Efficient mutation killers in action. In: 2011 Fourth IEEE International Conference on Software Testing, Verification and Validation, pp. 120–129 (2011). https://doi.org/10.1109/ICST.2011.57

  4. Aichernig, B.K., Brandl, H., Jöbstl, E., Krenn, W., Schlick, R., Tiran, S.: Killing strategies for model-based mutation testing. Softw. Test. Verif. Reliab. 25(8), 716–748 (2015)

    Article  Google Scholar 

  5. Alpern, B., et al.: Experiences porting the jikes rvm to linux/ia32. In: Java Virtual Machine Research and Technology Symposium, pp. 51–64 (2002)

    Google Scholar 

  6. Besnard, V., Brun, M., Dhaussy, P., Jouault, F., Olivier, D., Teodorov, C.: Towards one model interpreter for both design and deployment. In: Third International Workshop on Executable Modeling (EXE 2017), September 2017. https://hal.archives-ouvertes.fr/hal-01585318

  7. Black, A.P., Ducasse, S., Nierstrasz, O., Pollet, D., Cassou, D., Denker, M.: Pharo by Example. Square Bracket Associates, Kehrsatz, Switzerland (2009). http://books.pharo.org

  8. Bouraqadi, N., Mason, D.: Mocks, proxies, and transpilation as development strategies for web development. In: Proceedings of the 11th edition of the International Workshop on Smalltalk Technologies, pp. 1–6. IWST 2016, Association for Computing Machinery, August 2016. http://www.esug.org/data/ESUG2016/IWST/Papers/IWST_2016_paper_23.pdf

  9. Casey, K., Gregg, D., Ertl, M.A.: Tiger – an interpreter generation tool. In: Bodik, R. (ed.) CC 2005. LNCS, vol. 3443, pp. 246–249. Springer, Heidelberg (2005). https://doi.org/10.1007/978-3-540-31985-6_18

    Chapter  Google Scholar 

  10. Chen, J., et al.: A survey of compiler testing. ACM Comput. Surv. 53(1), 1–36 (2020). https://dl.acm.org/doi/10.1145/3363562

  11. Commitee, C.S.: C99 specification (2007). shorturl.at/goyJQ

  12. DeMillo, R.A., Lipton, R.J., Sayward, F.G.: Program mutation: a new approach to program testing. Infotech State Art Report Softw. Testing 2(1979), 107–126 (1979)

    Google Scholar 

  13. Ertl, M.A., Gregg, D.: Optimizing indirect branch prediction accuracy in virtual machine interpreters. In: Proceedings of the ACM SIGPLAN 2003 Conference on Programming Language Design and Implementation, pp. 278–288 (2003)

    Google Scholar 

  14. Fraser, G., Zeller, A.: Mutation-driven generation of unit tests and oracles. IEEE Trans. Softw. Eng. 38(2), 278–292 (2012). https://doi.org/10.1109/TSE.2011.93

    Article  Google Scholar 

  15. Gregg, D., Ertl, M.A.: A language and tool for generating efficient virtual machine interpreters. In: Lengauer, C., Batory, D., Consel, C., Odersky, M. (eds.) Domain-Specific Program Generation. LNCS, vol. 3016, pp. 196–215. Springer, Heidelberg (2004). https://doi.org/10.1007/978-3-540-25935-0_12

    Chapter  Google Scholar 

  16. Harman, M., Jia, Y., Langdon, W.B.: Strong higher order mutation-based test data generation. In: Proceedings of the 19th ACM SIGSOFT Symposium and the 13th European Conference on Foundations of Software Engineering, pp. 212–222. ESEC/FSE 2011, Association for Computing Machinery, New York (2011). https://doi.org/10.1145/2025113.2025144

  17. Hölzle, U., Chambers, C., Ungar, D.: Optimizing dynamically-typed object-oriented languages with polymorphic inline caches. In: America, P. (ed.) ECOOP 1991. LNCS, vol. 512, pp. 21–38. Springer, Heidelberg (1991). https://doi.org/10.1007/BFb0057013

    Chapter  Google Scholar 

  18. Ingalls, D., Kaehler, T., Maloney, J., Wallace, S., Kay, A.: Back to the future: the story of squeak, a practical smalltalk written in itself. In: Proceedings of Object-Oriented Programming, Systems, Languages, and Applications conference (OOPSLA 1997), pp. 318–326. ACM Press, November 1997. https://doi.org/10.1145/263700.263754

  19. Just, R., Jalali, D., Inozemtseva, L., Ernst, M.D., Holmes, R., Fraser, G.: Are mutants a valid substitute for real faults in software testing? In: Proceedings of the 22nd ACM SIGSOFT International Symposium on Foundations of Software Engineering, pp. 654–665 (2014)

    Google Scholar 

  20. Kotselidis, C., Nisbet, A., Zakkak, F.S., Foutris, N.: Cross-ISA debugging in meta-circular VMs. In: Proceedings of International Workshop on Virtual Machines and Intermediate Languages (VMIL 2017), pp. 1–9 (2017). https://doi.org/10.1145/3141871.3141872

  21. Le, V., Afshari, M., Su, Z.: Compiler validation via equivalence modulo inputs. In: Programming Language Design and Implementation, PLDI 2014 (2014). https://doi.org/10.1145/2594291.2594334

  22. Le, V., Sun, C., Su, Z.: Finding deep compiler bugs via guided stochastic program mutation. ACM SIGPLAN Notices 50, 386–399 (2015). https://doi.org/10.1145/2858965.2814319

  23. McKeeman, W.M.: Differential testing for software. Digital Tech. J. 10, 100–107 (1998)

    Google Scholar 

  24. Miranda, E.: The cog smalltalk virtual machine. In: Proceedings of VMIL 2011 (2011)

    Google Scholar 

  25. Miranda, E., Béra, C., Boix, E.G., Ingalls, D.: Two decades of smalltalk vm development: live vm development through simulation tools. In: Proceedings of International Workshop on Virtual Machines and Intermediate Languages (VMIL 2018), pp. 57–66. ACM (2018). https://doi.org/10.1145/3281287.3281295

  26. Misse-Chanabier, P., Aranega, V., Polito, G., Ducasse, S.: Illicium a modular transpilation toolchain from Pharo to c. In: International workshop of Smalltalk Technologies. Köln, Germany, August 2019

    Google Scholar 

  27. Papadakis, M., Henard, C., Harman, M., Jia, Y., Le Traon, Y.: Threats to the validity of mutation-based test assessment. In: Proceedings of the 25th International Symposium on Software Testing and Analysis, pp. 354–365. ISSTA 2016, Association for Computing Machinery, New York (2016). https://doi.org/10.1145/2931037.2931040

  28. Papadakis, M., Malevris, N.: Automatic mutation test case generation via dynamic symbolic execution. In: 2010 IEEE 21st International Symposium on Software Reliability Engineering, pp. 121–130 (2010). https://doi.org/10.1109/ISSRE.2010.38

  29. Person, S., Dwyer, M.B., Elbaum, S., Pǎsǎreanu, C.S.: Differential symbolic execution. In: Proceedings of the 16th ACM SIGSOFT International Symposium on Foundations of Software Engineering, pp. 226–237. SIGSOFT 2008/FSE-16, Association for Computing Machinery, November 2008. https://doi.org/10.1145/1453101.1453131

  30. Polito, G., et al.: Cross-ISA testing of the Pharo VM: lessons learned while porting to ARMv8. In: MPLR, Germany. Münster, Germany, September 2021. https://doi.org/10.1145/3475738.3480715

  31. Rigo, A., Pedroni, S.: PyPy’s approach to virtual machine construction. In: Proceedings of the 2006 conference on Dynamic languages symposium, ACM, New York (2006)

    Google Scholar 

  32. RPythonCommunity: Rpython documentation on test translation (2016). shorturl.at/gBDGT

  33. Rutherford, M., Carzaniga, A., Wolf, A.: Evaluating test suites and adequacy criteria using simulation-based models of distributed systems. Softw. Eng. IEEE Trans. 34, 452–470 (2008). https://doi.org/10.1109/TSE.2008.33

  34. Simon, D., Cifuentes, C., Cleal, D., Daniels, J., White, D.: Java on the bare metal of wireless sensor devices: the Squawk Java virtual machine. In: VEE 2006: Proceedings of the 2nd International Conference on Virtual Execution Environments, pp. 78–88. ACM Press, New York (2006). https://doi.org/10.1145/1134760.1134773

  35. Sun, C., Le, V., Su, Z.: Finding compiler bugs via live code mutation. In: Proceedings of the 2016 ACM SIGPLAN International Conference on Object-Oriented Programming, Systems, Languages, and Applications, pp. 849–863. OOPSLA 2016, Association for Computing Machinery, October 2016. https://doi.org/10.1145/2983990.2984038

  36. Terekhov, A.A., Verhoef, C.: The realities of language conversions. IEEE Softw. 17(6), 111–124 (2000). https://doi.org/10.1109/52.895180

    Article  Google Scholar 

  37. Ungar, D.: Generation scavenging: a non-disruptive high performance storage reclamation algorithm. ACM SIGPLAN Notices 19(5) (1984). https://doi.org/10.1145/390011.808261

  38. Ungar, D., Spitz, A., Ausch, A.: Constructing a metacircular virtual machine in an exploratory programming environment. In: Companion to Object-Oriented Programming, Systems, Languages, and Applications Conference (OOPSLA 2005), ACM (2005)

    Google Scholar 

  39. Wimmer, C., Haupt, M., Vanter, M.L.V.D., Jordan, M., Daynes, L., Simon, D.: Maxine: an approachable virtual machine for, and in, java. Technical Report 2012–0098, Oracle Labs (2012)

    Google Scholar 

  40. Würthinger, T., et al.: One VM to rule them all. In: International Symposium on New Ideas, New Paradigms, and Reflections on Programming & Software (ONWARD 2013) (2013)

    Google Scholar 

  41. Yang, X., Chen, Y., Eide, E., Regehr, J.: Finding and understanding bugs in C compilers. In: Programming Language Design and Implementation. PLDI 2011 (2011). https://doi.org/10.1145/1993498.1993532

Download references

Acknowledgement

This work was supported by Ministry of Higher Education and Research, Hauts de France Regional Council and the AlaMVic Action Exploratoire INRIA - Lille Nord Europe.

Author information

Authors and Affiliations

  1. Inria, Univ Lille, CNRS, Centrale Lille, UMR 9189 - CRIStAL, Lille, 59000, France

    Pierre Misse-chanabier & Stéphane Ducasse

  2. CNRS, UMR 9189 - CRIStAL, Univ Lille, Centrale Lille, Inria, Lille, 59000, France

    Guillermo Polito & Pablo Tesone

  3. IMT Nord Europe, Institut Mines-Télécom, Univ. Lille, Centre for Digital Systems, Douai, 59500, France

    Noury Bouraqadi & Luc Fabresse

Authors
  1. Pierre Misse-chanabier
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Guillermo Polito
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Noury Bouraqadi
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Stéphane Ducasse
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Luc Fabresse
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Pablo Tesone
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding authors

Correspondence to Pierre Misse-chanabier , Guillermo Polito , Noury Bouraqadi , Stéphane Ducasse , Luc Fabresse or Pablo Tesone .

Editor information

Editors and Affiliations

  1. University of Namur, Namur, Belgium

    Gilles Perrouin

  2. École de Technologie Supérieure, Montreal, QC, Canada

    Naouel Moha

  3. University of Montpellier, Montpellier, France

    Abdelhak-Djamel Seriai

Rights and permissions

Reprints and permissions

Copyright information

© 2022 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Misse-chanabier, P., Polito, G., Bouraqadi, N., Ducasse, S., Fabresse, L., Tesone, P. (2022). Differential Testing of Simulation-Based Virtual Machine Generators. In: Perrouin, G., Moha, N., Seriai, AD. (eds) Reuse and Software Quality. ICSR 2022. Lecture Notes in Computer Science, vol 13297. Springer, Cham. https://doi.org/10.1007/978-3-031-08129-3_7

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-08129-3_7

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-08128-6

  • Online ISBN: 978-3-031-08129-3

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation