Markus Becker
Tao Xie
ABSTRACT
This paper presents the XEMU framework for mutation based testing of embedded software binaries. We apply an extension of the QEMU software emulator, which injects mutations at run-time by dynamic code translation without affecting the binary software under test. The injection is based on a mutation table, which is generated by control flow graph (CFG) analysis of the disassembled code prior to its execution without presuming access to source code. We introduce our approach by the example of the ARM instruction set architecture for which a mutation taxonomy is presented. In addition to extending the testing scope to target specific low level faults, XEMU addresses the reduction of the mutants creation, execution, and detection overheads. Moreover, we reduce testing efforts by applying binary CFG analysis and constraint-based test generation for improved test quality. The experimental results of a car motor management software show significant improvements over conventional source code based approaches while providing 100% accuracy in terms of the computed test quality metrics.
- ARM Architecture Ref. Manuals. http://infocenter.arm.com.Google Scholar
- QEMU - Open Source Processor Emulator. http://www.qemu.org.Google Scholar
- TargetLink, dSPACE GmbH. http://www.dspace.com.Google Scholar
- J. ao A. Durães and H. S. Madeira. Emulation of Software Faults: A Field Data Study and a Practical Approach. IEEE Transactions on Software Engineering, 32:849--867, 2006. Google ScholarDigital Library
- ARM Ltd. ELF for the ARM Architecture, 2009.Google Scholar
- F. Bellard. QEMU, a Fast and Portable Dynamic Translator. In ATEC '05: Proceedings of the Annual Conference on USENIX Annual Technical Conference, pages 41--41, Berkeley, CA, USA, 2005. USENIX Association. Google ScholarDigital Library
- B. Bogacki and B. Walter. Aspect-Oriented Response Injection: An Alternative to Classical Mutation Testing. In K. Sacha, editor, Software Engineering Techniques: Design for Quality, volume 227, pages 273--282. 2007.Google Scholar
- C. Cifuentes. Interprocedural Ddata Flow Decompilation. Journal of Programming Languages, 4:77--99, 1996.Google Scholar
- C. Cifuentes, D. Simon, and A. Fraboulet. Assembly to High-Level Language Translation. In In Int. Conf. on Softw. Maint, pages 228--237. IEEE-CS Press, 1998. Google ScholarDigital Library
- R. DeMillo, E. Krauser, and A. Mathur. Compiler-Integrated Program Mutation. In Computer Software and Applications Conference, 1991. COMPSAC '91, Proceedings of the Fifteenth Annual International, pages 351--356, sep 1991.Google Scholar
- V. Ganesh and D. L. Dill. A Decision Procedure for Bit-Vectors and Arrays. In Computer Aided Verification (CAV '07), Berlin, Germany, July 2007. Springer-Verlag. Google ScholarDigital Library
- V. Guarnieri, N. Bombieri, G. Pravadelli, F. Fummi, H. Hantson, J. Raik, M. Jenihhin, and R. Ubar. Mutation Analysis for Systemc Designs at TLM. In Test Workshop (LATW), 2011 12th Latin American, pages 1--6, march 2011. Google ScholarDigital Library
- C. Guillon. Program Instrumentation with QEMU. In DATE '11: Proceedings of the Conference on Design, Automation and Test in Europe, Grenoble, France, 2011.Google Scholar
- Y. Jia and M. Harman. An Analysis and Survey of the Development of Mutation Testing. IEEE Transactions on Software Engineering, 2010. Google ScholarDigital Library
- P. Lisherness and K.-T. T. Cheng. SCEMIT: A SystemC Error and Mutation Injection Tool. In Proceedings of the 47th Design Automation Conference, DAC '10, pages 228--233, New York, NY, USA, 2010. ACM. Google ScholarDigital Library
- A. S. Namin, J. H. Andrews, and D. J. Murdoch. Sufficient Mutation Operators for Measuring Test Effectiveness. In IN PROC. ICSE, pages 351--360, 2008. Google ScholarDigital Library
- H. A. Richard, R. A. Demillo, and B. H. et al. Design of Mutant Operators for the C Programming Language. Technical report, 1989.Google Scholar
- R. A. DeMillo and J. A. Offut Constraint-Based Automatic Test Data Generation. In IEEE Trans on Software Eng., vol. 17, no. 9, pages 900--910, 1991. Google ScholarDigital Library
- Y. seung Ma, J. Offutt, and Y. R. Kwon. Mujava: An Automated Class Mutation System. Software Testing, Verification & Reliability, 15:97--133, 2005. Google ScholarDigital Library
- A. Sloss, D. Symes, and C. Wright. ARM System Developer's Guide: Designing and Optimizing System Software. Morgan Kaufmann Publishers Inc., San Francisco, CA, USA, 2004. Google ScholarDigital Library
- SpringSoft Inc. CERTITUDE Functional Qualification System. 2011.Google Scholar
- T. Xie, W. Mueller, and F. Letombe. IP-XACT Based System Level Mutation Testing. In High Level Design Validation and Test Workshop (HLDVT) IEEE International, 2011. Google ScholarDigital Library
- D. Baldin, S. Groesbrink, and S. Oberthuer. Enabling Constraint-Based Binary Reconfiguration by Binary Analysis. In International Journal on Computing (JoC), 2011.Google Scholar
Index Terms
- XEMU: an efficient QEMU based binary mutation testing framework for embedded software
Recommendations
Binary mutation testing through dynamic translation
DSN '12: Proceedings of the 2012 42nd Annual IEEE/IFIP International Conference on Dependable Systems and Networks (DSN)This paper presents a novel mutation based testing method through binary mutation. For this, a table of mutants is derived by control flow analysis of a disassembled binary under test. Mutations are injected at runtime by dynamic translation. Thus, our ...
Selection and Prioritization of Test Cases by Combining White-Box and Black-Box Testing Methods
ECBS-EERC '13: Proceedings of the 2013 3rd Eastern European Regional Conference on the Engineering of Computer Based SystemsIn this paper, we present a methodology that combines both white-box and black-box testing, in order to improve testing quality for a given class of embedded systems. The goal of this methodology is generation of test cases for the new functional ...
EmuID: Detecting presence of emulation through microarchitectural characteristic on ARM
AbstractSoftware emulation is at the core of efficient automated software analysis. It allows efficient use of computing resources by running multiple instances on a single machine. Also, software emulation naturally provides a strong ...
Comments