Anders Lundgren
ABSTRACT
One of the biggest challenges for conducting automated systematic testing on scientific programs is the oracle problem. This challenge is especially prevalent in the field of bioinformatics due to the inherent complexity of these programs. In this paper, we explore two approaches: pseudo-oracles and metamorphic testing for conducting automated systematic testing of bioinformatics programs. We use BBMap: an open source genome alignment tool as the system under test to evaluate the effectiveness of the above two approaches for identifying subtle faults.
Our results show that the pseudo-oracle testing does not provide a consistent basis for fault-detection. This is mainly due to not having a consistent rate of agreement between the outputs of the system under test and the pseudo-oracle. On the other hand, metamorphic testing detects a majority of subtle faults, suggesting that it is more effective in identifying these types of faults.
- S. Altschul, B. Demchak, R. Durbin, R. Gentleman, M. Krzywinski, H. Li, A. Nekrutenko, J. Robinson, W. Rasband, J. Taylor, and C. Trapnell. The anatomy of successful computational biology software. Nature Biotechnology, 31(10):894--897, 2010.Google Scholar
Cross Ref
- E. Barr, M. Harman, P. McMinn, M. Shahbaz, and S. Yoo. The oracle problem in software testing: A survey. Software Engineering, IEEE Transactions on, 41(5):507--525, May 2015.Google ScholarDigital Library
- T. Y. Chen, T. H. Tse, and Z. Q. Zhou. Fault-based testing without the need of oracles. Information and Software Technology, 45(1):1--9, 2003.Google ScholarDigital Library
- P. A. Fujita, B. Rhead, A. S. Zweig, A. S. Hinrichs, D. Karolchik, M. S. Cline, M. Goldman, G. P. Barber, H. Clawson, A. Coelho, et al. The ucsc genome browser database: update 2011. Nucleic acids research, page gkq963, 2010.Google Scholar
- E. Giannoulatou, S.-H. Park, D. T. Humphreys, and J. W. Ho. Verification and validation of bioinformatics software without a gold standard: a case study of bwa and bowtie. BMC bioinformatics, 15(Suppl 16):S15, 2014.Google ScholarCross Ref
- Y. Jia and M. Harman. An analysis and survey of the development of mutation testing. IEEE Transactions on Software Engineering, 37:649--678, 2011. Google ScholarDigital Library
- L. N. Joppa, G. McInerny, R. Harper, L. Salido, K. Takeda, K. O'Hara, D. Gavaghan, and S. Emmott. Troubling trends in scientific software use. Science, 340(6134):814--815, 2013.Google ScholarCross Ref
- U. Kanewala. Techniques for automatic detection of metamorphic relations. In Software Testing, Verification and Validation Workshops (ICSTW), 2014 IEEE Seventh International Conference on, pages 237--238, March 2014. Google ScholarDigital Library
- U. Kanewala and J. M. Bieman. Techniques for testing scientific programs without an oracle. In Proc. SE-CSE, pages 48--57. IEEE, 2013. Google ScholarDigital Library
- U. Kanewala and J. M. Bieman. Testing scientific software: A systematic literature review. Information and Software Technology, 56(10):1219--1232, 2014. Google ScholarDigital Library
- U. Kanewala, J. M. Bieman, and A. Ben-Hur. Predicting metamorphic relations for testing scientific software: A machine learning approach using graph kernels. Software testing, verification and reliability, 2015. In press.Google Scholar
- W. J. Kent. BLAT--the BLAST-like alignment tool. Genome Research, 12(4):656--664, Apr. 2002.Google ScholarCross Ref
- J. Offut and Y. S. Ma. Description of method-level mutation operators for java. https://cs.gmu.edu/~offutt/mujava/mutopsMethod.pdf. Accessed: Dec, 2015.Google Scholar
- M. Papadakis and N. Malevris. An empirical evaluation of the first and second order mutation testing strategies. In Software Testing, Verification, and Validation Workshops (ICSTW), 2010 Third International Conference on, pages 90--99, April 2010. Google ScholarDigital Library
- R. Sanders and D. Kelly. The challenge of testing scientific software. In In Proc. of the Conference for the Association for Software Testing (CAST), pages 30--36, Toronto, July 2008.Google Scholar
- B. Smith and L. Williams. An empirical evaluation of the mujava mutation operators. In Testing: Academic and Industrial Conference Practice and Research Techniques - MUTATION, 2007. TAICPART-MUTATION 2007, pages 193--202, Sept 2007. Google ScholarDigital Library
- A. Sung, B. Choi, W. E. Wong, and V. Debroy. Mutant generation for embedded systems using kernel-based software and hardware fault simulation. Information and Software Technology, 53(10):1153--1164, 2011.Google ScholarCross Ref
- M. Umarji, C. Seaman, A. Koru, and H. Liu. Software engineering education for bioinformatics. In Software Engineering Education and Training, 2009. CSEET '09. 22nd Conference on, pages 216--223, Feb 2009. Google ScholarDigital Library
- E. J. Weyuker. On testing non-testable programs. The Computer Journal, 25(4):465--470, 1982.Google ScholarCross Ref
- L. L. Wu and G. G. Kaiser. Constructing subtle concurrency bugs using synchronization-centric second-order mutation operators. Columbia University Academic Commons, 2011.Google Scholar
Recommendations
Conformance testing of boolean programs with multiple faults
FMOODS'12/FORTE'12: Proceedings of the 14th joint IFIP WG 6.1 international conference and Proceedings of the 32nd IFIP WG 6.1 international conference on Formal Techniques for Distributed SystemsConformance testing is the problem of constructing a complete test suite of inputs based on a specification S such that any implementation I (of size less than a given bound) that is not equivalent to S gives a different output on the test suite than S. ...
Detecting associative shift faults in predicate testing
Fault-based predicate testing is a promising strategy for generating effective software test data. The test target is faults that may appear in predicates found in the specification or implementation of a computer program. A hierarchy of fault classes ...
Pseudo-random pattern testing of bridging faults
ICCD '97: Proceedings of the 1997 International Conference on Computer Design (ICCD '97)While previous research has focused on deterministic testing of bridging faults, this paper studies pseudo-random testing of bridging faults and describes a means for achieving high fault coverage in a built-in self-test (BIST) environment. Bridging ...
Comments