Skip to main content
SpringerLink
Log in

Application of metamorphic testing monitored by test adequacy in a Monte Carlo simulation program

  • Published:
Software Quality Journal Aims and scope Submit manuscript

Abstract

One of the grand challenges for adequately testing complex software is due to the oracle problem. Metamorphic Testing (MT) is a promising technique to alleviate the oracle problem through using one or multiple Metamorphic Relations (MRs) as test oracles. MT checks the satisfaction of every MR among the outputs of the MR-related tests instead of the correctness of individual test outputs. In practice, it is fairly easy to find MRs for testing any program, but it is very difficult to develop “good” MRs and evaluate their adequacy. A systematic approach for developing MRs and evaluating their adequacy in MT remains to be developed. In this paper, we propose a framework for evaluating MT and iteratively developing adequate MRs monitored by MT adequacy evaluation. The MT adequacy is measured by program coverages, mutation testing, and testing MRs with mutation tests. The MT evaluation results are used for guiding the iterative development of MRs, generating tests, and analyzing test outputs. We explain the framework through a testing example on an image processing program that is used for building the 3-dimensional structure of a biology cell based on its confocal image sections. In order to demonstrate the effectiveness of the proposed framework, we reported a case study of testing a complex scientific program: a Monte Carlo modeling program that simulates photon propagations in turbid tissue phantoms for accurate and efficient generation of reflectance images from biological tissues. The case study has shown the effectiveness of proposed MT framework for testing scientific software in general and the necessity of the MT enhancement in the development of adequate MRs. The case study results can be easily adapted for testing other software.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11

Similar content being viewed by others

References

  • Arridge, S. R. (1999). Optical tomography in medical imaging. Inverse Problems, 15, R41–R93.

    Article  MathSciNet  MATH  Google Scholar 

  • Asrafi, M., Liu, H., & Kuo, F-C. (2011). On testing effectiveness of metamorphic relations: A Case study, In 5th International Conference on Secure Software Integration and Reliability Improvement (SSIRI), pp. 147–156.

  • Baresi L., & Young, M. (2001). Test oracles. Technical Report CIS-TR01 -02, Department of Computer and Information Science, University of Oregon.

  • Barr, E. T., Harman, M., McMinn, P., Shahbaz, M., & Yoo, S. (2015). The Oracle problem in software testing: A survey. IEEE Transactions on Software Engineering, 41(5), 507–525.

    Article  Google Scholar 

  • Chan, W. K., Cheung, S. C., & Leung, K. R. (2007). A metamorphic testing approach for online testing of service-oriented software applications. International Journal of Web Services Research, 4(1), 60–80.

    Google Scholar 

  • Chen, T. Y., Cheung, S. C., & Yiu, S. (1998). Metamorphic testing: A new approach for generating next test cases, Technical Report HKUST-CS98-01, Department of Computer Science, Hong Kong University of Science and Technology.

  • Chen, T.Y., Feng, J., Tse, T.H. (2002). Metamorphic testing of programs on partial differential equations: a case study. In Proceedings of 26th Annual International Computer Software and Applications Conference (COMPSAC), pp. 327–333.

  • Chen, T. Y., Ho, J. W. K., Liu, H., & Xie, X. (2009). An innovattive approach for testing bioinformatics programs using metamorphic testing, BMC Bioinformatics, pp. 10–24.

  • Chen, T. Y., Huang, D. H., Tse, T. H., & Zhou, Z. Q. (2004). Case studies on the selection of useful relations in metamorphic testing”, In Proceedings of the 4 th Ibero-American Symposium on Software Engineering and Knowledge Engineering, pp. 569–583.

  • Chen, C., Lu, J. Q., Li, K., Zhao, S., Brock, R. S., & Hu, X. H. (2007). Numerical study of reflectance imaging using a parallel Monte Carlo method. Medical Physics, 34, 2939–2948.

    Article  Google Scholar 

  • Chen, T. Y., Poon, P. L., & Xie, X. (2015). METRIC: METamorphic relation identification based on the category-choice framework. Journal of Systems and Software, 116(C), 177–190.

    Google Scholar 

  • Chen, T. Y., Tse, T. H., & Zhou, Z. Q. (2002). Semi-proving: An integrated method based on global symbolic evaluation and metamorphic testing. In Proceedings of the International Symposium on Software Testing and Analysis (ISSTA), pp. 191–195.

  • Confocal microscope, https://en.wikipedia.org/wiki/Confocal_microscopyWikipedia, last accessed on April 30, 2016.

  • Ding, J., & Zhang D. (2016). A machine learning approach for developing test Oracles for testing scientific software,In the 28 th SEKE (SEKE 2016), San Francisco, July 1–3.

  • Ding, J., Clarke, P. J., Argote-Garcia, G., & He, X. (2009). A methodology for evaluating test coverage criteria of high level Petri nets. Information and Software Technology, 51(11), 1520–1533.

    Article  Google Scholar 

  • Ding, J., Wu, T., Lu, J. Q., Hu, X. (2010). Self-checked metamorphic Testing of an image processing program, The 4th IEEE International Conference on Security Software Integration and Reliability Improvement, Singapore, June 9–11.

  • Ding, J., Zhang, D., Hu, X. (2016). An application of metamorphic testing for testing scientific software, In 1 st workshop on metamorphic testing with ICSE, Austin, TX, May 16.

  • Farrell, P. E., Pigott, M. D., Gorman, G. J., Ham, D. A., Wilson, C. R., & Bond, T. M. (2011). Automated continuous verification for numerical simulation. Geoscientific Model Development, 4(2), 435–449.

    Article  Google Scholar 

  • Gotleib, A. & Botella, B. (2003). Automated metamorphic testing, In Proceedings of 27 th Annual International Computer Software and Applications Conference, (pp. 34–40).

  • Gotlieb, A., Botella, B., & Rueher, M. (1998). Automatic test data generation using constraint solving techniques. In ACM International Symposium on Software Testing and Analysis (ISSTA). Software Engineering Notes, 23(2):53–62.

  • Guderlei, R., & Mayer, J. (2007). Statistical metamorphic testing—testing programs with random output by means of statistical hypothesis tests and metamorphic testing”, In Proceedings of the 7 th International Conference on Quality Software. pp. 404–409, 2007.

  • Hu, P., Zhang, Z., Chan, W. K., & Tse, T. H. (2006). An empirical comparison between direct and indirect test result checking approaches. In Proceedings of the 3 rd International Workshop on Soft. Quality Assurance, pp. 6–13.

  • Jameel, T., Lin, M., & Chao, L. (2015). Test oracles based on metamorphic relations for image processing applications. In 16 th International Conference on SE, AI, Networking and Parallel/Distributed Computing (SNPD), pp. 1–6.

  • Jia, Y., & Harman, M. (2011). An analysis and survey of the development of mutation testing. In IEEE Transactions on Software Engineering, 37(5), pp. 649–678, September–October 2011.

  • Kanewala, U., & Bieman, J. M. (2014). Testing scientific software: A systematic literature review. Information and Software Technology, 56(10), 1219–1232.

    Article  Google Scholar 

  • Kanewala, U., Bieman, J. M., & Ben-Hur, A. (2015). Predicting metamorphic relations for testing scientific software: A machine learning approach using graph kernels. Journal of Software Testing, Verification and Reliability, 26(3), 245–269.

    Article  Google Scholar 

  • Keijzer, M., Jacques, S. L., Prahl, S. A., & Welch, A. J. (1989). Light distributions in artery tissure: Monte Carlo simulations for finite-diameter laser beams. Lasers in Surgery and Medicine, 9, 148–154.

    Article  Google Scholar 

  • Le, V., Afshari, M., & Su, Z. (2014). Compiler validation via equivalence modulo inputs. In Proceedings of the 35th ACM SIGPLAN Conference on Programming Language Design and Implementation (PLDI ‘14). ACM, New York, NY, USA, pp.216–226.

  • Lindvall, M., Ganesan, D., Árdal, R., & Wiegand, R. E. (2015). “Metamorphic model-based testing applied on NASA DAT: an experience report”. In Proceedings of the 37 th International Conference on Software Engineering, Vol. 2 (ICSE ‘15), Vol. 2. pp. 129–138.

  • Liu, H., Kuo, F., Towey, D., & Chen, T. Y. (2014). How effectively does metamorphic testing alleviate the oracle problem? IEEE Transactions on Software Engineering, 40(1), 4–22.

    Article  Google Scholar 

  • Mayer, J., Guderlei, R. (2006). On random testing of image processing applications. In Proceedings of 6 th International Conference on Quality Software (QSIC’06), pp. 85–92.

  • Mayer, J. & Guderlei, R. (2006). An empirical study on the selection of good metamorphic relations. In Proceedings of the 30 th Annual International Computer Software and Applications Conference (COMPSAC), pp. 475–484.

  • Mayer, J., Infor, A. A., Ulm, U. (2005). On testing image procesing applications with staticstical methods. Software Engineering (SE 2005), Lecture Notes in Informatics, pp. 69–78.

  • Murphy, C. Kaiser, G., Hu, L. & Wu, L. (2008). Properties of machine learning applications for use in metamorphic testing”. In Proceedings of the 20 th International conference on software engineering and knowledge engineering (SEKE), pp. 867–872.

  • Murphy, C. (2010). Metamorphic testing techniques to detect defects in applications without test Oracles. Doctoral dissertation, Columbia University.

  • Murphy, C., Shen, K.,& Kaiser, G. (2009). Automatic system testing of program without test oracles. In Proceedings of 2009 ACM International Symposium of Software Testing and Analysis (ISSTA).

  • Murphy, C., Shen, K., & Kaiser, G. (2009). Using JML runtime assertion checking to automate metamorphic testing in applications without test oracles, In Proceedings of the 2 nd IEEE International Conference on Software Testing, Verification and Validation (ICST).

  • Nguyen-Hoan, L., Flint, S., Sankara, R. (2010). A survey of scienfitic software development, In 2010 ACM International Symposium on Empiricial Software Engineering and Measurement (ESSM’10), pp. 12:1–12:10.

  • Nie, C., & Leung, H. (2011). A survey of combinatorial testing. ACM Computing Survey, 43(2), 11.

    Article  MATH  Google Scholar 

  • Obayashi, M., Kubota, H., McCarron, S. P., & Mallet L. (1998). The assertion based testing tool for OOP: ADL2. http://adl.xopen.org/exgr/icse/icse98.htm May 1998.

  • Pacheco, P. (1996). Parallel Programming with MPI. Morgan Kaufmann; 1st edition.

  • Pezzè, M., & Young, M. (2007). Software testing and analysis: Process, principles, and techniques. New Jersey: Wiley.

    MATH  Google Scholar 

  • Segura, S., Fraser, G., Sanchez, A., & Ruiz-Cortes, A. (2016). A Survey on Metamorphic Testing. In IEEE Transaction on Software Engineering (vol. PP, no. 99). doi:10.1109/TSE.2016.2532875.

  • Sanders, R., Kelly, D. (2008). The challenge of testing scientific software.In Proceedings of the Conference for the Association for Software Testing (CAST), pp. 30–36, Toronto, July.

  • Shan, L. & Zhu, H. (2009). Generating structually complex test cases by data mutation: A case study of testing an automated modelling tool. The Computer Journal, 52(5).

  • Weyuker, E. J. (1982). On testing non-testable program. Computer Journal, 25(4), 465–470.

    Article  Google Scholar 

  • Xie, X., Ho, J., Murphy, C., Kaiser, G., Xu, B., & Chen, T. Y. (2011). Testing and validating machine learning classifiers by metamorphic testing. J. System and Software., 84(4), 544–558.

    Article  Google Scholar 

  • Zhou, Z. Q., Chan, W. K., Chan, W. K., Tse, T. H., & HU, P. (2009). Experimental study to compare the use of metamorphictesting and assertion checking. Journal of Software, 20(10), 2637–2654.

    Article  Google Scholar 

  • Zhou, Z.Q., Xiang, S., Chen, T.Y. (2015). Metamorphic testing for software quality assessment: A study of search engines. In IEEE Transactions on Software Engineering, PrePrints, doi:10.1109/TSE.2015.2478001.

  • Zhu, H., Hall, P. A., & May, J. H. (1997). Software unit test coverage and adequacy. ACM Computing Surveys, 29(4), 366–427.

    Article  Google Scholar 

  • Zhu, H., & He, X. (2002). A methodology of testing high-level petri nets. Journal of Information and Software Technology, 44, 473–489.

    Article  Google Scholar 

Download references

Acknowledgments

We thank Dr. Tong Wu for his implementation and testing of the Monte Carlo simulation program. This research is supported in part by Grant #1262933 and #1560037 from the National Science Foundation.

Author information

Authors and Affiliations

  1. Department of Computer Science, East Carolina University, Greenville, NC, USA

    Junhua Ding

  2. Department of Physics, East Carolina University, Greenville, NC, USA

    Xin-Hua Hu

Authors
  1. Junhua Ding
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Xin-Hua Hu
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Junhua Ding.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Ding, J., Hu, XH. Application of metamorphic testing monitored by test adequacy in a Monte Carlo simulation program. Software Qual J 25, 841–869 (2017). https://doi.org/10.1007/s11219-016-9337-3

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11219-016-9337-3

Keywords

Search

Navigation