Abstract
Metamorphic testing emerged as a solution to the Oracle problem, with its foundation deeply rooted in the concept of Metamorphic Relations (MRs). Researchers have made an intriguing discovery that certain diverse MRs exhibit similar fault detection capabilities as the test oracle. However, defining the criteria for diverse MRs has posed a challenge. Traditional metrics like Mutation Score (MS) and Fault Detection Rate (FDR) fail to assess the diversity of MRs. This paper introduces the MUT Model as a foundational framework for analyzing the "MR Diversity" between a pair of MRs. The word "diversity" in this paper pertains to the differences in the types of faults that two MRs are inclined to detect. The experimental findings indicate that by harnessing posterior knowledge, specifically by analyzing the MUT Model, it is possible to derive prior knowledge that can aid in the construction of Metamorphic Relations. Most importantly, the MUT Model may draw conclusions that violate intuition, exposing more details of the core essence of MR Diversity. Moreover, the concept of MR Diversity serves as a basis for MR selection, resulting in enhanced fault detection capabilities compared to the conventional MS-based approach. Additionally, it offers valuable insights into the construction of composite metamorphic relations, with the goal of amplifying their fault detection abilities beyond those of their individual MR components.
Similar content being viewed by others
Data availability
The data of this study is openly available in Github at https://github.com/VinylLee/MUTModel.
References
Asrafi, M., Liu, H., & Kuo, F. C. (2011). On testing effectiveness of metamorphic relations: A case study. 2011 Fifth International Conference on Secure Software Integration and Reliability Improvement (pp. 147–156). https://doi.org/10.1109/SSIRI.2011.21
Binder, R. V. (1999). Testing object-oriented systems: models, patterns, and tools. Addison-Wesley Longman Publishing Co., Inc.
Cao, Y., Zhou, Z. Q., & Chen, T. Y. (2013). On the correlation between the effectiveness of metamorphic relations and dissimilarities of test case executions. 2013 13th International Conference on Quality Software (pp. 153–162). https://doi.org/10.1109/QSIC.2013.43
Chan, F. T., Chen, T. Y., Cheung, S. C., Lau, M. F., & Yiu, S. M. (1998). Application of metamorphic testing in numerical analysis. Proceedings of the IASTED International Conference on Software Engineering (SE’98).
Chen, T. Y., Huang, D. H., Tse, T. H., & Zhou, Z. Q. (2004). Case studies on the selection of useful relations in metamorphic testing. Proceedings of the 4th Ibero-American Symposium on Software Engineering and Knowledge Engineering (JIISIC 2004) (pp. 569–583). Citeseer
Chen, T. Y., Cheung, S. C., & Yiu, S. M. (2020). Metamorphic testing: a new approach for generating next test cases. arXiv preprint. arXiv:2002.12543.
Chen, T. Y., Kuo, F.-C., Liu, H., Poon, P.-L., Towey, D., Tse, T. H., & Zhou, Z. Q. (2018). Metamorphic testing: A review of challenges and opportunities. ACM Computing Surveys. https://doi.org/10.1145/3143561
Chen, T. Y., Poon, P.-L., & Xie, X. (2016). Metric: Metamorphic relation identification based on the category-choice framework. Journal of Systems and Software, 116, 177–190. https://doi.org/10.1016/j.jss.2015.07.037
Chen, T. Y., Tse, T. H., & Zhiquan, Z. (2001). Fault-based testing in the absence of an oracle. 25th Annual International Computer Software and Applications Conference (pp. 172–178). COMPSAC 2001. https://doi.org/10.1109/CMPSAC.2001.960614
Dong, G., Xu, B., Chen, L., Nie, C., & Wang, L. (2008). Case studies on testing with compositional metamorphic relations. Journal of Southeast University (English Edition), 24, 437–443.
Hoffman, D. (1999). Heuristic test oracles for automated testing, expected results are generated using a test oracle. a look at how heuristic oracles can strike a balance between exhaustive comparison and no comparison at all. Software Testing and Quality Engineering, 1, 28–41.
Liu, H., Kuo, F.-C., Towey, D., & Chen, T. Y. (2014). How effectively does metamorphic testing alleviate the oracle problem? IEEE Transactions on Software Engineering, 40(1), 4–22. https://doi.org/10.1109/TSE.2013.46
Liu, H., Liu, X., & Chen, T. Y. (2012). A new method for constructing metamorphic relations. 2012 12th International Conference on Quality Software (pp. 59–68). https://doi.org/10.1109/QSIC.2012.10
Liu, S., Yan, S., & Yang, X. (2022). A similarity-based metamorphic relations selection strategy for numerical computation programs. 2022 4th International Conference on Frontiers Technology of Information and Computer (ICFTIC) (pp. 290–294). https://doi.org/10.1109/ICFTIC57696.2022.10075300
Mayer, J., & Guderlei, R. (2006). An empirical study on the selection of good metamorphic relations. 30th Annual International Computer Software and Applications Conference (COMPSAC’06) (Vol. 1, pp. 475–484). https://doi.org/10.1109/COMPSAC.2006.24
Orso, A., & Rothermel, G. (2014). Software testing: a research travelogue (2000–2014). Future of Software Engineering Proceedings (pp. 117–132). https://doi.org/10.1145/2593882.2593885
Qiu, K., Zheng, Z., Chen, T. Y., & Poon, P.-L. (2022). Theoretical and empirical analyses of the effectiveness of metamorphic relation composition. IEEE Transactions on Software Engineering, 48(3), 1001–1017. https://doi.org/10.1109/TSE.2020.3009698
Rong, W., & Kerong, B. (2012). Researches on basic criterion and strategy of constructing metamorphic relations. Computer Science, 39(001), 115–119.
Sadi, M. S., Kuo, F., Ho, J. W. K., Charleston, M. A., & Chen, T. Y. (2011). Verification of phylogenetic inference programs using metamorphic testing. Journal of Bioinformatics and Computational Biology, 9(6), 729–747. https://doi.org/10.1142/S021972001100563X
Weyuker, E. J. (1982). On testing non-testable programs. The Computer Journal, 25(4), 465–470. https://doi.org/10.1093/comjnl/25.4.465
Wu, P., Xiao-Chun, S. H. I., Jiang-Jun, T., & Hui-Min, L. I. N. (2005). Metamorphic testing and special case testing: A case study. Journal of Software. https://doi.org/10.1360/jos161210
Yan, L., Sheng-ming, P., Cheng, W., Kang-wei, W., Tian, W., & Xiao-dong, X. (2016). Sensitivity of metamorphic relationships and its cluster analysis. Acta Electonica Sinica, 44(5), 1196–1201. https://doi.org/10.3969/j.issn.0372-2112.2016.05.026
Xie, X., Wong, W. E., Chen, T. Y., & Xu, B. (2013). Metamorphic slice: An application in spectrum-based fault localization. Information and Software Technology, 55(5), 866–879. https://doi.org/10.1016/j.infsof.2012.08.008
Xie, X., Li, J., Wang, C., & Chen, T. Y. (2016). Looking for an mr? try metwiki today. 2016 IEEE/ACM 1st International Workshop on Metamorphic Testing (MET) (pp. 1–4). https://doi.org/10.1145/2896971.2896976
Funding
This work was partly supported by the National Natural Science Foundation of China (NSFC) (Grant nos. 62172194, 62202206 and U1836116), the Natural Science Foundation of Jiangsu Province, China (Grant no. BK20220515), the Leading edge Technology Program of Jiangsu Natural Science Foundation, China (Grant no. BK20202001), the China Postdoctoral Science Foundation, China (Grant no. 2021M691310), and Qinglan Project of Jiangsu Province, China.
Author information
Authors and Affiliations
Contributions
Xiaodong Xie: Investigation, Methodology, Writing - original draft. Zhehao Li: Methodology, Writing - original draft. Jinfu Chen: Investigation, Data curation. Yue Zhang: Software, Validation, Supervision, Writing - review and editing. Xiangxiang Wang: Data curation, Visualization. Patrick Kwaku Kudjo: Writing - review and editing.
Corresponding author
Ethics declarations
Conflicts of interest
The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Appendix 1: Long table
Appendix 1: Long table
The Metamorphic Relations of Earth Distance (ED), Sin, Sparse Matrix Multiplication algorithm (SMM), and DSPA are placed here due to their length. The result of metamorphic relation (MR) clustering is presented here for the same result. We strive to offer comprehensive explanations for all MRs. Nevertheless, due to the complexity inherent in certain programs, such as SMM and DNA, it becomes challenging to provide exhaustive explanations or test case illustrations. For the detailed exploration of the MRs pertaining to DNA, we refer to previous researches (Sadi et al., 2011; Qiu et al., 2022). In the case of SMM, we will utilize a \(2\times 2\) matrix as an example to demonstrate the MRs, assuming the original input is represented by Eq. 8.
Rights and permissions
Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.
About this article
Cite this article
Xie, X., Li, Z., Chen, J. et al. MUT Model: a metric for characterizing metamorphic relations diversity. Software Qual J 32, 1413–1455 (2024). https://doi.org/10.1007/s11219-024-09689-x
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11219-024-09689-x