Skip to main content
Log in

MUT Model: a metric for characterizing metamorphic relations diversity

  • Research
  • Published:
Software Quality Journal Aims and scope Submit manuscript

Abstract

Metamorphic testing emerged as a solution to the Oracle problem, with its foundation deeply rooted in the concept of Metamorphic Relations (MRs). Researchers have made an intriguing discovery that certain diverse MRs exhibit similar fault detection capabilities as the test oracle. However, defining the criteria for diverse MRs has posed a challenge. Traditional metrics like Mutation Score (MS) and Fault Detection Rate (FDR) fail to assess the diversity of MRs. This paper introduces the MUT Model as a foundational framework for analyzing the "MR Diversity" between a pair of MRs. The word "diversity" in this paper pertains to the differences in the types of faults that two MRs are inclined to detect. The experimental findings indicate that by harnessing posterior knowledge, specifically by analyzing the MUT Model, it is possible to derive prior knowledge that can aid in the construction of Metamorphic Relations. Most importantly, the MUT Model may draw conclusions that violate intuition, exposing more details of the core essence of MR Diversity. Moreover, the concept of MR Diversity serves as a basis for MR selection, resulting in enhanced fault detection capabilities compared to the conventional MS-based approach. Additionally, it offers valuable insights into the construction of composite metamorphic relations, with the goal of amplifying their fault detection abilities beyond those of their individual MR components.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Algorithm 1
Algorithm 2
Algorithm 3
Algorithm 4
Fig. 9
Fig. 10
Fig. 11

Similar content being viewed by others

Data availability

The data of this study is openly available in Github at https://github.com/VinylLee/MUTModel.

References

  • Asrafi, M., Liu, H., & Kuo, F. C. (2011). On testing effectiveness of metamorphic relations: A case study. 2011 Fifth International Conference on Secure Software Integration and Reliability Improvement (pp. 147–156). https://doi.org/10.1109/SSIRI.2011.21

    Chapter  Google Scholar 

  • Binder, R. V. (1999). Testing object-oriented systems: models, patterns, and tools. Addison-Wesley Longman Publishing Co., Inc.

    Google Scholar 

  • Cao, Y., Zhou, Z. Q., & Chen, T. Y. (2013). On the correlation between the effectiveness of metamorphic relations and dissimilarities of test case executions. 2013 13th International Conference on Quality Software (pp. 153–162). https://doi.org/10.1109/QSIC.2013.43

    Chapter  Google Scholar 

  • Chan, F. T., Chen, T. Y., Cheung, S. C., Lau, M. F., & Yiu, S. M. (1998). Application of metamorphic testing in numerical analysis. Proceedings of the IASTED International Conference on Software Engineering (SE’98).

  • Chen, T. Y., Huang, D. H., Tse, T. H., & Zhou, Z. Q. (2004). Case studies on the selection of useful relations in metamorphic testing. Proceedings of the 4th Ibero-American Symposium on Software Engineering and Knowledge Engineering (JIISIC 2004) (pp. 569–583). Citeseer

  • Chen, T. Y., Cheung, S. C., & Yiu, S. M. (2020). Metamorphic testing: a new approach for generating next test cases. arXiv preprint. arXiv:2002.12543.

  • Chen, T. Y., Kuo, F.-C., Liu, H., Poon, P.-L., Towey, D., Tse, T. H., & Zhou, Z. Q. (2018). Metamorphic testing: A review of challenges and opportunities. ACM Computing Surveys. https://doi.org/10.1145/3143561

    Article  Google Scholar 

  • Chen, T. Y., Poon, P.-L., & Xie, X. (2016). Metric: Metamorphic relation identification based on the category-choice framework. Journal of Systems and Software, 116, 177–190. https://doi.org/10.1016/j.jss.2015.07.037

    Article  Google Scholar 

  • Chen, T. Y., Tse, T. H., & Zhiquan, Z. (2001). Fault-based testing in the absence of an oracle. 25th Annual International Computer Software and Applications Conference (pp. 172–178). COMPSAC 2001. https://doi.org/10.1109/CMPSAC.2001.960614

    Chapter  Google Scholar 

  • Dong, G., Xu, B., Chen, L., Nie, C., & Wang, L. (2008). Case studies on testing with compositional metamorphic relations. Journal of Southeast University (English Edition), 24, 437–443.

    Google Scholar 

  • Hoffman, D. (1999). Heuristic test oracles for automated testing, expected results are generated using a test oracle. a look at how heuristic oracles can strike a balance between exhaustive comparison and no comparison at all. Software Testing and Quality Engineering, 1, 28–41.

    Google Scholar 

  • Liu, H., Kuo, F.-C., Towey, D., & Chen, T. Y. (2014). How effectively does metamorphic testing alleviate the oracle problem? IEEE Transactions on Software Engineering, 40(1), 4–22. https://doi.org/10.1109/TSE.2013.46

    Article  Google Scholar 

  • Liu, H., Liu, X., & Chen, T. Y. (2012). A new method for constructing metamorphic relations. 2012 12th International Conference on Quality Software (pp. 59–68). https://doi.org/10.1109/QSIC.2012.10

    Chapter  Google Scholar 

  • Liu, S., Yan, S., & Yang, X. (2022). A similarity-based metamorphic relations selection strategy for numerical computation programs. 2022 4th International Conference on Frontiers Technology of Information and Computer (ICFTIC) (pp. 290–294). https://doi.org/10.1109/ICFTIC57696.2022.10075300

    Chapter  Google Scholar 

  • Mayer, J., & Guderlei, R. (2006). An empirical study on the selection of good metamorphic relations. 30th Annual International Computer Software and Applications Conference (COMPSAC’06) (Vol. 1, pp. 475–484). https://doi.org/10.1109/COMPSAC.2006.24

    Chapter  Google Scholar 

  • Orso, A., & Rothermel, G. (2014). Software testing: a research travelogue (2000–2014). Future of Software Engineering Proceedings (pp. 117–132). https://doi.org/10.1145/2593882.2593885

  • Qiu, K., Zheng, Z., Chen, T. Y., & Poon, P.-L. (2022). Theoretical and empirical analyses of the effectiveness of metamorphic relation composition. IEEE Transactions on Software Engineering, 48(3), 1001–1017. https://doi.org/10.1109/TSE.2020.3009698

    Article  Google Scholar 

  • Rong, W., & Kerong, B. (2012). Researches on basic criterion and strategy of constructing metamorphic relations. Computer Science, 39(001), 115–119.

    Google Scholar 

  • Sadi, M. S., Kuo, F., Ho, J. W. K., Charleston, M. A., & Chen, T. Y. (2011). Verification of phylogenetic inference programs using metamorphic testing. Journal of Bioinformatics and Computational Biology, 9(6), 729–747. https://doi.org/10.1142/S021972001100563X

    Article  Google Scholar 

  • Weyuker, E. J. (1982). On testing non-testable programs. The Computer Journal, 25(4), 465–470. https://doi.org/10.1093/comjnl/25.4.465

    Article  Google Scholar 

  • Wu, P., Xiao-Chun, S. H. I., Jiang-Jun, T., & Hui-Min, L. I. N. (2005). Metamorphic testing and special case testing: A case study. Journal of Software. https://doi.org/10.1360/jos161210

    Article  Google Scholar 

  • Yan, L., Sheng-ming, P., Cheng, W., Kang-wei, W., Tian, W., & Xiao-dong, X. (2016). Sensitivity of metamorphic relationships and its cluster analysis. Acta Electonica Sinica, 44(5), 1196–1201. https://doi.org/10.3969/j.issn.0372-2112.2016.05.026

    Article  Google Scholar 

  • Xie, X., Wong, W. E., Chen, T. Y., & Xu, B. (2013). Metamorphic slice: An application in spectrum-based fault localization. Information and Software Technology, 55(5), 866–879. https://doi.org/10.1016/j.infsof.2012.08.008

    Article  Google Scholar 

  • Xie, X., Li, J., Wang, C., & Chen, T. Y. (2016). Looking for an mr? try metwiki today. 2016 IEEE/ACM 1st International Workshop on Metamorphic Testing (MET) (pp. 1–4). https://doi.org/10.1145/2896971.2896976

    Chapter  Google Scholar 

Download references

Funding

This work was partly supported by the National Natural Science Foundation of China (NSFC) (Grant nos. 62172194, 62202206 and U1836116), the Natural Science Foundation of Jiangsu Province, China (Grant no. BK20220515), the Leading edge Technology Program of Jiangsu Natural Science Foundation, China (Grant no. BK20202001), the China Postdoctoral Science Foundation, China (Grant no. 2021M691310), and Qinglan Project of Jiangsu Province, China.

Author information

Authors and Affiliations

Authors

Contributions

Xiaodong Xie: Investigation, Methodology, Writing - original draft. Zhehao Li: Methodology, Writing - original draft. Jinfu Chen: Investigation, Data curation. Yue Zhang: Software, Validation, Supervision, Writing - review and editing. Xiangxiang Wang: Data curation, Visualization. Patrick Kwaku Kudjo: Writing - review and editing.

Corresponding author

Correspondence to Jinfu Chen.

Ethics declarations

Conflicts of interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Appendix 1: Long table

Appendix 1: Long table

The Metamorphic Relations of Earth Distance (ED), Sin, Sparse Matrix Multiplication algorithm (SMM), and DSPA are placed here due to their length. The result of metamorphic relation (MR) clustering is presented here for the same result. We strive to offer comprehensive explanations for all MRs. Nevertheless, due to the complexity inherent in certain programs, such as SMM and DNA, it becomes challenging to provide exhaustive explanations or test case illustrations. For the detailed exploration of the MRs pertaining to DNA, we refer to previous researches (Sadi et al., 2011; Qiu et al., 2022). In the case of SMM, we will utilize a \(2\times 2\) matrix as an example to demonstrate the MRs, assuming the original input is represented by Eq. 8.

$$\begin{aligned} A^s = \begin{bmatrix} a_{11} &{} a_{12}\\ a_{21} &{} a_{22} \end{bmatrix}, B^s = \begin{bmatrix} b_{11} &{} b_{12} \\ b_{21} &{} b_{22} \end{bmatrix}, C^s = A^s \cdot B^s = \begin{bmatrix} c_{11} &{} c_{12} \\ c_{21} &{} c_{22} \end{bmatrix} \end{aligned}$$
(8)
Table 4 Metamorphic relations of earth distance
Table 5 Metamorphic relations of SIN
Table 6 Metamorphic relations of DSPA
Table 7 Metamorphic relations of Dnapars
Table 8 Metamorphic relations of SMM
Table 9 Clustering result

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Xie, X., Li, Z., Chen, J. et al. MUT Model: a metric for characterizing metamorphic relations diversity. Software Qual J 32, 1413–1455 (2024). https://doi.org/10.1007/s11219-024-09689-x

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11219-024-09689-x

Keywords

Navigation