research-article

Search-based inference of polynomial metamorphic relations

Authors:

Hong MeiAuthors Info & Claims

ASE '14: Proceedings of the 29th ACM/IEEE International Conference on Automated Software Engineering

Pages 701 - 712

https://doi.org/10.1145/2642937.2642994

Published: 15 September 2014 Publication History

Abstract

Metamorphic testing (MT) is an effective methodology for testing those so-called ``non-testable'' programs (e.g., scientific programs), where it is sometimes very difficult for testers to know whether the outputs are correct. In metamorphic testing, metamorphic relations (MRs) (which specify how particular changes to the input of the program under test would change the output) play an essential role. However, testers may typically have to obtain MRs manually.

In this paper, we propose a search-based approach to automatic inference of polynomial MRs for a program under test. In particular, we use a set of parameters to represent a particular class of MRs, which we refer to as polynomial MRs, and turn the problem of inferring MRs into a problem of searching for suitable values of the parameters. We then dynamically analyze multiple executions of the program, and use particle swarm optimization to solve the search problem. To improve the quality of inferred MRs, we further use MR filtering to remove some inferred MRs.

We also conducted three empirical studies to evaluate our approach using four scientific libraries (including 189 scientific functions). From our empirical results, our approach is able to infer many high-quality MRs in acceptable time (i.e., from 9.87 seconds to 1231.16 seconds), which are effective in detecting faults with no false detection.

References

[1]

E. Alba and F. Chicano. Management of software projects with gas. In Proc. MIC, pages 13--18, 2005.

[2]

J. H. Andrews, L. C. Briand, and Y. Labiche. Is mutation an appropriate tool for testing experiments? In Proc. ICSE, pages 402--411, 2005.

Digital Library

[3]

J. H. Andrews, T. Menzies, and F. C. Li. Genetic algorithms for randomized unit testing. IEEE Transactions on Software Engineering, 37(1):80--94, 2011.

Digital Library

[4]

A. Baars, M. Harman, Y. Hassoun, K. Lakhotia, P. McMinn, P. Tonella, and T. Vos. Symbolic search-based testing. In Proc. ASE, pages 53--62, 2011.

Digital Library

[5]

E. T. Barr, T. Vo, V. Le, and Z. Su. Automatic detection of floating-point exceptions. In Proc. POPL, pages 549--560, 2013.

Digital Library

[6]

S. Bensalem, M. Bozga, B. Boyer, and A. Legay. Incremental generation of linear invariants for component-based systems. In Proc. ACSD, pages 80--89, 2013.

Digital Library

[7]

F. Benz, A. Hildebrandt, and S. Hack. A dynamic program analysis to find floating-point accuracy problems. In Proc. PLDI, pages 453--462, 2012.

Digital Library

[8]

A. Bertolino. Software testing research: Achievements, challenges, dreams. In Proc. FOSE, pages 85--103, 2007.

Digital Library

[9]

J. Cabot, R. Clarisó, E. Guerra, and J. De Lara. An invariant-based method for the analysis of declarative model-to-model transformations. In Proc. MODELS, pages 37--52, 2008.

Digital Library

[10]

G. Canfora and M. Di Penta. New frontiers of reverse engineering. In Proc. FOSE, pages 326--341, 2007.

Digital Library

[11]

T. Y. Chen, S. C. Cheung, and S. M. Yiu. Metamorphic testing: A new approach for generating next test cases. Technical Report HKUST-CS98-01, Hong Kong University of Science and Technology, 1998.

[12]

T. Y. Chen, J. Feng, and T. H. Tse. Metamorphic testing of programs on partial differential equations: A case study. In Proc. COMPSAC, pages 327--333, 2002.

Digital Library

[13]

T. Y. Chen, D. H. Huang, T. H. Tse, and Z. Q. Zhou. Case studies on the selection of useful relations in metamorphic testing. In Proc. JIISIC, pages 569--583, 2004.

[14]

T. Y. Chen, F.-C. Kuo, T. H. Tse, and Z. Q. Zhou. Metamorphic testing and beyond. In Proc. STEP, pages 94--100, 2003.

Digital Library

[15]

T. Y. Chen, T. H. Tse, and Z. Q. Zhou. Fault-based testing without the need of oracles. Information and Software Technology, 45(1):1--9, 2003.

Digital Library

[16]

C. Csallner, N. Tillmann, and Y. Smaragdakis. Dysy: Dynamic symbolic execution for invariant inference. In Proc. ICSE, pages 281--290, 2008.

Digital Library

[17]

E. Di Nitto, M. Di Penta, A. Gambi, G. Ripa, and M. L. Villani. Negotiation of service level agreements: An architecture and a search-based approach. In Proc. ICSOC, pages 295--306, 2007.

Digital Library

[18]

M. Di Penta, M. Harman, and G. Antoniol. The use of search-based optimization techniques to schedule and staff software projects: An approach and an empirical study. Software: Practice and Experience, 41(5):495--519, 2011.

Digital Library

[19]

J. J. Dolado. A validation of the component-based method for software size estimation. IEEE Transactions on Software Engineering, 26(10):1006--1021, 2000.

Digital Library

[20]

M. D. Ernst. Dynamically discovering likely program invariants. PhD thesis, University of Washington, 2000.

Digital Library

[21]

M. D. Ernst, J. H. Perkins, P. J. Guo, S. McCamant, C. Pacheco, M. S. Tschantz, and C. Xiao. The Daikon system for dynamic detection of likely invariants. Science of Computer Programming, 69(1):35--45, 2007.

Digital Library

[22]

I. M. Gelfand and M. Saul. Trigonometry. Springer, 2001.

[23]

P. Godefroid and J. Kinder. Proving memory safety of floating-point computations by combining static and dynamic program analysis. In Proc. ISSTA, pages 1--12, 2010.

Digital Library

[24]

F. Gross, G. Fraser, and A. Zeller. Search-based system testing: high coverage, no false alarms. In Proc. ISSTA, pages 67--77, 2012.

Digital Library

[25]

M. Harman. The current state and future of search based software engineering. In Proc. FOSE, pages 342--357, 2007.

Digital Library

[26]

M. Harman and B. F. Jones. Search-based software engineering. Information and Software Technology, 43(14):833--839, 2001.

[27]

M. Harman, J. Krinke, J. Ren, and S. Yoo. Search based data sensitivity analysis applied to requirement engineering. In Proc. GECCO, pages 1681--1688, 2009.

Digital Library

[28]

M. Harman, P. McMinn, M. Shahbaz, and S. Yoo. A comprehensive survey of trends in oracles for software testing. Technical Report CS-13-01, University of Shefield, 2013.

[29]

M. Harman and L. Tratt. Pareto optimal search based refactoring at the design level. In Proc. GECCO, pages 1106--1113, 2007.

Digital Library

[30]

S. Hayashi, Y. Tsuda, and M. Saeki. Search-based refactoring detection from source code revisions. IEICE Transactions on Information and Systems, 93(4):754--762, 2010.

[31]

S. Huang, M. B. Cohen, and A. M. Memon. Repairing GUI test suites using a genetic algorithm. In Proc. ICST, pages 245--254, 2010.

Digital Library

[32]

G. Jiang, H. Chen, and K. Yoshihira. Discovering likely invariants of distributed transaction systems for autonomic system management. In Proc. ICAC, pages 199--208, 2006.

Digital Library

[33]

U. Kanewala and J. M. Bieman. Using machine learning techniques to detect metamorphic relations for program without test oracles. In Proc. ISSRE, pages 1--10, 2013.

[34]

J. Kennedy and R. C. Eberhart. Particle swarm optimization. In Encyclopedia of Machine Learning, pages 1942--1948, 1995.

[35]

I. Krka, Y. Brun, D. Popescu, J. Garcia, and N. Medvidovic. Using dynamic execution traces and program invariants to enhance behavioral model inference. In Proc. ICSE, pages 179--182, 2010.

Digital Library

[36]

J. J. Liang, A. K. Qin, P. N. Suganthan, and S. Baskar. Comprehensive learning particle swarm optimizer for global optimization of multimodal functions. IEEE Transactions on Evolutionary Computation, 10(3):281--295, 2006.

Digital Library

[37]

W. Lin, M. Wu, Z. Yang, and Z. Zeng. Exact safety verification of hybrid systems using sums-of-squares representation. Science China Information Sciences, 57(5):1--13, 2014.

[38]

H. Liu, X. Liu, and T. Y. Chen. A new method for constructing metamorphic relations. In Proc. QSIC, pages 59--68, 2012.

Digital Library

[39]

M. T. Llano, A. Ireland, and A. Pease. Discovery of invariants through automated theory formation. Formal Aspects of Computing, 26(2):203--249, 2014.

Digital Library

[40]

D. Lo and S. Maoz. Mining scenario-based specifications with value-based invariants. In Proc. OOPSLA, pages 755--756, 2009.

Digital Library

[41]

R. E. Lopez-Herrejon and A. Egyed. Sbse4vm: Search based software engineering for variability management. In Proc. CSMR, pages 441--444, 2013.

Digital Library

[42]

M. Z. Malik, A. Pervaiz, and S. Khurshid. Generating representation invariants of structurally complex data. In Proc. TACAS, pages 34--49, 2007.

Digital Library

[43]

N. Mansour, R. Bahsoon, and G. Baradhi. Empirical comparison of regression test selection algorithms. Journal of Systems and Software, 57(1):79--90, 2001.

[44]

J. Mayer and R. Guderlei. An empirical study on the selection of good metamorphic relations. In Proc. COMPSAC, pages 475--484, 2006.

Digital Library

[45]

J. Mayer and R. Guderlei. An empirical study on the selection of good metamorphic relations. In Proc. COMPSAC, pages 475--484, 2006.

Digital Library

[46]

H. Mei, D. Hao, L. Zhang, L. Zhang, J. Zhou, and G. Rothermel. A static approach to prioritizing junit test cases. IEEE Transactions on Software Engineering, 38(6):1258--1275, 2012.

Digital Library

[47]

C. Murphy. Metamorphic testing techniques to detect defects in applications without test oracles. PhD thesis, Columbia University, 2010.

Digital Library

[48]

C. Murphy, K. Shen, and G. Kaiser. Automatic system testing of programs without test oracles. In Proc. ISSTA, pages 189--200, 2009.

Digital Library

[49]

T. Nguyen, D. Kapur, W. Weimer, and S. Forrest. Using dynamic analysis to discover polynomial and array invariants. In Proc. ICSE, pages 683--693, 2012.

Digital Library

[50]

T. Nguyen, D. Kapur, W. Weimer, and S. Forrest. Using dynamic analysis to generate disjunctive invariants. In Proc. ICSE, pages 608--619, 2014.

Digital Library

[51]

N. Pan, F. Zeng, and Y.-H. Huang. Test case reduction based on program invariant and genetic algorithm. In Proc. WiCOM, pages 1--5, 2010.

[52]

C. S. Pasareanu and W. Visser. Verification of java programs using symbolic execution and invariant generation. In Proc. SPIN, pages 164--181, 2004.

[53]

Y. Pavlov and G. Fraser. Semi-automatic search-based test generation. In Proc. ICST, pages 777--784, 2012.

Digital Library

[54]

R. Poli, J. Kennedy, and T. Blackwell. Particle swarm optimization. Swarm Intelligence, 1(1):33--57, 2007.

[55]

F. Qayum and R. Heckel. Search-based refactoring using unfolding of graph transformation systems. Electronic Communications of the EASST, 38, 2011.

[56]

D. Romano, M. Di Penta, and G. Antoniol. An approach for search based testing of null pointer exceptions. In Proc. ICST, pages 160--169, 2011.

Digital Library

[57]

C. Ryan. Automatic re-engineering of software using genetic programming, volume 2. Springer, 2000.

Digital Library

[58]

S. Segura, R. M. Hierons, D. Benavides, and A. Ruiz-Cortés. Automated test data generation on the analyses of feature models: a metamorphic testing approach. In Proc. ICST, pages 35--44, 2010.

Digital Library

[59]

S. Segura, R. M. Hierons, D. Benavides, and A. Ruiz-Cortés. Automated metamorphic testing on the analyses of feature models. Information and Software Technology, 53(3):245--258, 2011.

Digital Library

[60]

Y. Shi and R. C. Eberhart. Empirical study of particle swarm optimization. In Proc. CEC, pages 1945--1950, 1999.

[61]

Y. Shi, H. Liu, L. Gao, and G. Zhang. Cellular particle swarm optimization. Information Sciences, 181(20):4460--4493, 2011.

Digital Library

[62]

B. H. Smith and L. Williams. On guiding the augmentation of an automated test suite via mutation analysis. Empirical Software Engineering, 14(3):341--369, 2009.

Digital Library

[63]

E. Tang, E. Barr, X. Li, and Z. Su. Perturbing numerical calculations for statistical analysis of floating-point program (in)stability. In Proc. ISSTA, pages 131--142, 2010.

Digital Library

[64]

P. Tonella, A. Susi, and F. Palma. Interactive requirements prioritization using a genetic algorithm. Information and Software Technology, 55(1):173--187, 2013.

Digital Library

[65]

S. Wang, D. Lo, L. Jiang, and H. C. Lau. Search-based fault localization. In Proc. ASE, pages 556--559, 2011.

Digital Library

[66]

E. J. Weyuker. On testing non-testable programs. The Computer Journal, 25(4):465--470, 1982.

[67]

A. Windisch, S. Wappler, and J. Wegener. Applying particle swarm optimization to software testing. In Proc. GECCO, pages 1121--1128, 2007.

Digital Library

[68]

W.K.Chan, S.C.Cheung, and K. R.P.H.Leung. Towards a metamorphic testing methodology for service-oriented software applications. In Proc. QSIC, pages 470--476, 2005.

Digital Library

[69]

W. E. Wong, J. R. Horgan, S. London, and H. Agrawal. A study of effective regression testing in practice. In Proc. ISSRE, pages 264--274, 1997.

Digital Library

[70]

P. Wu. Iterative metamorphic testing. In Proc. COMPSAC, pages 19--24, 2005.

Digital Library

[71]

X. Xie, J. Ho, C. Murphy, G. Kaiser, B. Xu, and T. Y. Chen. Application of metamorphic testing to supervised classifiers. In Proc. QSIC, pages 135--144, 2009.

Digital Library

[72]

Z. Xu, M. B. Cohen, and G. Rothermel. Factors affecting the use of genetic algorithms in test suite augmentation. In Proc. GECCO, pages 1365--1372, 2010.

Digital Library

[73]

J. Yang, D. Evans, D. Bhardwaj, T. Bhat, and M. Das. Perracotta: mining temporal API rules from imperfect traces. In Proc. of ICSE, pages 282--291, 2006.

Digital Library

[74]

S. Yoo, M. Harman, and S. Ur. GPGPU test suite minimisation: search based software engineering performance improvement using graphics cards. Empirical Software Engineering, 18(3):550--593, 2013.

[75]

Y. Yuan, Z. Fanping, Z. Guanmiao, D. Chaoqiang, and X. Neng. Test case generation based on program invariant and adaptive random algorithm. In Proc. CSE, pages 274--282, 2011.

[76]

A. Zeller. Search-based program analysis. In Proc. SSBSE, pages 1--4, 2011.

Digital Library

[77]

L. Zhang, D. Hao, L. Zhang, G. Rothermel, and H. Mei. Bridging the gap between the total and additional test-case prioritization strategies. In Proc. ICSE, pages 192--201. IEEE, 2013.

Digital Library

[78]

L. Zhang, D. Marinov, L. Zhang, and S. Khurshid. Regression mutation testing. In Proc. ISSTA, pages 331--341, 2012.

Digital Library

[79]

L. Zhang, G. Yang, N. Rungta, S. Person, and S. Khurshid. Feedback-driven dynamic invariant discovery. In Proc. ISSTA, pages 362--372, 2014.

Digital Library

[80]

J. Zhao, C. Han, and B. Wei. Binary particle swarm optimization with multiple evolutionary strategies. Science China Information Sciences, 55(11):2485--2494, 2012.

[81]

H. Zhong, T. Xie, L. Zhang, J. Pei, and H. Mei. MAPO: Mining and recommending API usage patterns. In Proc. of ECOOP, pages 318--343, 2009.

Digital Library

[82]

H. Zhong, L. Zhang, T. Xie, and H. Mei. Inferring resource specifications from natural language API documentation. In Proc. of ASE, pages 307--318, 2009.

Digital Library

[83]

Z. Q. Zhou, D. H. Huang, T. H. Tse, Z. Yang, H. Huang, and T. Y. Chen. Metamorphic testing and its applications. In Proc. ISFST, pages 346--351, 2004.

Cited By

Xu CChen SWu JCheung STerragni VZhu HCao JFilkov VRay BZhou M(2024)MR-Adopt: Automatic Deduction of Input Transformation Function for Metamorphic TestingProceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering10.1145/3691620.3696020(557-569)Online publication date: 27-Oct-2024
https://dl.acm.org/doi/10.1145/3691620.3696020
Pedram SLabiche YLiu HAleti AArrieta A(2024)Using Category Partition to Detect Metamorphic RelationsProceedings of the 9th ACM International Workshop on Metamorphic Testing10.1145/3679006.3685068(10-17)Online publication date: 13-Sep-2024
https://dl.acm.org/doi/10.1145/3679006.3685068
Alshahwan NHarman MMarginean ATal RWang Ed'Amorim M(2024)Observation-Based Unit Test Generation at MetaCompanion Proceedings of the 32nd ACM International Conference on the Foundations of Software Engineering10.1145/3663529.3663838(173-184)Online publication date: 10-Jul-2024
https://dl.acm.org/doi/10.1145/3663529.3663838
Show More Cited By

Index Terms

Search-based inference of polynomial metamorphic relations
1. Software and its engineering
  1. Software creation and management
    1. Software verification and validation
      1. Software defect analysis
        Software testing and debugging

Recommendations

Feedback-Directed Metamorphic Testing
Over the past decade, metamorphic testing has gained rapidly increasing attention from both academia and industry, particularly thanks to its high efficacy on revealing real-life software faults in a wide variety of application domains. On the basis of a ...
Metamorphic Testing: A Review of Challenges and Opportunities

Metamorphic testing is an approach to both test case generation and test result verification. A central element is a set of metamorphic relations, which are necessary properties of the target function or algorithm in relation to multiple inputs and ...
A New Method for Constructing Metamorphic Relations
QSIC '12: Proceedings of the 2012 12th International Conference on Quality Software

A fundamental problem for software testing is the oracle problem, which means that in many practical situations, it is extremely expensive, if not impossible, to verify the test result given any possible program input. Metamorphic testing is an approach ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

ASE '14: Proceedings of the 29th ACM/IEEE International Conference on Automated Software Engineering

September 2014

934 pages

ISBN:9781450330138

DOI:10.1145/2642937

General Chair:
Ivica Crnkovic
Mälardalen University, Sweden
,
Program Chairs:
Marsha Chechik
University of Toronto, Canada
,
Paul Grünbacher
Johannes Kepler Universität Linz, Austria

Copyright © 2014 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGAI: ACM Special Interest Group on Artificial Intelligence
SIGSOFT: ACM Special Interest Group on Software Engineering
Mälardalen University: Mälardalen University
IEEE CS

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 15 September 2014

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

Conference

ASE '14

Sponsor:

SIGAI
SIGSOFT
Mälardalen University

ASE '14: ACM/IEEE International Conference on Automated Software Engineering

September 15 - 19, 2014

Vasteras, Sweden

Acceptance Rates

ASE '14 Paper Acceptance Rate 82 of 337 submissions, 24%;

Overall Acceptance Rate 82 of 337 submissions, 24%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

92
Total Citations
View Citations
724
Total Downloads

Downloads (Last 12 months)47
Downloads (Last 6 weeks)5

Reflects downloads up to 28 Feb 2025

Other Metrics

View Author Metrics

Citations

Cited By

Xu CChen SWu JCheung STerragni VZhu HCao JFilkov VRay BZhou M(2024)MR-Adopt: Automatic Deduction of Input Transformation Function for Metamorphic TestingProceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering10.1145/3691620.3696020(557-569)Online publication date: 27-Oct-2024
https://dl.acm.org/doi/10.1145/3691620.3696020
Pedram SLabiche YLiu HAleti AArrieta A(2024)Using Category Partition to Detect Metamorphic RelationsProceedings of the 9th ACM International Workshop on Metamorphic Testing10.1145/3679006.3685068(10-17)Online publication date: 13-Sep-2024
https://dl.acm.org/doi/10.1145/3679006.3685068
Alshahwan NHarman MMarginean ATal RWang Ed'Amorim M(2024)Observation-Based Unit Test Generation at MetaCompanion Proceedings of the 32nd ACM International Conference on the Foundations of Software Engineering10.1145/3663529.3663838(173-184)Online publication date: 10-Jul-2024
https://dl.acm.org/doi/10.1145/3663529.3663838
Xu CTerragni VZhu HWu JCheung S(2024)MR-Scout: Automated Synthesis of Metamorphic Relations from Existing Test CasesACM Transactions on Software Engineering and Methodology10.1145/365634033:6(1-28)Online publication date: 29-Jun-2024
https://dl.acm.org/doi/10.1145/3656340
Nolasco AMolina FDegiovanni RGorla AGarbervetsky DPapadakis MUchitel SAguirre NFrias M(2024)Abstraction-Aware Inference of Metamorphic RelationsProceedings of the ACM on Software Engineering10.1145/36437471:FSE(450-472)Online publication date: 12-Jul-2024
https://dl.acm.org/doi/10.1145/3643747
Zhang QZhai JFang CLiu JSun WHu HWang Q(2024)Machine Translation Testing via Syntactic Tree PruningACM Transactions on Software Engineering and Methodology10.1145/364032933:5(1-39)Online publication date: 4-Jun-2024
https://dl.acm.org/doi/10.1145/3640329
Ayerdi JTerragni VJahangirova GArrieta ATonella P(2024)GenMorph: Automatically Generating Metamorphic Relations via Genetic ProgrammingIEEE Transactions on Software Engineering10.1109/TSE.2024.340784050:7(1888-1900)Online publication date: 1-Jul-2024
https://dl.acm.org/doi/10.1109/TSE.2024.3407840
Huang HZhou APayer MZhang C(2024)Everything is Good for Something: Counterexample-Guided Directed Fuzzing via Likely Invariant Inference2024 IEEE Symposium on Security and Privacy (SP)10.1109/SP54263.2024.00142(1956-1973)Online publication date: 19-May-2024
https://doi.org/10.1109/SP54263.2024.00142
Zhao YLi MSu YZi KZeng C(2024)MetBench: A Numerical Expression Metamorphic Relations Benchmark Data Set2024 11th International Conference on Dependable Systems and Their Applications (DSA)10.1109/DSA63982.2024.00026(120-131)Online publication date: 2-Nov-2024
https://doi.org/10.1109/DSA63982.2024.00026
Bose DDavid-John BBrown C(2024)Optimizing AR Application Testing: Integrating Metamorphic Testing to Address Developer and End-User ChallengesHCI International 2024 – Late Breaking Papers10.1007/978-3-031-76812-5_2(20-33)Online publication date: 15-Dec-2024
https://doi.org/10.1007/978-3-031-76812-5_2
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten