skip to main content
10.1145/3468264.3473920acmconferencesArticle/Chapter ViewAbstractPublication PagesfseConference Proceedingsconference-collections
research-article

Generating metamorphic relations for cyber-physical systems with genetic programming: an industrial case study

Published: 18 August 2021 Publication History

Abstract

One of the major challenges in the verification of complex industrial Cyber-Physical Systems is the difficulty of determining whether a particular system output or behaviour is correct or not, the so-called test oracle problem. Metamorphic testing alleviates the oracle problem by reasoning on the relations that are expected to hold among multiple executions of the system under test, which are known as Metamorphic Relations (MRs). However, the development of effective MRs is often challenging and requires the involvement of domain experts. In this paper, we present a case study aiming at automating this process. To this end, we implemented GAssertMRs, a tool to automatically generate MRs with genetic programming. We assess the cost-effectiveness of this tool in the context of an industrial case study from the elevation domain. Our experimental results show that in most cases GAssertMRs outperforms the other baselines, including manually generated MRs developed with the help of domain experts. We then describe the lessons learned from our experiments and we outline the future work for the adoption of this technique by industrial practitioners.

References

[1]
Hiralal Agrawal, Richard DeMillo, R Hathaway, William Hsu, Wynne Hsu, Edward W Krauser, Rhonda J Martin, Aditya P Mathur, and Eugene Spafford. 1989. Design of mutant operators for the C programming language. Technical Report SERC-TR-41-P, Software Engineering Research Center, Purdue ….
[2]
John Ahlgren, Maria Eugenia Berezin, Kinga Bojarczuk, Elena Dulskyte, Inna Dvortsova, Johann George, Natalija Gucevska, Mark Harman, Maria Lomeli, and Erik Meijer. 2021. Testing Web Enabled Simulation at Scale Using Metamorphic Testing. In Proceedings of the 43rd International Conference on Software Engineering. https://doi.org/10.1109/ICSE-SEIP52600.2021.00023
[3]
Rajeev Alur. 2015. Principles of cyber-physical systems. MIT Press.
[4]
Aitor Arrieta, Shuai Wang, Goiuria Sagardui, and Leire Etxeberria. 2019. Search-Based test case prioritization for simulation-Based testing of cyber-Physical system product lines. Journal of Systems and Software, 149 (2019), 1–34. https://doi.org/10.1016/j.jss.2018.09.055
[5]
Jon Ayerdi, Sergio Segura, Aitor Arrieta, Goiuria Sagardui, and Maite Arratibel. 2020. QoS-aware Metamorphic Testing: An Elevation Case Study. In 2020 IEEE 31st International Symposium on Software Reliability Engineering (ISSRE). IEEE, 104–114. https://doi.org/10.1109/ISSRE5003.2020.00019
[6]
Jon Ayerdi, Valerio Terragni, Aitor Arrieta, Paolo Tonella, Goiuria Sagardui, and Maite Arratibel. 2021. Experimental Data and Results. https://drive.google.com/drive/folders/18XwGyV1tjFkqSvpT-gs-i35uJEnTlEsI Last access: Jul 2021.
[7]
Thomas Back. 1996. Evolutionary algorithms in theory and practice: evolution strategies, evolutionary programming, genetic algorithms. Oxford university press.
[8]
Radhakisan Baheti and Helen Gill. 2011. Cyber-physical systems. The impact of control technology, 12, 1 (2011), 161–166.
[9]
Gina Barney and Lutfi Al-Sharif. 2015. Elevator traffic handbook: theory and practice. Routledge. https://doi.org/10.4324/9780203301333
[10]
Earl T Barr, Mark Harman, Phil McMinn, Muzammil Shahbaz, and Shin Yoo. 2014. The oracle problem in software testing: A survey. IEEE transactions on software engineering, 41, 5 (2014), 507–525. https://doi.org/10.1109/TSE.2014.2372785
[11]
Mohamed Boussaa, Olivier Barais, Gerson Sunyé, and Benoit Baudry. 2020. Leveraging metamorphic testing to automatically detect inconsistencies in code generator families. Software Testing, Verification and Reliability, 30, 1 (2020), e1721. https://doi.org/10.1002/stvr.1721 arxiv:https://onlinelibrary.wiley.com/doi/pdf/10.1002/stvr.1721. e1721 stvr.1721.
[12]
Markus F Brameier and Wolfgang Banzhaf. 2007. A Comparison with Tree-Based Genetic Programming. Linear Genetic Programming, 173–192. https://doi.org/10.1007/978-0-387-31030-5_8
[13]
WK Chan, Tsong Y Chen, Shing Chi Cheung, TH Tse, and Zhenyu Zhang. 2007. Towards the testing of power-aware software applications for wireless sensor networks. In International Conference on Reliable Software Technologies. 84–99. https://doi.org/10.1007/978-3-540-73230-3_7
[14]
T. Y. Chen, S. C. Cheung, and S. M. Yiu. 1998. Metamorphic Testing: A New Approach for Generating Next Test Cases. Technical Report HKUST-CS98-01, Department of Computer Science, The Hong Kong University of Science and Technology.
[15]
T. Y. Chen, D. H. Huang, T. H. Tse, and Z. Q. Zhou. 2004. Case Studies on the Selection of Useful Relations in Metamorphic Testing. In Proceedings of the 4th Ibero-American Symposium on Software Engineering and Knowledge Engineering (JIISIC 2004). 569–583.
[16]
Tsong Yueh Chen, Fei-Ching Kuo, Huai Liu, Pak-Lok Poon, Dave Towey, T. H. Tse, and Zhi Quan Zhou. 2018. Metamorphic Testing: A Review of Challenges and Opportunities. Comput. Surveys, 51, 1 (2018), Article 4, Jan., 27 pages. issn:0360-0300 https://doi.org/10.1145/3143561
[17]
Kalyanmoy Deb, Amrit Pratap, Sameer Agarwal, and TAMT Meyarivan. 2002. A Fast and Elitist Multiobjective Genetic Algorithm: NSGA-II. IEEE Transactions on Evolutionary Computation, 6, 2 (2002), 182–197. https://doi.org/10.1109/4235.996017
[18]
Gunel Jahangirova, David Clark, Mark Harman, and Paolo Tonella. 2016. Test oracle assessment and improvement. In Proceedings of the 25th International Symposium on Software Testing and Analysis. 247–258. https://doi.org/10.1145/2931037.2931062
[19]
Gunel Jahangirova, David Clark, Mark Harman, and Paolo Tonella. 2019. An Empirical Validation of Oracle Improvement. IEEE Transactions on Software Engineering, https://doi.org/10.1109/TSE.2019.2934409
[20]
Yue Jia and Mark Harman. 2010. An analysis and survey of the development of mutation testing. IEEE transactions on software engineering, 37, 5 (2010), 649–678. https://doi.org/10.1109/TSE.2010.62
[21]
O. Johnston, D. Jarman, J. Berry, Z. Q. Zhou, and T. Y. Chen. 2019. Metamorphic Relations for Detection of Performance Anomalies. In 2019 IEEE/ACM 4th International Workshop on Metamorphic Testing (MET). 63–69. https://doi.org/10.1109/MET.2019.00017
[22]
Aaron Kane, Thomas Fuhrman, and Philip Koopman. 2014. Monitor based oracles for cyber-physical system testing: Practical experience report. In 2014 44th Annual IEEE/IFIP International Conference on Dependable Systems and Networks. 148–155. https://doi.org/10.1109/DSN.2014.28
[23]
John R Koza and John R Koza. 1992. Genetic Programming: On the Programming of Computers by Means of Natural Selection. 1, MIT press.
[24]
Edward Ashford Lee and Sanjit A Seshia. 2016. Introduction to embedded systems: A cyber-physical systems approach. Mit Press.
[25]
Mikael Lindvall, Adam Porter, Gudjon Magnusson, and Christoph Schulze. 2017. Metamorphic model-based testing of autonomous systems. In 2017 IEEE/ACM 2nd International Workshop on Metamorphic Testing (MET). 35–41. https://doi.org/10.1109/MET.2017.6
[26]
Bing Liu, Shiva Nejati, and Lionel C Briand. 2017. Improving fault localization for Simulink models using search-based testing and prediction models. In 2017 IEEE 24th International Conference on Software Analysis, Evolution and Reengineering (SANER). 359–370. https://doi.org/10.1109/SANER.2017.7884636
[27]
Reza Matinnejad, Shiva Nejati, Lionel C Briand, and Thomas Bruckmann. 2018. Test generation and test prioritization for simulink models with dynamic behavior. IEEE Transactions on Software Engineering, 45, 9 (2018), 919–944. https://doi.org/10.1109/TSE.2018.2811489
[28]
Brad L. Miller, Brad L. Miller, David E. Goldberg, and David E. Goldberg. 1995. Genetic Algorithms, Tournament Selection, and the Effects of Noise. Complex Systems, 9, 3 (1995), 193–212.
[29]
Orona. 2021. Orona Group. https://www.orona-group.com/ Last access: Jan 2021.
[30]
Mike Papadakis, Yue Jia, Mark Harman, and Yves Le Traon. 2015. Trivial compiler equivalence: A large scale empirical study of a simple, fast and effective equivalent mutant detection technique. In 2015 IEEE/ACM 37th IEEE International Conference on Software Engineering. 1, 936–946. https://doi.org/10.1109/ICSE.2015.103
[31]
Jeanine Romano, Jeffrey. D. Kromrey, Jesse Coraggio, Jeff Skowronek, and Linda Devine. 2006. Exploring methods for evaluating group differences on the NSSE and other surveys: Are the t-test and Cohen’sd indices the most appropriate choices. In annual meeting of the Southern Association for Institutional Research. 1–51.
[32]
S. Segura, G. Fraser, A. Sanchez, and A. Ruiz-Cortés. 2016. A Survey on Metamorphic Testing. IEEE Transactions on Software Engineering, 42, 9 (2016), Sept, 805–824. issn:0098-5589 https://doi.org/10.1109/TSE.2016.2532875
[33]
S. Segura, J.A. Parejo, J. Troya, and A. Ruiz-Cortés. 2018. Metamorphic Testing of RESTful Web APIs. IEEE Transactions on Software Engineering, 44, 11 (2018), Nov, 1083–1099. issn:0098-5589 https://doi.org/10.1109/TSE.2017.2764464
[34]
S. Segura, D. Towey, Z.Q. Zhou, and T.Y. Chen. 2020. Metamorphic Testing: Testing the Untestable. IEEE Software, 37, 3 (2020), 46–53. https://doi.org/10.1109/MS.2018.2875968
[35]
S. Segura, J. Troya, A. Durán, and A. Ruiz-Cortés. 2017. Performance Metamorphic Testing: Motivation and Challenges. In 2017 IEEE/ACM 39th International Conference on Software Engineering: New Ideas and Emerging Technologies Results Track (ICSE-NIER). 7–10. https://doi.org/10.1109/ICSE-NIER.2017.16
[36]
Sergio Segura, Javier Troya, Amador Durán, and Antonio Ruiz-Cortés. 2018. Performance metamorphic testing: A Proof of concept. Information and Software Technology, 98 (2018), 1 – 4. issn:0950-5849 https://doi.org/10.1016/j.infsof.2018.01.013
[37]
Hisashi Tamaki, Hajime Kita, and Shigenobu Kobayashi. 1996. Multi-objective optimization by genetic algorithms: A review. In Proceedings of IEEE international conference on evolutionary computation. 517–522. https://doi.org/10.1109/ICEC.1996.542653
[38]
Valerio Terragni, Gunel Jahangirova, Mauro Pezzè, and Paolo Tonella. 2021. Improving Assertion Oracles with Evolutionary Computation. In Proceedings of the Genetic and Evolutionary Computation Conference, Hot Off the Press track (GECCO 2021). https://doi.org/10.1145/3449726.3462722
[39]
Valerio Terragni, Gunel Jahangirova, Paolo Tonella, and Mauro Pezzè. 2020. Evolutionary Improvement of Assertion Oracles. In Proceedings of the 28th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, ESEC/FSE 2020, Virtual Event, USA, 8–13 November, 2020. 1178–1189. https://doi.org/10.1145/3368089.3409758
[40]
Valerio Terragni, Gunel Jahangirova, Paolo Tonella, and Mauro Pezzè. 2021. GAssert: A Fully Automated Tool to Improve Assertion Oracles. In Proceedings of the 43nd IEEE/ACM International Conference on Software Engineering, Demonstration Track (ICSE 2021). https://doi.org/10.1109/ICSE-Companion52605.2021.00042
[41]
Yuchi Tian, Kexin Pei, Suman Jana, and Baishakhi Ray. 2018. Deeptest: Automated testing of deep-neural-network-driven autonomous cars. In Proceedings of the 40th international conference on software engineering. 303–314. https://doi.org/10.1145/3180155.3180220
[42]
J. Troya, S. Segura, and A. Ruiz-Cortés. 2018. Automated inference of likely metamorphic relations for model transformations. Journal of Systems and Software, 136 (2018), 188 – 208. issn:0164-1212 https://doi.org/10.1016/j.jss.2017.05.043
[43]
Zhenyu Wang, Dave Towey, Zhi Quan Zhou, and Tsong Yueh Chen. 2018. Metamorphic testing for Adobe Analytics data collection JavaScript library. In Proceedings of the 3rd International Workshop on Metamorphic Testing. 34–37. https://doi.org/10.1145/3193977.3193979
[44]
Darrell Whitley. 1994. A Genetic Algorithm Tutorial. Statistics and Computing, 4, 2 (1994), 65–85. https://doi.org/10.1007/BF00175354
[45]
J. Zhang, J. Chen, D. Hao, Y. Xiong, B. Xie, L. Zhang, and H. Mei. 2014. Search-based Inference of Polynomial Metamorphic Relations. In Proceedings of the 29th ACM/IEEE International Conference on Automated Software Engineering (ASE ’14). ACM, New York, NY, USA. 701–712. isbn:978-1-4503-3013-8 https://doi.org/10.1145/2642937.2642994
[46]
Zhi Q Zhou and Liqun Sun. 2019. Metamorphic testing of driverless cars.
[47]
Z. Q. Zhou, L. Sun, T. Y. Chen, and D. Towey. 2018. Metamorphic Relations for Enhancing System Understanding and Use. IEEE Transactions on Software Engineering, issn:0098-5589 https://doi.org/10.1109/TSE.2018.2876433

Cited By

View all
  • (2025)Defining and generating multi-level and uncertainty-wise test oracles for cyber-physical systemsSoftware and Systems Modeling10.1007/s10270-025-01271-8Online publication date: 14-Feb-2025
  • (2024)MR-Adopt: Automatic Deduction of Input Transformation Function for Metamorphic TestingProceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering10.1145/3691620.3696020(557-569)Online publication date: 27-Oct-2024
  • (2024)MET-MAPF: A Metamorphic Testing Approach for Multi-Agent Path Finding AlgorithmsACM Transactions on Software Engineering and Methodology10.1145/366966333:8(1-37)Online publication date: 12-Jun-2024
  • Show More Cited By

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences
ESEC/FSE 2021: Proceedings of the 29th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering
August 2021
1690 pages
ISBN:9781450385626
DOI:10.1145/3468264
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 18 August 2021

Permissions

Request permissions for this article.

Check for updates

Author Tags

  1. cyber physical systems
  2. evolutionary algorithm
  3. genetic programming
  4. metamorphic testing
  5. mutation testing
  6. oracle generation
  7. oracle improvement
  8. quality of service

Qualifiers

  • Research-article

Funding Sources

Conference

ESEC/FSE '21
Sponsor:

Acceptance Rates

Overall Acceptance Rate 112 of 543 submissions, 21%

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)46
  • Downloads (Last 6 weeks)5
Reflects downloads up to 20 Feb 2025

Other Metrics

Citations

Cited By

View all
  • (2025)Defining and generating multi-level and uncertainty-wise test oracles for cyber-physical systemsSoftware and Systems Modeling10.1007/s10270-025-01271-8Online publication date: 14-Feb-2025
  • (2024)MR-Adopt: Automatic Deduction of Input Transformation Function for Metamorphic TestingProceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering10.1145/3691620.3696020(557-569)Online publication date: 27-Oct-2024
  • (2024)MET-MAPF: A Metamorphic Testing Approach for Multi-Agent Path Finding AlgorithmsACM Transactions on Software Engineering and Methodology10.1145/366966333:8(1-37)Online publication date: 12-Jun-2024
  • (2024)Application of Quantum Extreme Learning Machines for QoS Prediction of Elevators’ Software in an Industrial ContextCompanion Proceedings of the 32nd ACM International Conference on the Foundations of Software Engineering10.1145/3663529.3663859(399-410)Online publication date: 10-Jul-2024
  • (2024)MR-Scout: Automated Synthesis of Metamorphic Relations from Existing Test CasesACM Transactions on Software Engineering and Methodology10.1145/365634033:6(1-28)Online publication date: 29-Jun-2024
  • (2024)AIM: Automated Input Set Minimization for Metamorphic Security TestingIEEE Transactions on Software Engineering10.1109/TSE.2024.3488525(1-31)Online publication date: 2024
  • (2024)GenMorph: Automatically Generating Metamorphic Relations via Genetic ProgrammingIEEE Transactions on Software Engineering10.1109/TSE.2024.340784050:7(1888-1900)Online publication date: 1-Jul-2024
  • (2024)Metamorphic Testing of an Autonomous Delivery Robots Scheduler2024 IEEE Conference on Software Testing, Verification and Validation (ICST)10.1109/ICST60714.2024.00040(361-372)Online publication date: 27-May-2024
  • (2024)Towards Generating Executable Metamorphic Relations Using Large Language ModelsQuality of Information and Communications Technology10.1007/978-3-031-70245-7_9(126-141)Online publication date: 11-Sep-2024
  • (2023)Stress Testing Control Loops in Cyber-physical SystemsACM Transactions on Software Engineering and Methodology10.1145/362474233:2(1-58)Online publication date: 20-Sep-2023
  • Show More Cited By

View Options

Login options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Figures

Tables

Media

Share

Share

Share this Publication link

Share on social media