research-article

Mutation-based test generation for quantum programs with multi-objective search

Authors:

Shaukat AliAuthors Info & Claims

GECCO '22: Proceedings of the Genetic and Evolutionary Computation Conference

Pages 1345 - 1353

https://doi.org/10.1145/3512290.3528869

Published: 08 July 2022 Publication History

Abstract

Mutation testing is often used for designing new tests, and involves changing a program in minor ways, which results in mutated versions of the program, i.e., mutants. An effective test suite should find faults (or kill mutants) with a minimum number of test cases, to save resources required for executing test cases. In this paper, in the context of mutation testing for quantum programs, we present a multi-objective and search-based approach (MutTG) to generate the minimum number of test cases killing as many mutants as possible. MutTG tries to estimate the likelihood that a mutant is equivalent, and uses this as a discount factor in the fitness definition to avoid keeping on trying to kill mutants that cannot be killed. We employed NSGA-II as the multi-objective search algorithm. Then, we compared MutTG with another version of the approach that does not use the discount factor in its fitness definition, and with random search (RS), over a set of open-source quantum programs and their mutants of varying complexity. Results show that the discount factor does indeed help in guiding the test generation, as the approach with the discount factor performs better than the one without it.

Supplemental Material

ZIP File

Supplemental material.

Download
5.19 MB

References

[1]

Alan Agresti. 2019. An introduction to categorical data analysis (3 ed.). Wiley-Blackwell.

[2]

Shaukat Ali, Paolo Arcaini, Xinyi Wang, and Tao Yue. 2021. Assessing the Effectiveness of Input and Output Coverage Criteria for Testing Quantum Programs. In 2021 14th IEEE Conference on Software Testing, Verification and Validation (ICST). 13--23.

[3]

Andrea Arcuri and Lionel Briand. 2011. A Practical Guide for Using Statistical Tests to Assess Randomized Algorithms in Software Engineering. In Proceedings of the 33rd International Conference on Software Engineering (Waikiki, Honolulu, HI, USA) (ICSE '11). ACM, New York, NY, USA, 1--10.

Digital Library

[4]

Andrea Arcuri and Gordon Fraser. 2013. Parameter tuning or default values? An empirical investigation in search-based software engineering. Empirical Software Engineering 18 (2013), 594--623.

[5]

Antonio Benítez-Hidalgo, Antonio J. Nebro, José García-Nieto, Izaskun Oregi, and Javier Del Ser. 2019. jMetalPy: A Python framework for multi-objective optimization with metaheuristics. Swarm and Evolutionary Computation 51 (2019), 100598.

[6]

Ethan Bernstein and Umesh Vazirani. 1993. Quantum Complexity Theory. In Proceedings of the Twenty-Fifth Annual ACM Symposium on Theory of Computing (San Diego, California, USA) (STOC '93). Association for Computing Machinery, New York, NY, USA, 11--20.

Digital Library

[7]

Thierry Titcheu Chekam, Mike Papadakis, Yves Le Traon, and Mark Harman. 2017. An Empirical Study on Mutation, Statement and Branch Coverage Fault Revelation That Avoids the Unreliable Clean Program Assumption. In 2017 IEEE/ACM 39th International Conference on Software Engineering (ICSE). 597--608.

Digital Library

[8]

Antonia Estero-Botaro, Antonio García-Domínguez, Juan José Domínguez-Jiménez, Francisco Palomo-Lozano, and Inmaculada Medina-Bulo. 2014. A Framework for Genetic Test-Case Generation for WS-BPEL Compositions. In Proceedings of the 26th IFIP WG 6.1 International Conference on Testing Software and Systems - Volume 8763 (Madrid, Spain) (ICTSS 2014). Springer-Verlag, Berlin, Heidelberg, 1--16.

Digital Library

[9]

Gordon Fraser and Andrea Arcuri. 2015. Achieving Scalable Mutation-Based Generation of Whole Test Suites. Empirical Softw. Engg. 20, 3 (jun 2015), 783--812.

[10]

Gordon Fraser and Andreas Zeller. 2012. Mutation-Driven Generation of Unit Tests and Oracles. IEEE Transactions on Software Engineering 38, 2 (2012), 278--292.

Digital Library

[11]

M. Gimeno-Segovia, N. Harrigan, and E.R. Johnston. 2019. Programming Quantum Computers: Essential Algorithms and Code Samples. O'Reilly Media, Incorporated.

[12]

Mark Harman, Yue Jia, and William B. Langdon. 2011. Strong Higher Order Mutation-Based Test Data Generation. In Proceedings of the 19th ACM SIGSOFT Symposium and the 13th European Conference on Foundations of Software Engineering (Szeged, Hungary) (ESEC/FSE '11). Association for Computing Machinery, New York, NY, USA, 212--222.

Digital Library

[13]

Shahin Honarvar, Mohammad Reza Mousavi, and Rajagopal Nagarajan. 2020. Property-Based Testing of Quantum Programs in Q#. In Proceedings of the IEEE/ACM 42nd International Conference on Software Engineering Workshops (Seoul, Republic of Korea) (ICSEW'20). Association for Computing Machinery, New York, NY, USA, 430--435.

Digital Library

[14]

Nishtha Jatana, Bharti Suri, Sanjay Misra, Prateek Kumar, and Amit Roy Choudhury. 2016. Particle Swarm Based Evolution and Generation of Test Data Using Mutation Testing. In Computational Science and Its Applications - ICCSA 2016. Springer International Publishing, Cham, 585--594.

[15]

Yue Jia and Mark Harman. 2011. An Analysis and Survey of the Development of Mutation Testing. IEEE Transactions on Software Engineering 37, 5 (2011), 649--678.

Digital Library

[16]

Yue Jia, Fan Wu, Mark Harman, and Jens Krinke. 2015. Genetic Improvement Using Higher Order Mutation. In Proceedings of the Companion Publication of the 2015 Annual Conference on Genetic and Evolutionary Computation (Madrid, Spain) (GECCO Companion '15). Association for Computing Machinery, New York, NY, USA, 803--804.

Digital Library

[17]

Miqing Li and Xin Yao. 2019. Quality Evaluation of Solution Sets in Multiobjective Optimisation: A Survey. ACM Comput. Surv. 52, 2, Article 26 (March 2019), 38 pages.

Digital Library

[18]

Jackson A. Prado Lima and Silvia Regina Vergilio. 2018. Search-Based Higher Order Mutation Testing: A Mapping Study. In Proceedings of the III Brazilian Symposium on Systematic and Automated Software Testing (SAO CARLOS, Brazil) (SAST '18). Association for Computing Machinery, New York, NY, USA, 87--96.

Digital Library

[19]

Lech Madeyski, Wojciech Orzeszyna, Richard Torkar, and Mariusz Józala. 2014. Overcoming the Equivalent Mutant Problem: A Systematic Literature Review and a Comparative Experiment of Second Order Mutation. IEEE Transactions on Software Engineering 40, 1 (2014), 23--42.

Digital Library

[20]

Eñaut Mendiluze, Shaukat Ali, Paolo Arcaini, and Tao Yue. 2021. Muskit: A Mutation Analysis Tool for Quantum Software Testing. In 2021 36th IEEE/ACM International Conference on Automated Software Engineering (ASE). 1266--1270.

Digital Library

[21]

Andriy Miranskyy and Lei Zhang. 2019. On Testing Quantum Programs. In Proceedings of the 41st International Conference on Software Engineering: New Ideas and Emerging Results (Montreal, Quebec, Canada) (ICSE-NIER '19). IEEE Press, 57--60.

Digital Library

[22]

Quang Vu Nguyen and Lech Madeyski. 2015. Searching for strongly subsuming higher order mutants by applying multi-objective optimization algorithm. In Advanced Computational Methods for Knowledge Engineering. Springer, 391--402.

[23]

Quang Vu Nguyen and Lech Madeyski. 2016. Empirical evaluation of multiobjective optimization algorithms searching for higher order mutants. Cybernetics and Systems 47, 1-2 (2016), 48--68.

[24]

Quang Vu Nguyen and Lech Madeyski. 2016. Higher order mutation testing to drive development of new test cases: An empirical comparison of three strategies. In Asian Conference on Intelligent Information and Database Systems. Springer, 235--244.

[25]

Mike Papadakis, Marinos Kintis, Jie Zhang, Yue Jia, Yves Le Traon, and Mark Harman. 2019. Chapter Six - Mutation Testing Advances: An Analysis and Survey. Advances in Computers, Vol. 112. Elsevier, 275--378.

[26]

Mike Papadakis and Nicos Malevris. 2013. Searching and generating test inputs for mutation testing. SpringerPlus 2, 1 (2013), 1--12.

[27]

Mike Papadakis, Nicos Malevris, and Maria Kallia. 2010. Towards Automating the Generation of Mutation Tests. In Proceedings of the 5th Workshop on Automation of Software Test (Cape Town, South Africa) (AST '10). Association for Computing Machinery, New York, NY, USA, 111--118.

Digital Library

[28]

Mike Papadakis, Donghwan Shin, Shin Yoo, and Doo-Hwan Bae. 2018. Are Mutation Scores Correlated with Real Fault Detection? A Large Scale Empirical Study on the Relationship Between Mutants and Real Faults. In 2018 IEEE/ACM 40th International Conference on Software Engineering (ICSE). 537--548.

Digital Library

[29]

Shweta Rani and Bharti Suri. 2015. An Approach for Test Data Generation Based on Genetic Algorithm and Delete Mutation Operators. In 2015 Second International Conference on Advances in Computing and Communication Engineering. 714--718.

[30]

C Prakasa Rao and P Govindarajulu. 2015. Genetic algorithm for automatic generation of representative test suite for mutation testing. International Journal of Computer Science and Network Security (IJCSNS) 15, 2 (2015), 11.

[31]

Gregg Rothermel, Mary Jean Harrold, Jeffery von Ronne, and Christie Hong. 2002. Empirical studies of test-suite reduction. Software Testing, Verification and Reliability 12, 4 (2002), 219--249.

[32]

Ke Shang, Hisao Ishibuchi, Linjun He, and Lie Meng Pang. 2021. A Survey on the Hypervolume Indicator in Evolutionary Multiobjective Optimization. IEEE Transactions on Evolutionary Computation 25, 1 (2021), 1--20.

[33]

Francisco Carlos M. Souza, Mike Papadakis, Yves Le Traon, and Márcio E. Delamaro. 2016. Strong Mutation-Based Test Data Generation Using Hill Climbing. In Proceedings of the 9th International Workshop on Search-Based Software Testing (Austin, Texas) (SBST '16). Association for Computing Machinery, New York, NY, USA, 45--54.

Digital Library

[34]

Jiyuan Wang, Fucheng Ma, and Yu Jiang. 2021. Poster: Fuzz Testing of Quantum Program. In 2021 14th IEEE Conference on Software Testing, Verification and Validation (ICST). 466--469.

[35]

Xinyi Wang, Paolo Arcaini, Tao Yue, and Shaukat Ali. 2021. Generating Failing Test Suites for Quantum Programs With Search. In Search-Based Software Engineering, Una-May O'Reilly and Xavier Devroey (Eds.). Springer International Publishing, Cham, 9--25.

Digital Library

[36]

Xinyi Wang, Paolo Arcaini, Tao Yue, and Shaukat Ali. 2021. Quito: a Coverage-Guided Test Generator for Quantum Programs. In 2021 36th IEEE/ACM International Conference on Automated Software Engineering (ASE). 1237--1241.

Digital Library

[37]

Xinyi Wang, Tongxuan Yu, Paolo Arcaini, Tao Yue, and Shaukat Ali. 2022. Mutation-Based Test Generation for Quantum Programs with Multi-Objective Search - Supplementary material. https://github.com/Simula-COMPLEX/MutTG-paper.

[38]

Robert Wille, Rod Van Meter, and Yehuda Naveh. 2019. IBM's Qiskit Tool Chain: Working with and Developing for Real Quantum Computers. In 2019 Design, Automation Test in Europe Conference Exhibition (DATE). 1234--1240.

[39]

Fan Wu, Mark Harman, Yue Jia, and Jens Krinke. 2016. Homi: Searching higher order mutants for software improvement. In International Symposium on Search Based Software Engineering. Springer, 18--33.

[40]

Tao Yue, Paolo Arcaini, and Shaukat Ali. 2022. Quantum Software Testing: Challenges, Early Achievements, and Opportunities. ERCIM News 2022, 128 (2022). https://ercim-news.ercim.eu/en128/special/quantum-software-testing-challenges-early-achievements-and-opportunities

[41]

Jianjun Zhao. 2020. Quantum Software Engineering: Landscapes and Horizons. CoRR abs/2007.07047 (2020). arXiv:2007.07047

Cited By

Gemeinhardt FKlikovits SWimmer M(2025)GeQuPI: Quantum Program Improvement with Multi-Objective Genetic ProgrammingJournal of Systems and Software10.1016/j.jss.2024.112223219(112223)Online publication date: Jan-2025
https://doi.org/10.1016/j.jss.2024.112223
Muqeet AAli SArcaini PFilkov VRay BZhou M(2024)Quantum Program Testing Through Commuting Pauli Strings on IBM's Quantum ComputersProceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering10.1145/3691620.3695275(2130-2141)Online publication date: 27-Oct-2024
https://dl.acm.org/doi/10.1145/3691620.3695275
Li YPei HHuang LYin BCai K(2024)Automatic Repair of Quantum Programs via Unitary OperationACM Transactions on Software Engineering and Methodology10.1145/366460433:6(1-43)Online publication date: 28-Jun-2024
https://dl.acm.org/doi/10.1145/3664604
Show More Cited By

Index Terms

Mutation-based test generation for quantum programs with multi-objective search
1. Software and its engineering
  1. Software creation and management
    1. Search-based software engineering
2. Theory of computation
  1. Models of computation
    1. Quantum computation theory

Recommendations

Generating failing test suites for quantum programs with search (hot off the press track at GECCO 2022)
GECCO '22: Proceedings of the Genetic and Evolutionary Computation Conference Companion

The inherent complexity of quantum programs, due to features such as superposition and entanglement, makes their testing particularly challenging. To tackle these challenges, we present a search-based approach, called Quantum Search-Based Testing (QuSBT),...
QuSBT: search-based testing of quantum programs
ICSE '22: Proceedings of the ACM/IEEE 44th International Conference on Software Engineering: Companion Proceedings

Generating a test suite for a quantum program such that it has the maximum number of failing tests is an optimization problem. For such optimization, search-based testing has shown promising results in the context of classical programs. To this end, we ...
Achieving scalable mutation-based generation of whole test suites

Without complete formal specification, automatically generated software tests need to be manually checked in order to detect faults. This makes it desirable to produce the strongest possible test set while keeping the number of tests as small as ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

GECCO '22: Proceedings of the Genetic and Evolutionary Computation Conference

July 2022

1472 pages

ISBN:9781450392372

DOI:10.1145/3512290

Editor:
Jonathan E. Fieldsend
University of Exeter
,
General Chair:
Markus Wagner
The University of Adelaide

Copyright © 2022 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGEVO: ACM Special Interest Group on Genetic and Evolutionary Computation

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 08 July 2022

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Data Availability

Supplemental material. https://dl.acm.org/doi/10.1145/3512290.3528869#p1345-wang-suppl.zip

Funding Sources

ERATO HASUO Metamathematics for Systems Design Project
The Netional Natural Science Foundation of China
Research Corporation for Science Advancement Council of Norway

Conference

GECCO '22

Sponsor:

SIGEVO

GECCO '22: Genetic and Evolutionary Computation Conference

July 9 - 13, 2022

Massachusetts, Boston

Acceptance Rates

Overall Acceptance Rate 1,669 of 4,410 submissions, 38%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

15
Total Citations
View Citations
200
Total Downloads

Downloads (Last 12 months)35
Downloads (Last 6 weeks)3

Reflects downloads up to 25 Feb 2025

Other Metrics

View Author Metrics

Citations

Cited By

Gemeinhardt FKlikovits SWimmer M(2025)GeQuPI: Quantum Program Improvement with Multi-Objective Genetic ProgrammingJournal of Systems and Software10.1016/j.jss.2024.112223219(112223)Online publication date: Jan-2025
https://doi.org/10.1016/j.jss.2024.112223
Muqeet AAli SArcaini PFilkov VRay BZhou M(2024)Quantum Program Testing Through Commuting Pauli Strings on IBM's Quantum ComputersProceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering10.1145/3691620.3695275(2130-2141)Online publication date: 27-Oct-2024
https://dl.acm.org/doi/10.1145/3691620.3695275
Li YPei HHuang LYin BCai K(2024)Automatic Repair of Quantum Programs via Unitary OperationACM Transactions on Software Engineering and Methodology10.1145/366460433:6(1-43)Online publication date: 28-Jun-2024
https://dl.acm.org/doi/10.1145/3664604
Long PZhao J(2024)Testing Multi-Subroutine Quantum Programs: From Unit Testing to Integration TestingACM Transactions on Software Engineering and Methodology10.1145/365633933:6(1-61)Online publication date: 27-Jun-2024
https://dl.acm.org/doi/10.1145/3656339
Pascoal Faria JVerbeek FRita Fasolino A(2024)Report from the 14th International Workshop on Automating Test Case Design, Selection, and Evaluation (A-TEST 2023)ACM SIGSOFT Software Engineering Notes10.1145/3650142.365014949:2(22-24)Online publication date: 3-Apr-2024
https://dl.acm.org/doi/10.1145/3650142.3650149
Muqeet AYue TAli SArcaini P(2024)Mitigating Noise in Quantum Software Testing Using Machine LearningIEEE Transactions on Software Engineering10.1109/TSE.2024.346297450:11(2947-2961)Online publication date: Nov-2024
https://doi.org/10.1109/TSE.2024.3462974
Long PZhao J(2024)Equivalence, identity, and unitarity checking in black-box testing of quantum programsJournal of Systems and Software10.1016/j.jss.2024.112000211:COnline publication date: 2-Jul-2024
https://dl.acm.org/doi/10.1016/j.jss.2024.112000
Khan AAkbar MLahtinen VPaavola MNiazi MAlatawi MAlotaibi S(2024)Agile meets quantum: a novel genetic algorithm model for predicting the success of quantum software development projectAutomated Software Engineering10.1007/s10515-024-00434-z31:1Online publication date: 4-Apr-2024
https://doi.org/10.1007/s10515-024-00434-z
Fortunato DJiménez-Navajas LCampos JAbreu R(2024)Verification and Validation of Quantum SoftwareQuantum Software10.1007/978-3-031-64136-7_5(93-123)Online publication date: 23-Aug-2024
https://doi.org/10.1007/978-3-031-64136-7_5
Khan AAkbar MAhmad AFahmideh MShameem MLahtinen VWaseem MMikkonen T(2023)Agile Practices for Quantum Software Development: Practitioners’ Perspectives2023 IEEE International Conference on Quantum Software (QSW)10.1109/QSW59989.2023.00012(9-20)Online publication date: Jul-2023
https://doi.org/10.1109/QSW59989.2023.00012
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten