Abstract
Testing quantum programs requires systematic, automated, and intelligent methods due to their inherent complexity, such as their superposition and entanglement. To this end, we present a search-based approach, called Quantum Search-Based Testing (QuSBT), for automatically generating test suites of a given size depending on available testing budget, with the aim of maximizing the number of failing test cases in the test suite. QuSBT consists of definitions of the problem encoding, failure types, test assessment with statistical tests, fitness function, and test case generation with a Genetic Algorithm (GA). To empirically evaluate QuSBT, we compared it with Random Search (RS) by testing six quantum programs. We assessed the effectiveness of QuSBT and RS with 30 carefully designed faulty versions of the six quantum programs. Results show that QuSBT provides a viable solution for testing quantum programs, and achieved a significant improvement over RS in 87% of the faulty programs, and no significant difference in the rest of 13% of the faulty programs.
QuSBT is supported by the National Natural Science Foundation of China under Grant No. 61872182 and Qu-Test (Project#299827) funded by Research Council of Norway. Paolo Arcaini is supported by ERATO HASUO Metamathematics for Systems Design Project (No. JPMJER1603), JST. Funding Reference number: 10.13039/501100009024 ERATO.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Notes
- 1.
Note that \(I\) and \(O\) do not need to be disjoint, i.e., an input qubit can also be an output qubit. Moreover, there could also be qubits that are neither inputs nor outputs, i.e., \(I \cup O \subseteq Q \).
- 2.
In this case, we directly set \(\mathtt {fail} _{\textit{wodf}}\) (see Eq. 1 later) to false.
- 3.
Note that the assessment for wodf is done only if the assessment for uof did not reveal any failure. If this is the case, it is guaranteed that the program returned outputs only from \(j_1, \ldots , j_k\), i.e., those having their expected probabilities being non-zero. Therefore, it is guaranteed that each returned output is considered in one of the counts \(c_{j_1}, \ldots , c_{j_k}\).
- 4.
Note that a quantum program can still be deterministic for some given inputs.
- 5.
- 6.
Note that we manually approximated the value of programs with 1024 inputs values. Indeed, the correct number of tests would be \(\left\lceil {0.05 \cdot 1024}\right\rceil = 52\).
References
Agresti, A.: An Introduction to Categorical Data Analysis, 3 edn. Wiley-Blackwell (2019)
Ali, S., Arcaini, P., Wang, X., Yue, T.: Assessing the effectiveness of input and output coverage criteria for testing quantum programs. In: 2021 14th IEEE Conference on Software Testing, Verification and Validation (ICST), pp. 13–23 (2021). https://doi.org/10.1109/ICST49551.2021.00014
Arcuri, A., Briand, L.: A practical guide for using statistical tests to assess randomized algorithms in software engineering. In: Proceedings of the 33rd International Conference on Software Engineering, ICSE 2011, pp. 1–10. ACM, New York (2011)
Arcuri, A., Fraser, G.: Parameter tuning or default values? An empirical investigation in search-based software engineering. Empir. Softw. Eng. 18, 594–623 (2013)
BenÃtez-Hidalgo, A., Nebro, A.J., GarcÃa-Nieto, J., Oregi, I., Del Ser, J.: jMetalPy: a Python framework for multi-objective optimization with metaheuristics. Swarm Evol. Comput. 51, 100598 (2019)
Gimeno-Segovia, M., Harrigan, N., Johnston, E.: Programming Quantum Computers: Essential Algorithms and Code Samples. O’Reilly Media, Inc., Newton (2019)
Honarvar, S., Mousavi, M.R., Nagarajan, R.: Property-based testing of quantum programs in Q#. In: Proceedings of the IEEE/ACM 42nd International Conference on Software Engineering Workshops, ICSEW 2020, pp. 430–435. Association for Computing Machinery, New York (2020)
Huang, Y., Martonosi, M.: QDB: from quantum algorithms towards correct quantum programs. In: 9th Workshop on Evaluation and Usability of Programming Languages and Tools (PLATEAU 2018). OpenAccess Series in Informatics (OASIcs), vol. 67, pp. 4:1–4:14. Schloss Dagstuhl-Leibniz-Zentrum fuer Informatik, Dagstuhl (2019)
Li, G., Zhou, L., Yu, N., Ding, Y., Ying, M., Xie, Y.: Projection-based runtime assertions for testing and debugging quantum programs. Proc. ACM Program. Lang. 4(OOPSLA), 1–29 (2020)
Liu, J., Byrd, G.T., Zhou, H.: Quantum circuits for dynamic runtime assertions in quantum computation. In: Proceedings of the Twenty-Fifth International Conference on Architectural Support for Programming Languages and Operating Systems, pp. 1017–1030 (2020)
Miranskyy, A., Zhang, L.: On testing quantum programs. In: 2019 IEEE/ACM 41st International Conference on Software Engineering: New Ideas and Emerging Results (ICSE-NIER), pp. 57–60 (2019)
Wang, J., Ma, F., Jiang, Y.: Poster: fuzz testing of quantum program. In: 2021 14th IEEE Conference on Software Testing, Verification and Validation (ICST), pp. 466–469 (2021). https://doi.org/10.1109/ICST49551.2021.00061
Wille, R., Van Meter, R., Naveh, Y.: IBM’s Qiskit tool chain: working with and developing for real quantum computers. In: 2019 Design, Automation Test in Europe Conference Exhibition (DATE), pp. 1234–1240 (2019)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2021 Springer Nature Switzerland AG
About this paper
Cite this paper
Wang, X., Arcaini, P., Yue, T., Ali, S. (2021). Generating Failing Test Suites for Quantum Programs With Search. In: O'Reilly, UM., Devroey, X. (eds) Search-Based Software Engineering. SSBSE 2021. Lecture Notes in Computer Science(), vol 12914. Springer, Cham. https://doi.org/10.1007/978-3-030-88106-1_2
Download citation
DOI: https://doi.org/10.1007/978-3-030-88106-1_2
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-88105-4
Online ISBN: 978-3-030-88106-1
eBook Packages: Computer ScienceComputer Science (R0)