Abstract
Violating the safety of autonomous driving systems (ADSs) could lead to fatal accidents. ADSs are complex, constantly-evolving and software-intensive systems. Testing an individual ADS is challenging and expensive on its own, and consequently testing its multiple versions (due to evolution) becomes much more costly. Thus, it is needed to develop approaches for selecting and prioritizing tests for newer versions of ADSs based on historical test execution data of their previous versions. To this end, we propose a multi-objective search-based approach for Selection and Prioritization of tEst sCenarios for auTonomous dRiving systEms (SPECTRE) to test newer versions of an ADS based on four optimization objectives, e.g., demand of a test scenario put on an ADS. We experimented with five commonly used multi-objective evolutionary algorithms and used a repository of 60,000 test scenarios. Among all the algorithms, IBEA achieved the best performance for solving all the optimization problems of varying complexity.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Ali, S., Arcaini, P., Pradhan, D., Safdar, S.A., Yue, T.: Quality indicators in search-based software engineering: an empirical evaluation. ACM Trans. Softw. Eng. Methodol. (TOSEM) 29(2), 1–29 (2020)
Arcuri, A., Briand, L.: A practical guide for using statistical tests to assess randomized algorithms in software engineering. In: Proceedings of the 33rd International Conference on Software Engineering (ICSE 2011), pp. 1–10 (2011)
Arcuri, A., Fraser, G.: On parameter tuning in search based software engineering. In: Cohen, M.B., Ó Cinnéide, M. (eds.) SSBSE 2011. LNCS, vol. 6956, pp. 33–47. Springer, Heidelberg (2011). https://doi.org/10.1007/978-3-642-23716-4_6
Ben Abdessalem, R., Nejati, S., Briand, L.C., Stifter, T.: Testing advanced driver assistance systems using multi-objective search and neural networks. In: Proceedings of the Conference on Automated Software Engineering, pp. 63–74. ACM (2016)
Ben Abdessalem, R., Nejati, S., Briand, L.C., Stifter, T.: Testing vision-based control systems using learnable evolutionary algorithms. In: Proceedings of the Conference on Software Engineering, pp. 1016–1026. ACM (2018)
Ben Abdessalem, R., Panichella, A., Nejati, S., Briand, L.C., Stifter, T.: Testing autonomous cars for feature interaction failures using many-objective search. In: Proceedings of the Conference on Automated Software Engineering, pp. 143–154. ACM (2018)
Corso, A., Du, P., Driggs-Campbell, K., Kochenderfer, M.J.: Adaptive stress testing with reward augmentation for autonomous vehicle validatio. In: 2019 IEEE Intelligent Transportation Systems Conference (ITSC), pp. 163–168. IEEE (2019)
Czarnecki, K.: Operational design domain for automated driving systems: Taxonomy of basic terms. Waterloo Intelligent Systems Engineering (WISE) Lab, University of Waterloo, Canada (2018)
Durillo, J.J., Nebro, A.J.: jMetal: a Java framework for multi-objective optimization. Adv. Eng. Softw. 42(10), 760–771 (2011)
Gambi, A., Mueller, M., Fraser, G.: Automatically testing self-driving cars with search-based procedural content generation. In: Proceedings of International Symposium on Software Testing and Analysis, pp. 318–328. ACM (2019)
Greer, D., Ruhe, G.: Software release planning: an evolutionary and iterative approach. Inf. Softw. Technol. 46(4), 243–253 (2004)
Li, G., et al.: AV-FUZZER: finding safety violations in autonomous driving systems. In: International Symposium on Software Reliability Engineering, pp. 25–36. IEEE (2020)
Li, Z., Harman, M., Hierons, R.M.: Search algorithms for regression test case prioritization. IEEE Trans. Softw. Eng. 33(4), 225–237 (2007)
Luo, Q., Moran, K., Poshyvanyk, D., Di Penta, M.: Assessing test case prioritization on real faults and mutants. In: 2018 IEEE International Conference on Software Maintenance and Evolution (ICSME), pp. 240–251. IEEE (2018)
Pradhan, D., Wang, S., Ali, S., Yue, T., Liaaen, M.: STIPI: using search to prioritize test cases based on multi-objectives derived from industrial practice. In: Wotawa, F., Nica, M., Kushik, N. (eds.) ICTSS 2016. LNCS, vol. 9976, pp. 172–190. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-47443-4_11
Ramirez, A., Romero, J.R., Ventura, S.: A survey of many-objective optimisation in search based software engineering. Syst. Softw. Eng. 149, 382–395 (2019)
Singh, Y., Kaur, A., Suri, B.: Test case prioritization using ant colony optimization. ACM SIGSOFT Softw. Eng. Notes 35(4), 1–7 (2010)
Wang, S., Ali, S., Yue, T., Bakkeli, Ø., Liaaen, M.: Enhancing test case prioritization in an industrial setting with resource awareness and multi-objective search. In: Proceedings of the 38th International Conference on Software Engineering Companion, pp. 182–191 (2016)
Yoo, S., Harman, M.: Pareto efficient multi-objective test case selection. In: Proceedings of the International Symposium on Software Testing and Analysis, pp. 140–150 (2007)
Zhang, H., Zhang, M., Yue, T., Ali, S., Li, Y.: Uncertainty-wise requirements prioritization with search. ACM Trans. Softw. Eng. Methodol. (TOSEM) 30(1), 1–54 (2020)
Zhang, M., Ali, S., Yue, T.: Uncertainty-wise test case generation and minimization for cyber-physical systems. J. Syst. Softw. 153, 1–21 (2019)
Zitzler, E., Künzli, S.: Indicator-based selection in multiobjective search. In: Yao, X., et al. (eds.) PPSN 2004. LNCS, vol. 3242, pp. 832–842. Springer, Heidelberg (2004). https://doi.org/10.1007/978-3-540-30217-9_84
Acknowledgements
The work is supported by the National Natural Science Foundation of China under Grant No. 61872182. The work is also partially supported by the Co-evolver project (No. 286898/F20) funded by the Research Council of Norway. Huihui Zhang is supported by the Science and Technology Program of Public Wellbeing (No. 2020KJHM01).
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2021 Springer Nature Switzerland AG
About this paper
Cite this paper
Lu, C., Zhang, H., Yue, T., Ali, S. (2021). Search-Based Selection and Prioritization of Test Scenarios for Autonomous Driving Systems. In: O'Reilly, UM., Devroey, X. (eds) Search-Based Software Engineering. SSBSE 2021. Lecture Notes in Computer Science(), vol 12914. Springer, Cham. https://doi.org/10.1007/978-3-030-88106-1_4
Download citation
DOI: https://doi.org/10.1007/978-3-030-88106-1_4
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-88105-4
Online ISBN: 978-3-030-88106-1
eBook Packages: Computer ScienceComputer Science (R0)