Abstract
As society increasingly relies on safety- and security- critical systems, the need for confirming their dependability becomes essential. Adequate V&V (verification and validation) methods must be employed, e.g., for system testing. When selecting and using the methods, it is important to analyze their possible gaps and limitations, such as scalability issues. However, and as we have experienced, common, explicitly defined criteria are seldom used for such analyses. This results in analyses that consider different aspects and to a different extent, hindering their comparison and thus the comparison of the V&V methods. As a solution, we present a set of criteria for the analysis of gaps and limitations of V&V methods for safety- and security-critical systems. The criteria have been identified in the scope of the VALU3S project. Sixty-two people from 33 organizations agreed upon the use of nine criteria: functionality, accuracy, scalability, deployment, learning curve, automation, reference environment, cost, and standards. Their use led to more homogeneous and more detailed analyses when compared to similar previous efforts. We argue that the proposed criteria can be helpful to others when having to deal with similar activities.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Agirre, J., et al.: The VALU3S ECSEL project: verification and validation of automated systems safety and security. Microprocess. Microsyst. 87, 104349 (2021)
Aguirre, J., et al.: Multidimensional framework for characterizing verification and validation of automated systems. In: EDCC (2022)
Amalthea4public project: D3.1 - Analysis of state of the art V&V techniques (2015)
AMASS project: https://cordis.europa.eu/project/id/692474
AMASS project: D1.7 - AMASS solution benchmarking (2019)
AMASS project: D3.1 - Baseline and requirements for architecture-driven assurance (2018)
Bauer, T., et al.: Cross-domain modelling of verification and validation workflows in the large scale European research project VALU3S. In: Orailoglu, A., Jung, M., Reichenbach, M. (eds) Embedded Computer Systems: Architectures, Modeling, and Simulation. SAMOS 2021. LNCS, vol. 13227. Springer, Cham (2022). https://doi.org/10.1007/978-3-031-04580-6_25
de la Vara, J.L., et al.: A proposal for the classification of methods for verification and validation of safety, cybersecurity, and privacy of automated systems. In: QUATIC (2021)
ElasTest project: D7.3 - Public demonstrator artifacts (2019)
Fenton, N.E., Pfleeger, S.L.: Software Metrics - A Rigorous and Practical Approach, 3rd edn. CRC Press, Boca Raton (2015)
Folkesson, P., Ayatolahi, F., Sangchoolie, B., Vinter, J., Islam, M., Karlsson, J.: Back-to-back fault injection testing in model-based development. In: Koornneef, F., van Gulijk, C. (eds.) SAFECOMP 2015. LNCS, vol. 9337, pp. 135–148. Springer, Cham (2015). https://doi.org/10.1007/978-3-319-24255-2_11
Grinschgl, J., et al.: Efficient fault emulation using automatic pre-injection memory access analysis. In: 2012 IEEE International SOC Conference, pp. 277–282. Niagara Falls, NY (2012)
INCOSE: Guide for Writing Requirements (2019)
iRel40 project: https://cordis.europa.eu/project/id/876659
ISO: ISO/IEC/IEEE 24765: Systems and software engineering - Vocabulary (2017)
Kan, S.: Metrics and Models in Software Quality Engineering, 2nd edn. Addison Wesley, Boston (2002)
Kramer, A., Legeard, B.: Model-Based Testing Essentials-Guide to the ISTQB Certified Model-Based Tester: Foundation Level. Wiley, Hoboken (2016)
Ma, T., Ali, S., Yue, T.: Testing self-healing cyber-physical systems under uncertainty with reinforcement learning: an empirical study. Empir. Softw. Eng. 26(3), 1–54 (2021). https://doi.org/10.1007/s10664-021-09941-z
Miller, L., et al.: Guidelines for the verification and validation of expert system software and conventional software. US Nuclear Regulatory Commission (1995)
Moreno, V., Génova, G., Parra, E., Fraga, A.: Application of machine learning techniques to the flexible assessment and improvement of requirements quality. Softw. Qual. J. 28(4), 1645–1674 (2020). https://doi.org/10.1007/s11219-020-09511-4
Natella, R., et al.: Assessing dependability with software fault injection: a survey. ACM Comput. Surv. 48(3), 44 (2016)
OPENCOSS project: https://cordis.europa.eu/project/id/289011
OPENCOSS project: D1.3 - Evaluation framework and quality metrics (2013)
Oxford UK Dictionary: Method. https://www.lexico.com/definition/method (2021)
Panesar-Walawege, R.K., et al.: Supporting the verification of compliance to safety standards via model-driven engineering: approach, tool-support and empirical validation. Inf. Softw. Technol. 55(3), 836–864 (2013)
Parra, E., et al.: Advances in artefact quality analysis for safety-critical systems. In: 30th International Symposium on Software Reliability Engineering (ISSRE) (2019)
Roza, M.: Verification, validation and uncertainty quantification methods and techniques. NATO (2014)
Svenningsson, R., Vinter, J., Eriksson, H., Törngren, M.: MODIFI: a MODel-implemented fault injection tool. In: Schoitsch, E. (ed.) SAFECOMP 2010. LNCS, vol. 6351, pp. 210–222. Springer, Heidelberg (2010). https://doi.org/10.1007/978-3-642-15651-9_16
VALU3S project: D3.1 - V&V methods for SCP evaluation of automated systems (2020)
VALU3S project: D3.3 - Identified gaps and limitations of the V&V methods listed in D3.1 (2021)
VALU3S project: D3.6 - Final description of methods designed to improve the V&V process (2022)
Acknowledgments
The research leading to this paper has received funding from the VALU3S (H2020-ECSEL grant agreement no 876852; MCIN/AEI ref. PCI2020-112001; NextGen.EU/ PRTR), iRel4.0 (H2020-ECSEL grant agreement no 876659; MCIN/AEI ref. PCI2020-112240; NextGen.EU/PRTR), ETHEREAL (MICINN/AEI ref. PID2020-115220RB-C21; ERDF), and Treasure (JCCM SBPLY/19/180501/ 000270; ERDF) projects, and from the Ramon y Cajal Program (MICINN RYC-2017-22836; ESF). We are also grateful to all the VALU3S partners that have provided input and feedback for the selection of the criteria and that applied them.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Ferrari, E., Schlick, R., de la Vara, J.L., Folkesson, P., Sangchoolie, B. (2022). Criteria for the Analysis of Gaps and Limitations of V&V Methods for Safety- and Security-Critical Systems. In: Trapp, M., Schoitsch, E., Guiochet, J., Bitsch, F. (eds) Computer Safety, Reliability, and Security. SAFECOMP 2022 Workshops . SAFECOMP 2022. Lecture Notes in Computer Science, vol 13415. Springer, Cham. https://doi.org/10.1007/978-3-031-14862-0_9
Download citation
DOI: https://doi.org/10.1007/978-3-031-14862-0_9
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-14861-3
Online ISBN: 978-3-031-14862-0
eBook Packages: Computer ScienceComputer Science (R0)