B model quality assessments on automated reachability repair with ISO/IEC 25010
Introduction
In software engineering, formal verification methods have been used to rigorously verify the correctness of software design. For example, in the B-method [1], a software architecture can be abstractly represented as a design model, which describes how key components interact with each other, but omits implementation details of the key components. Using model checkers such as ProB, [2], the design model can be formally verified against a set of given properties to ensure the correctness of interaction processes for the key components, so that inconsistencies in the software architecture can be detected before actually implementing a software product. However, the model checker only indicates single flaws, but may not reflect the overall quality of the design model. If the overall quality can be measured, then designers can decide whether and how to further improve the models before actually implementing software products.
As a post-processing step of formal verification, automated model repair has attracted people's attention in recent years [3], [4], [5], [6]. Model repair algorithms are expected to detect faulty code in design models and work out feasible repairs to update the faulty code. Model repair is similar to automated software repair [7], [8], [9] that concerns how to automatically eliminate faults in source code. To repair models and software, automatic programming techniques [10], [11], [12] can be used to synthesise code of repair. In the B-method, a technique on semi-automated (interactive) model repair has been proposed in 2016 [13] and improved in 2018 [14]. This technique uses an inductive programming algorithm to synthesise repairs, e.g., state transition insertion, invariant weakening and precondition strengthening, from examples provided by users. However, to the best of our knowledge, there is no standard that can be used to systematically certify the repaired B models. This problem motivates us to study how to borrow standards of software product to assess the quality of model repair. As standards need to be proposed by experts and agreed by the industry, our study just aims to find feasible and reasonable criteria to fast measure B model quality in automatic model repair, but does not form a standard that covers all aspects of the B method.
This article is an extension of our conference paper titled Measuring the Quality of B Abstract Machines with ISO/IEC 25010, originally presented in TASE 2020 [15]. The conference paper aims to provide a set of quality measurement formulae based on the B-method [1] and the ISO/IEC 25010 standard [16]. The contributions of the original conference paper include:
- •
criteria for measuring functional suitability of formal design models with respect to given functional requirements
- •
criteria for measuring security and reliability of formal design models with respect to given invariants
- •
criteria for measuring maintainability, performance efficiency and usability of formal design models using model checking techniques
- •
the application of the above criteria to the B-method
- •
an algorithm, called Goal Predicate Reachability Repair (GPRR), to automatically repair B models that fail to achieve goals described by predicates
- •
an integration of automatic model repair and model quality measurement
- •
an empirical study on how automatic model repair improves model quality characteristics such as functional suitability and goal appropriateness
The rest of this paper is organised as follows. Section 2 introduces preliminary knowledge such as ISO/IEC 25010 and the B-method. Section 3 introduces the criteria of functional suitability. Section 4 introduces the criteria of security and reliability. Section 5 introduces the criteria of maintainability, performance efficiency and usability. Section 6 introduces an automated model repair algorithm, and shows how to use the ISO/IEC 25010 criteria to evaluate the quality of model repair. Section 7 compares the proposed criteria with other existing criteria. Section 8 concludes our work.
Section snippets
Preliminaries
In this section, we introduce preliminary knowledge of our work, including ISO/IEC 25010, which is one of the international standards of software quality, and model checking techniques for computing state transition systems.
Criteria of functional suitability
To evaluate the quality of B models, we derive a number of criteria from ISO/IEC 25010. Among the eight characteristics of ISO/IEC 25010, six characteristics including functional suitability, reliability, performance efficiency, usability, security and maintainability can be reflected by B models. The remaining two characteristics, i.e., compatibility and portability, cannot be reflected by B models because the two characteristics are measured after software is actually implemented, installed
Criteria of security and reliability
In B models, properties of security and reliability are usually specified using invariants. As B assertions are a special type of invariants, in the following discussions, “invariants” include assertions, and “invariant violations” include assertion violations. In our work, deadlock-freeness, which requires that each state has at least one outgoing transition, is considered as an inherent invariant and is always checked even if it is not specified.
Sub-characteristics of security include
Criteria of maintainability, performance efficiency and usability
Maintainability measures whether it is easy to modify the given model M. Analysability and modifiability, which are sub-characteristics of maintainability, are computed based on because they are observed after changing M. The analysability measures whether changes to M can significantly impact its functionality and the satisfiability to the invariants. Intuitively, a maintainer who is not developers of a given model may consider the model as a black-box. If a slight change to the model
Measuring the quality of automated B model repair
Model checking techniques enable developers to formally verify whether design models of software products satisfy desired properties in their state transition systems [18]. However, if faults are detected in a design model, software engineers still need to manually work out plans of repair, update the design model and use model checkers to verify the updated model. If the design model has multiple faults, such verification-repair processes may be repeated several times. Consequently, the
Related work
In the history of software engineering, the ISO/IEC standards have formed a number of software certification systems to evaluate software products on different aspects, e.g., source code [27], functional size [28], maintenance [29], assurance [30], risk management [31], project management [32], application management [33], resource utilisation [34], software quality [16], informational websites [35], information security [36], etc. Software certification is vitally important in safety-critical
Conclusion
Based on ISO/IEC 25010, we have proposed a set of criteria for evaluating design model quality. These criteria can be used together with model checking techniques to quantify the quality of state transition systems of design models. Moreover, we have proposed the GPRR algorithm for solving unreachable goal predicates. Based on GPRR, we have demonstrated that the ISO/IEC 25010 criteria can be used to measure model quality characteristics of automated B model repair. The advantages of the
CRediT authorship contribution statement
Cheng-Hao Cai: Conceptualization, Formal analysis, Investigation, Methodology, Software, Validation, Writing – original draft. Jing Sun: Conceptualization, Funding acquisition, Investigation, Methodology, Project administration, Resources, Supervision, Writing – review & editing. Gillian Dobbie: Conceptualization, Methodology, Supervision, Writing – review & editing.
Declaration of Competing Interest
The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.
References (60)
- et al.
Symbolic model checking: 1020 states and beyond
Inf. Comput.
(1992) - et al.
Teaching ISO/IEC 12207 software lifecycle processes: a serious game approach
Comput. Stand. Interfaces
(2017) - et al.
Implementing information security best practices on software lifecycle processes: the ISO/IEC 15504 security extension
Comput. Secur.
(2015) - et al.
Definitions and approaches to model quality in model-based software development - a review of literature
Inf. Softw. Technol.
(2009) The B-Book - Assigning Programs to Meanings
(1996)- et al.
ProB: an automated analysis toolset for the B method
Int. J. Softw. Tools Technol. Transf.
(2008) - et al.
CTL model update for system modifications
J. Artif. Intell. Res.
(2008) - et al.
Revision of CTL models
- et al.
Abstract model repair
- et al.
Non-intrusive repair of safety and liveness violations in reactive programs
Trans. Comput. Collect. Intell.
(2014)