B model quality assessments on automated reachability repair with ISO/IEC 25010

https://doi.org/10.1016/j.scico.2021.102732Get rights and content

Highlights

  • ISO/IEC 25010 can be used to evaluate the quality of state spaces.

  • Twenty criteria for measuring the quality of abstract machines are proposed.

  • The reachability of abstract machines can be automatically repaired.

  • The quality of reachability repair can be evaluated using ISO/IEC 25010.

Abstract

In software engineering, formal methods are often used to specify and verify design models of software products. Whether design models are consistent with required properties can significantly impact the quality of final software products. In this work, we study B model quality measurements based on the ISO/IEC 25010 standard. These measurements are formulated as domain-independent formulae and computed by model checking. Moreover, we study how to enable machines to automatically solve unreachable goals in B models. We suggest to use constraint solvers and semantic learners to discover state transitions to the goals. To demonstrate the effectiveness of the model repair technique, a set of experiments are conducted based on the model quality measurements. The results demonstrate that the model repair technique can solve unreachable goals while preserving the quality of models.

Introduction

In software engineering, formal verification methods have been used to rigorously verify the correctness of software design. For example, in the B-method [1], a software architecture can be abstractly represented as a design model, which describes how key components interact with each other, but omits implementation details of the key components. Using model checkers such as ProB, [2], the design model can be formally verified against a set of given properties to ensure the correctness of interaction processes for the key components, so that inconsistencies in the software architecture can be detected before actually implementing a software product. However, the model checker only indicates single flaws, but may not reflect the overall quality of the design model. If the overall quality can be measured, then designers can decide whether and how to further improve the models before actually implementing software products.

As a post-processing step of formal verification, automated model repair has attracted people's attention in recent years [3], [4], [5], [6]. Model repair algorithms are expected to detect faulty code in design models and work out feasible repairs to update the faulty code. Model repair is similar to automated software repair [7], [8], [9] that concerns how to automatically eliminate faults in source code. To repair models and software, automatic programming techniques [10], [11], [12] can be used to synthesise code of repair. In the B-method, a technique on semi-automated (interactive) model repair has been proposed in 2016 [13] and improved in 2018 [14]. This technique uses an inductive programming algorithm to synthesise repairs, e.g., state transition insertion, invariant weakening and precondition strengthening, from examples provided by users. However, to the best of our knowledge, there is no standard that can be used to systematically certify the repaired B models. This problem motivates us to study how to borrow standards of software product to assess the quality of model repair. As standards need to be proposed by experts and agreed by the industry, our study just aims to find feasible and reasonable criteria to fast measure B model quality in automatic model repair, but does not form a standard that covers all aspects of the B method.

This article is an extension of our conference paper titled Measuring the Quality of B Abstract Machines with ISO/IEC 25010, originally presented in TASE 2020 [15]. The conference paper aims to provide a set of quality measurement formulae based on the B-method [1] and the ISO/IEC 25010 standard [16]. The contributions of the original conference paper include:

  • criteria for measuring functional suitability of formal design models with respect to given functional requirements

  • criteria for measuring security and reliability of formal design models with respect to given invariants

  • criteria for measuring maintainability, performance efficiency and usability of formal design models using model checking techniques

  • the application of the above criteria to the B-method

Based on the conference paper, this article has the following new contributions:
  • an algorithm, called Goal Predicate Reachability Repair (GPRR), to automatically repair B models that fail to achieve goals described by predicates

  • an integration of automatic model repair and model quality measurement

  • an empirical study on how automatic model repair improves model quality characteristics such as functional suitability and goal appropriateness

The rest of this paper is organised as follows. Section 2 introduces preliminary knowledge such as ISO/IEC 25010 and the B-method. Section 3 introduces the criteria of functional suitability. Section 4 introduces the criteria of security and reliability. Section 5 introduces the criteria of maintainability, performance efficiency and usability. Section 6 introduces an automated model repair algorithm, and shows how to use the ISO/IEC 25010 criteria to evaluate the quality of model repair. Section 7 compares the proposed criteria with other existing criteria. Section 8 concludes our work.

Section snippets

Preliminaries

In this section, we introduce preliminary knowledge of our work, including ISO/IEC 25010, which is one of the international standards of software quality, and model checking techniques for computing state transition systems.

Criteria of functional suitability

To evaluate the quality of B models, we derive a number of criteria from ISO/IEC 25010. Among the eight characteristics of ISO/IEC 25010, six characteristics including functional suitability, reliability, performance efficiency, usability, security and maintainability can be reflected by B models. The remaining two characteristics, i.e., compatibility and portability, cannot be reflected by B models because the two characteristics are measured after software is actually implemented, installed

Criteria of security and reliability

In B models, properties of security and reliability are usually specified using invariants. As B assertions are a special type of invariants, in the following discussions, “invariants” include assertions, and “invariant violations” include assertion violations. In our work, deadlock-freeness, which requires that each state has at least one outgoing transition, is considered as an inherent invariant and is always checked even if it is not specified.

Sub-characteristics of security include

Criteria of maintainability, performance efficiency and usability

Maintainability measures whether it is easy to modify the given model M. Analysability and modifiability, which are sub-characteristics of maintainability, are computed based on Mchanged because they are observed after changing M. The analysability measures whether changes to M can significantly impact its functionality and the satisfiability to the invariants. Intuitively, a maintainer who is not developers of a given model may consider the model as a black-box. If a slight change to the model

Measuring the quality of automated B model repair

Model checking techniques enable developers to formally verify whether design models of software products satisfy desired properties in their state transition systems [18]. However, if faults are detected in a design model, software engineers still need to manually work out plans of repair, update the design model and use model checkers to verify the updated model. If the design model has multiple faults, such verification-repair processes may be repeated several times. Consequently, the

Related work

In the history of software engineering, the ISO/IEC standards have formed a number of software certification systems to evaluate software products on different aspects, e.g., source code [27], functional size [28], maintenance [29], assurance [30], risk management [31], project management [32], application management [33], resource utilisation [34], software quality [16], informational websites [35], information security [36], etc. Software certification is vitally important in safety-critical

Conclusion

Based on ISO/IEC 25010, we have proposed a set of criteria for evaluating design model quality. These criteria can be used together with model checking techniques to quantify the quality of state transition systems of design models. Moreover, we have proposed the GPRR algorithm for solving unreachable goal predicates. Based on GPRR, we have demonstrated that the ISO/IEC 25010 criteria can be used to measure model quality characteristics of automated B model repair. The advantages of the

CRediT authorship contribution statement

Cheng-Hao Cai: Conceptualization, Formal analysis, Investigation, Methodology, Software, Validation, Writing – original draft. Jing Sun: Conceptualization, Funding acquisition, Investigation, Methodology, Project administration, Resources, Supervision, Writing – review & editing. Gillian Dobbie: Conceptualization, Methodology, Supervision, Writing – review & editing.

Declaration of Competing Interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

References (60)

  • L. Gazzola et al.

    Automatic software repair: a survey

    IEEE Trans. Softw. Eng.

    (2019)
  • C. Le Goues et al.

    GenProg: a generic method for automatic software repair

    IEEE Trans. Softw. Eng.

    (2012)
  • D. Alrajeh et al.

    Automated support for diagnosis and repair

    Commun. ACM

    (2015)
  • Z. Manna et al.

    Fundamentals of deductive program synthesis

    IEEE Trans. Softw. Eng.

    (1992)
  • P. Flener et al.

    An introduction to inductive programming

    Artif. Intell. Rev.

    (2008)
  • E. Kitzelmann

    Inductive Programming: A Survey of Program Synthesis Techniques

    Approaches and Applications of Inductive Programming

    (4 September 2009)
  • J. Schmidt et al.

    Interactive model repair by synthesis

  • J. Schmidt et al.

    Repair and generation of formal models using synthesis

  • C. Cai et al.

    Measuring the quality of B abstract machines with ISO/IEC 25010

  • ISO/IEC 25010, Systems and software engineering - Systems and software Quality Requirements and Evaluation (SQuaRE) -...
  • A. Idri et al.

    On the use of software quality standard ISO/IEC 9126 in mobile environments

  • E.M. Clarke et al.

    Model Checking

    (2001)
  • C.-H. Cai et al.

    Automatic B-model repair using model checking and machine learning

    Autom. Softw. Eng.

    (2019)
  • D. Cotroneo et al.

    Fault injection for software certification

    IEEE Secur. Priv.

    (2013)
  • C.-H. Cai et al.

    Achieving abstract machine reachability with learning-based model fulfilment

  • T.K. Ho

    Random decision forests

  • C.M. Bishop

    Pattern recognition and machine learning

  • V. Attasena et al.

    Secret sharing for cloud data security: a survey

    VLDB J.

    (2017)
  • A.A. Ciré et al.

    Multivalued decision diagrams for sequencing problems

    Oper. Res.

    (2013)
  • ISO/IEC 5055, Information technology - Software measurement - Software quality measurement - Automated source code...
  • Cited by (0)

    View full text