A formal specification animation method for operation validation,☆☆

https://doi.org/10.1016/j.jss.2021.110948Get rights and content

Highlights

  • A novel automatic animation data generation technique is proposed.

  • A software tool to support the animation of operations, data items, and logical expressions is described.

  • An industrial application is conducted to evaluate the proposed animation technique.

Abstract

Formal specification can benefit software quality by precisely defining the behaviors of operations to prevent primary mistakes in the early phase of software projects, but a remaining challenge is how such a specification can be checked comprehensibly to show whether it satisfies the user’s perception of requirements. In this paper, we describe a new technique for animating operation specifications as a means to address this problem. The technique offers new ways to do (1) automatic animation data generation for both input and output of an operation based on pre- and post-conditions, (2) visualized demonstration of the relationships between input and the corresponding output, (3) comprehensible animation of data items, and (4) illustrative animation of logical expressions and the operators used in them. We discuss these issues and present a prototype tool that supports the automation of the proposed technique. We also report an industrial application as a trial experiment to validate the technique. Finally, we conclude the paper and point out future research directions.

Introduction

Formal specification is a technique for defining what to be done by a potential software system using a mathematically-based language or notation. If used properly to appropriate application domains, formal specification can produce benefits in terms of improving software reliability and reducing the cost of its development (Kurita et al., 2008a, Woodcock et al., 2009). However, formal specification also faces challenges that keep many practitioners away from utilizing its advantages (Knight et al., 1997, Parnas, 2010). Our experience in applying formal specification in industry (Liu et al., 1998a, Luo et al., 2016) has made us realize that formal operation specifications are the places where mistakes often occur.

In realistic software projects, formal specifications are mainly written and read by the developers who are well trained in formal methods, but they may occasionally be used as a vehicle for communication between the developer and some users who have a good background of mathematics or formal methods. However, as far as our experience in Japan and China can tell, most users of software systems developed in industry would not be expected to understand formal specifications (Liu et al., 2008). Therefore, the formal specification written by the developer will need to be validated against the user’s requirements in a comprehensible manner.

Specification animation was proposed as a means for formal specification validation in 1990’s (Hazel et al., 1997, Miller and Strooper, 2003b). The essential idea is trying to make formal specifications in pre- and post-conditions executable so that sample input data, which is called animation data, can be properly selected to demonstrate the behavior dynamically. But since the proposed approach requires automatic translation from formal specifications in a formal notation (e.g., Z) to code, it can only deal with a subset of the formal notation because not all of the specifications in pre- and post-conditions can be automatically refined into code for execution (Hayes and Jones, 1989). Almost in the same period, Chen and Liu proposed an alternative technique called specification testing that does not require translation of formal specifications to code but can automatically generate test data for input variables and expected output data from pre- and post-conditions (Chen and Liu, 1996, Liu, 1999a). Although this work has set up an important foundation for building a more useful technique for specification animation, no much progress in technical details was made until we undertake the study presented in this paper. The main difference between the specification testing proposed previously and the specification animation proposed in this work is that the latter supports visualized demonstration of both operation behaviors and data characteristics for validation while the former only focuses on the generation of input and output values for an operation to check the consistency of its specification.

As is well known, the animation of software specifications for validation is notoriously difficult, primarily because it requires that humans understand what the animation is demonstrating. Furthermore, animation also faces another technical difficulty due to the fact that logical formulas and expressions often contain nonconstructive and abstract elements. To deal with these challenges, the strategy in our proposed animation technique is to generate comprehensible and representative animation data based on the pre- and post-conditions of an operation to visually demonstrate the relationships between input and output values and the characteristics of the values for the user who is usually interested in the behavior of the system under construction. For the analyst who writes the specification, we choose to animate the process of evaluating designated logical expressions to help him or her confirm whether the expressions are specified as desired.

This paper discusses the specification animation technique in four aspects: (1) how animation data for both input and output of an operation can be automatically generated based on its pre- and post-conditions, (2) how a visualized demonstration of the operation behaviors in terms of the input–output relation can be conducted, (3) how input and output variables of the operation can be confirmed to be properly declared with appropriate types, and (4) how the logical expressions involved in the pre- and post-conditions of the operation can be ensured to be valid.

We have made the following three contributions in this paper:

  • A novel automatic animation data generation technique is proposed.

  • A tool-supported approach to animating operations, data items, and logical expressions is established.

  • An industrial application is conducted to evaluate the proposed animation technique in terms of detecting faults in specifications.

In general, a software system is composed of more than one operation, therefore mistakes may occur in the integration of operations. For this reason, both the integration of operations and each individual operation in formal specifications need to be validated. We have put forward an approach to handling the animation of operation integrations in formal specifications in a previous work (Li and Liu, 2015). On the other hand, in many cases of realistic software engineering projects, a specification is often written as a set of rather independent operations and each operation is specified separately. Integrating the operations into an architecture is often regarded as part of software design rather than specification. This style becomes more and more adopted in the Agile development paradigm where comprehensive documentation is not emphasized. Of course, different schools may have different opinions on this since the Agile paradigm has both advantages and disadvantages. Our position is on the side of the combination of formal specification with the Agile principles under necessary compromises from both sides because our work on the Agile formal engineering methods (Liu, 2018) has shown that such a combination helps enhance both software productivity and quality. In this paper, we only focus on the discussion of the animation of individual operations specified in pre- and post-conditions. As mentioned above, our experience suggests that faults should often be introduced into individual operation specifications due to the difficulty in achieving abstract but precise mathematical expressions with high complexity.

The remainder of the paper is organized as follows. Section 2 briefly describes the goal of specification animation. Section 3 discusses the issues in relation to animation data generation. Section 4 presents a series of algorithms for generating animation data from various logical expressions in formal specifications. Section 5 focuses on a supporting tool we have built for our animation approach. Section 6 presents an industrial application of our technique to confirm its feasibility and effectiveness. Section 7 reviews related work and compare our work with existing work. Finally, in Section 8, we conclude the paper and discuss the future research topics.

Section snippets

Goal of specification animation

Given an operation specification in pre- and post-conditions, the goal of the specification animation is to analyze and confirm the functionality of the operation defined in the specification, the characteristics of the related data items, and the meaning of the formal expressions involved in the formal specification through visualized demonstrations.

Animation data generation from specifications

We first discuss the structure of a process specification and then proceed to describe the criteria for animation data generation based on the structure.

Animation data generation algorithms

To validate a process using animation, we must carry out an animation for every acceptable functional scenario of the process in order to ensure that each potentially valid functional behavior is demonstrated at least once. To this end, animation data generation must focus on each individual functional scenario PiQj where Pi only contains input variables whilst Qj contains output variables (possibly input variables as well). Apparently, as long as we generate an animation data satisfying this

Animation tool

We have developed a prototype tool to support the animation of a process specification. In addition to supporting animation data generation based on the criteria and algorithms discussed above, the tool mainly offers three animation functions: (1) input–output relation animation, (2) data animation, and (3) logical expression animation. The tool is developed using C# in the Visual Studio 2012 environment.

Application in practice

As is well-known, conducting a credible experiment for evaluation of software engineering techniques is notoriously difficult due to the fact that many uncertain factors may prevent us from setting up a context in which our method is comparable with any other method. For this reason, we do not believe that using controlled experiments is definitely a credible way to evaluate software engineering techniques within a limited resources (e.g., time, budget, subjects), especially when human

Related work

Existing work on specification animation either supports the process of model checking or executing formal specifications by means of translating them into executable code. To the best of our knowledge, no related work supports the same technique proposed in this paper.

One of the early studies on requirements specification animation was done by Kramer and Ng in Kramer and Ng (1988). They built an animator to provide facilities for selecting and executing transactions in order to show the

Conclusion and future work

We have presented a tool-supported specification animation technique that can be used to validate formal specifications by comprehensibly demonstrating three aspects for an operation specified using pre- and post-conditions: (1) the input–output relation of the operation, (2) the characteristics of the relevant data items of the operation with voice explanations, and (3) logical expressions and operators used in the formal specification of the operation. We have discussed how adequate animation

Declaration of Competing Interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Acknowledgment

We thank our students for their great contributions to the development of the prototype supporting tool for our animation method. We also thank all the anonymous reviewers for their kind efforts in reviewing our manuscript. This work was supported by JSPS KAKENHI Grant Number 26240008 and NSFC of China 61402178.

Shaoying Liu is currently a Professor of Software Engineering at Hiroshima University, Japan. Previously, he was a Professor at Hosei University from April 2000 to March 2020. He received the Ph.D. in Computer Science from the University of Manchester, U.K in 1992. His research interests include Formal Methods and Formal Engineering Methods for Software Development, Specification Verification and Validation, Specification-based Program Inspection, Automatic Specification-based Testing,

References (64)

  • ChenJ. et al.

    An approach to testing object-oriented formal specifications

  • CrowJ. et al.

    Evaluating, Testing, and Animating PVS Specifications

    (2001)
  • DutertreB.

    Yices 2.2

  • FitzgeraldJ. et al.

    Vdmtools: Advances in support for formal modeling in VDM

    ACM Sigplan Notices

    (2008)
  • FredjM. et al.

    OVADO: enhancing data valiation for safety-critical railway systems

  • GargantiniA. et al.

    Automatic model driven animation of SCR specifications

  • GeryE. et al.

    Rhapsody: A complete life-cycle model-based development system

  • HallerstedeS. et al.

    Refinement animation for event-b - towards a method of validation

  • HayesI. et al.

    Specifications are not (necessarily) executable

    Softw. Eng. J.

    (1989)
  • HazelDaniel et al.

    Possum: an animator for the SUM specification language

  • HewittM. et al.

    Experiences with piza: an animator for z

  • JacksonD.

    Alloy: A lightweight object modeling notation

    ACM Trans. Softw. Eng. Methodol.

    (2002)
  • JacquotJ.P. et al.

    The role of validation in refinement-based formal software development

  • KnightJ.C. et al.

    Why are formal methods not used more widely?

  • KramerJ. et al.

    Animation of requirements specifications

    Softw. - Pract. Exp.

    (1988)
  • KuritaT. et al.

    Application of a formal specification language in the development of the mobile feliCa IC chip firmware for embedding in mobile phones

  • KuritaT. et al.

    Application of a formal specification language in the development of the mobile feliCa IC chip firmware for embedding in mobile phone

  • LarsenP.G. et al.

    The overture initiative integrating tools for VDM

    SIGSOFT Softw. Eng. Notes

    (2010)
  • LarsenP.G. et al.

    Triumphs and challenges for the industrial application of model-oriented formal methods

    (2007)
  • LarsenP.G. et al.

    Combinatorial testing for VDM

  • LeuschelM. et al.

    From animation to data validation: The prob constraint solver 10 years on

  • LeuschelM. et al.

    Prob: A model checker for b

  • Cited by (3)

    Shaoying Liu is currently a Professor of Software Engineering at Hiroshima University, Japan. Previously, he was a Professor at Hosei University from April 2000 to March 2020. He received the Ph.D. in Computer Science from the University of Manchester, U.K in 1992. His research interests include Formal Methods and Formal Engineering Methods for Software Development, Specification Verification and Validation, Specification-based Program Inspection, Automatic Specification-based Testing, Testing-Based Formal Verification, and Intelligent Software Engineering Environments. He has published a book entitled “Formal Engineering for Industrial Software Development” with Springer-Verlag, twelve edited conference proceedings, and over 200 academic papers in refereed journals and international conferences. He proposed to use the terminology of “Formal Engineering Methods” in 1997, and has established Formal Engineering Methods as a research area based on his extensive research on the SOFL (Structured Object-Oriented Formal Language) method since 1989, and the development of ICFEM conference series since 1997. In recent years, he has served as the General Chair of ICFEM 2017 and PC member for numerous international conferences. He is an Associate Editor for IEEE Transactions on Reliability and the Journal of Software Testing, Verification and Reliability (STVR), respectively. He is IEEE Fellow, BCS Fellow and member of JSSST.

    Weikai Miao is an Associate Professor at East China Normal University. He received the Ph.D. in Information Science at Hosei University, Japan in 2014. His research interests include Formal Engineering Methods, SOFL, Software Testing, and Software Verification and have made over 50 publications in various refereed journals and international conferences.

    This work was supported by JSPS, Japan KAKENHI Grant Number 26240008 and NSFC of China 61402178 and 61872144.

    ☆☆

    Editor: David Shepherd.

    View full text