Efficient validation of self-adaptive applications by counterexample probability maximization

https://doi.org/10.1016/j.jss.2017.12.009Get rights and content

Highlights

  • We improve the efficiency of validating self-adaptive applications.

  • We propose to find high-probability counterexamples for validation.

  • We reduce the problem to a constrained optimization problem.

  • We use the genetic algorithm to effectively search for the optimal solution.

  • We evaluated our approach’s effectiveness on real-world applications.

Abstract

Self-adaptive applications’ executions can be affected by uncertainty factors like unreliable sensing and flawed adaptation and therefore often error-prone. Existing methods can verify the applications suffering uncertainty and report counterexamples. However, such verification results can deviate from reality when the uncertainty specification used in verification is itself imprecise. This thus calls for further validation of reported counterexamples. One outstanding challenge in counterexample validation is that the probabilities of counterexamples occurring in real environment are usually very low, which makes the validation extremely inefficient. In this paper, we propose a novel approach to systematically deriving path-equivalent counterexamples with respect to original ones. The derived counterexamples guarantee to have higher probabilities, making them capable of being validated efficiently in field test. We evaluated our approach with real-world self-adaptive applications. The results reported that our approach significantly increased counterexample probabilities, and the derived counterexamples were also consistently and efficiently validated in both real environment and simulation.

Introduction

Self-adaptive applications have been capturing increasing attention recently because of their capabilities to adjust their behavior in response to their perception of environment (Cheng, de Lemos, Giese, et al., 2009, de Lemos, Giese, Müller, et al., 2013, Salehie, Tahvildari, 2009). However, developing high-assurance self-adaptive applications needs nontrivial effort, since developers have to guarantee the correctness of such applications’ predefined business logics while considering the applications’ interactions with environment in presence of uncertainty (de Lemos, Giese, Müller, et al., 2013, Esfahani, Malek, 2013, Giese, Bencomo, Pasquale, et al., 2014). As a result, self-adaptive applications are error-prone (Xu, Cheung, Ma, Cao, Lu, 2012, Kulkarni, Tripathi, 2010, Wang, Elbaum, Rosenblum, 2007), which makes checking the correctness of self-adaptive applications vitally important and significant. Verification, which is regarded as effective methods to achieve a dependable self-adaptive application, can provide evidence that the set of stated functional and non-functional properties are satisfied during the application’s operation (Weyns, Iftikhar, de la Iglesia, Ahmad, 2012, Baier, Katoen, et al., 2008). However, verification results themselves can be imprecise when uncertainty is not appropriately considered. This thus calls for further validation of the verification results.

There are many ways to obtain verification results for self-adaptive applications. Recent promising approaches, developed by others and us, have proposed to verify self-adaptive applications against various properties (e.g., stability and reliability) (Sama, Elbaum, Raimondi, Rosenblum, Wang, 2010, Bartels, Kleine, 2011, Camara, De Lemos, 2012, Filieri, Tamburrelli, 2013, Liu, Xu, Cheung, 2013, Yang, Xu, Liu, Cao, Ma, Lu, 2014). The verification techniques in article (Sama et al., 2010) explore an application’s state space to detect faults, yet they can report numerous false positives that can never happen in the application’s running environment, as the model may over-approximate the behavior of the application, while Liu et al. (2013) and Yang et al. (2014) eliminate the false positives by leveraging environmental constraints. However, most of them did not address the orthogonal but crucial issue of uncertainty in verifying self-adaptive applications. Uncertainty is the gap between certainty and what is currently known (Esfahani, Malek, 2013, Elbaum, Rosenblum, 2014, Aughenbaugh, 2006).

According to existing literature (Esfahani and Malek, 2013), uncertainty can be categorized as aleatory or epistemic. Aleatory uncertainty captures the uncertainty that is caused by randomness and is usually modeled using probabilities, while epistemic uncertainty corresponds to the lack of knowledge and it is usually not possible to represent this kind of uncertainty as a probability distribution. For example, oftentimes the user’s uncertainty in specifying system’s requirements and objectives can be considered as epistemic, since it is due to the lack of knowledge, and not randomness. In self-adaptive applications, uncertainty arises naturally when the applications interact with the environment through inevitably imprecise sensors or flawed actors (Ramirez, Jensen, Cheng, 2012, Esfahani, Kouroshfar, Malek, 2011). This type of uncertainty is deemed to be aleatory, since it is due to randomness. In paper Yang et al. (2014) we proposed to model a self-adaptive application with Interactive State Machine which is a variant of Finite State Machine and exploit an SMT solver (De Moura and Bjørner, 2008) to verify the State Machine with uncertainty modeled by approximated error ranges and distributions. Specifically, we depict the application’s failures by assertions (failure conditions) and collect the application’s behavior by enumerating each candidate path within a bound of path length in the State Machine to get a path condition, in which environment-related variables are augmented with error ranges to include uncertainty. The failure condition and the path condition are then joined and checked for satisfiability. If satisfied, it means the application could fail, and one concrete solution will be returned by the solver. The solution contains an assignment of values for the variables in the failure and path conditions and specifies the environment settings sensed by the application and the application’s internal states that causes the application failure. As this path and the corresponding solution give an example of an application failure which counters the quality requirement, they are together called a counterexample.

While the verification results of self-adaptive applications are informative, they can deviate from reality because of the inaccurate specification of uncertainty (Esfahani, Malek, 2013, D’Ippolito, Braberman, Kramer, Magee, Sykes, Uchitel, 2014). This stresses the need for further validation, especially for critical applications (Zelkowitz and Rus, 2001). The accuracy of verification results has a significant impact on the debugging process, since imprecise results would mislead developers’ attentions. Less important counterexamples could be falsely highlighted, or false counterexamples could be reported. To refine the verification results, validation is a natural and widely-adopted approach. Simulation has been a successful validation approach in many cases (Arrieta, Sagardui, Etxeberria, 2015, Arrieta, Sagardui, Etxeberria, 2014), however, its power is limited for self-adaptive applications, since an imprecise uncertainty specification can result in an inaccurate simulated environment. Thus, in order to validate the verification results (i.e., counterexamples), one has to run the application in real deployment under the environment setting realizing the conditions specified by the counterexample, and observe whether the application follows the path specified by the counterexample. This process is labor-intensive and time-consuming. It is often very difficult to force the application to execute as specified by the counterexample. When validating a self-adaptive application, sensing variables’ values are difficult to control because they depend on the environmental attributes sensed from its environment setting. It is practically infeasible to continually change the environment setting, when the application is running, to make the concerned variables take specific values. Furthermore, the sensing will be affected by uncertainty such as unpredictable noises in its environmental sensing. If one tried to directly set the error values caused by uncertainty, the validation would be meaningless, since these chosen values may not even exist in a real environment. Thus, in order to validate the verification results (i.e., counterexamples), one has to run the application in real deployment, and may have to repeatedly run the application until the counterexample is witnessed to be a real failure. In other words, there is a probability that a counterexample will happen in the real deployment.

The probability has a significant effect on the validation cost for a counterexample. For instance, if a counterexample’s probability is 0.001, in theory, we may need to run the application in real deployment under this counterexample’s setting for about 1000 times to observe an occurrence of the counterexample. This could require a considerable investment. For example, for our experiment subject of a robot-car application, to validate one behavior that only contains ten adaptation steps costs about one minute averagely. Besides, SMT solvers only give counterexamples that satisfy the failure conditions but not guarantee their probabilities, so it is possible many counterexamples’ probability values are small, according to the studies in Yang et al. (2014). It will definitely hinder efficient validation of self-adaptive applications. To expedite the process and mitigate the validation cost, in this paper we propose to efficiently validate a self-adaptive application’s counterexample by finding a path-equivalent counterexample with a higher probability. Counterexamples are path-equivalent if they comprise the same application path but different solutions of the path conditions. With a high probability counterexample, it will be more likely for the application to follow the counterexample’s specified path and fail. This can save much time in validation. As shown in our experiment, the searching for a high probability counterexample only costs several minutes, which is just a small fragment compared with the validation time saved by our approach.

For a self-adaptive application’s counterexample, the root cause of its probability is uncertainty, since uncertainty can affect the application’s sensing and adaptation, which further impact the application’s internal states. Uncertainty that arises from sensing and adaptation can be specified by error ranges and distributions (Esfahani, Malek, 2013, Yang, Xu, Liu, Cao, Ma, Lu, 2014), and with this information, we first propose to formulate a counterexample’s probability into a function. The inputs of the function include variables reflecting the environmental sensing, adaptation, application states and uncertainty. Apparently their values cannot be arbitrary, but on the contrary, subject to constraints (e.g., within some ranges). With this probability function, to find a higher probability counterexample, we just need to maximize the function, i.e., finding a set of inputs such that the probability function can reach its maximum value. Thus, the problem of finding a path-equivalent counterexample with a higher probability is reduced to a constrained optimization problem. To alleviate the cost of solving the optimization problem, we propose to separate the optimization problem into several sub-problems. These smaller problems require less solving cost; and what’s more, their results can be reused when they appear in other counterexamples’ optimization problems. Furthermore, for each (sub-)problem, we devise a method to simplify its objective function for easier computation. We then leverage the genetic algorithm to solve the problem. We show that by maximizing each counterexample’s probability, self-adaptive applications’ verification results can be validated in real deployment more efficiently. The efficiency is two-fold— counterexamples can be validated more efficiently provided that uncertainty is correctly modeled, and imprecise uncertainty modeling can also be exposed faster otherwise. We conducted a series of experiments on real applications and the results confirmed our approach’s effectiveness and efficiency. The main contributions of this paper are:

  • We propose a novel approach to improving the efficiency of validating verification results for self-adaptive applications by finding high-probability counterexamples. By exploiting the probability function of the counterexample, the problem is reduced to a constrained optimization problem.

  • We leverage multiple techniques to save the solving cost of the constrained optimization problem: a technique to reduce the problem into smaller sub-problems, a technique to simplify the objective function, and a genetic algorithm based technique to effectively solve the problem.

  • We implement all concerned algorithms in a prototype tool and evaluate our validation approach on self-adaptive applications with both real and simulation experiments.

The remainder of this paper is organized as follows. Section 2 presents some basic notions concerning self-adaptive applications and uncertainty thereof. Section 3 uses a motivating example to explain the inadequacy of existing work and motivate our work. Section 4 presents our validation approach in detail. Section 5 evaluates our approach with self-adaptive applications. Section 6 discusses related work, and finally Section 7 concludes this paper.

Section snippets

Self-adaptive applications suffering uncertainty

Self-adaptation endows an application with the ability to satisfy certain objectives by automatically modifying its behavior based on the environment at runtime (Cheng, de Lemos, Giese, et al., 2009, Esfahani, Malek, 2013). However, the ever-growing complexity of applications is challenging the development of self-adaptive applications, since self-adaptation can manifest itself in different forms, such as web applications (Moreno et al., 2015) that can adapt to changing load or robotics

Motivating example

We now provide an example that illustrates how the verification is conducted. The verification results will present the challenges of validating self-adaptive applications, which motivates our work.

Consider the simplified code in Fig. 1(a). It is a fragment of a self-adaptive robot-car application that explores an unknown area by the car’s built-in ultrasonic sensors in four directions (Yang, Xu, Liu, Cao, Ma, Lu, 2014, Xu, Yang, Ma, Cao, Lu, 2013, Yang, Xu, Zhang, 2013). The application

Efficient validation approach

In this section we present our approach to validating self-adaptive applications. We begin with an overview of the approach, followed by a fundamental introduction to the optimization theory and a detailed presentation of our approach.

Experimental evaluation

We implemented our approach as a prototype, and our evaluation addresses the following research questions:

  • RQ1: Can our approach effectively improve the efficiency of validating counterexamples for self-adaptive applications?

  • RQ2: How does the precision of uncertainty model affect the effectiveness of our approach?

  • RQ3: How does our genetic algorithm perform in solving the optimization problem compared with other algorithms?

Related work

In this section, we discuss some closely related work concerning quality assurance for self-adaptive applications, uncertainty handling in self-adaptive applications and probabilistic analysis.

Quality assurance. Developing high-quality self-adaptive applications is confronted with stiff challenges (Yang, Liu, Xu, Cheung, 2015, Cheng, Eder, Gogolla, et al., 2014), and many research efforts are made to assure their quality. Some studies address the problem of testing self-adaptive applications (

Conclusion

In this paper, we introduce a novel approach to improving the efficiency of validating counterexamples for self-adaptive applications by finding path-equivalent counterexamples of higher probabilities with respect to original ones. To achieve so, a counterexample’s probability is formulated into a probability function, and acts as the objective function to be maximized in a constrained optimization problem. With the proposed multiple optimization techniques, the optimization problem can be

Acknowledgments

This work was supported in part by National Key R&D Program (Grant No. 2017YFB1001801), National Natural Science Foundation (Grant Nos. 61690204, 61472174), and Jiangsu Natural Science Foundation (Grant No. BK20150589) of China. The authors thank the anonymous reviewers for valuable comments and suggestions for improving this article. The authors would also like to thank the support from the Collaborative Innovation Center of Novel Software Technology and Industrialization, Jiangsu, China.

Wenhua Yang obtained his Ph.D. degree in computer science and technology from the Department of Computer Science and Technology of Nanjing University. His research interests include self-adaptive application verification and validation.

References (79)

  • J.M. Aughenbaugh

    Managing Uncertainty in Engineering Design Using Imprecise Probabilities and Principles of Information Economics

    (2006)
  • C. Baier et al.

    Principles of Model Checking

    (2008)
  • B. Barshan et al.

    Active sonar for obstacle localization using envelope shape information

    Proceedingsof ICASSP 91: 1991 International Conference on Acoustics, Speech, and Signal Processing

    (1991)
  • B. Bartels et al.

    A csp-based framework for the specification, verification, and implementation of adaptive systems

    Proceedings of SEAMS ’11, Honolulu, USA

    (2011)
  • M. Borges et al.

    Compositional solution space quantification for probabilistic software analysis

    Proceedings of PLDI ’14, Edinburgh, United Kingdom

    (2014)
  • M. Borges et al.

    Iterative distribution-aware sampling for probabilistic symbolic execution

    Proceedings of Joint ESEC/FSE ’15, Bergamo, Italy

    (2015)
  • Y. Brun et al.

    Engineering self-adaptive systems through feedback loops

    Software Engineering for Self-Adaptive Systems

    (2009)
  • H.R. Byrd et al.

    A trust region method based on interior point techniques for nonlinear programming

    Math. Program.

    (2000)
  • R. Calinescu et al.

    Self-adaptive software needs quantitative verification at runtime

    Commun. ACM

    (2012)
  • J. Camara et al.

    Evaluation of resilience in self-adaptive systems using probabilistic model-checking

    Proceedings of SEAMS ’12, Zurich, Switzerland

    (2012)
  • L. Capra et al.

    Carisma: context-aware reflective middleware system for mobile applications

    IEEE Trans. Softw. Eng.

    (2003)
  • G. de Caso et al.

    Automated abstractions for contract validation

    IEEE Trans. Softw. Eng.

    (2012)
  • G.D. Caso et al.

    Enabledness-based program abstractions for behavior validation

    ACM Trans. Softw. Eng. Methodol.

    (2013)
  • P. Charbonneau

    Genetic algorithms in astronomy and astrophysics

    Astrophys. J. Suppl. Ser.

    (1995)
  • B. Cheng et al.

    Using models at runtime to address assurance for self-adaptive systems

    [email protected]

    (2014)
  • B.H.C. Cheng et al.

    Software engineering for self-adaptive systems: aresearch roadmap

    Software Engineering for Self-Adaptive Systems

    (2009)
  • B.H.C. Cheng et al.

    A goal-based modeling approach to develop requirements of an adaptive system with environmental uncertainty

    MODELS

    (2009)
  • A. Cimatti et al.

    Satisfiability modulo the theory of costs: foundations and applications

    Proceedings of TACAS ’10, Paphos, Cyprus

    (2010)
  • A.R. Conn et al.

    A globally convergent Lagrangian barrier algorithm for optimization with general inequality constraints and simple bounds

    Math. Comput.

    (1997)
  • L. De Moura et al.

    Z3: an efficient SMT solver

    Proceedings of TACAS ’08/ETAPS ’08, Budapest, Hungary

    (2008)
  • K. Deb

    An efficient constraint handling method for genetic algorithms

    Comput Methods Appl. Mech. Eng.

    (2000)
  • S. Dobson et al.

    A survey of autonomic communications

    ACM Trans. Auton. Adapt. Syst.

    (2006)
  • S. Droste et al.

    On the analysis of the (1+1) evolutionary algorithm

    Theor. Comput. Sci.

    (2002)
  • N. Esfahani et al.

    Taming uncertainty in self-adaptive software

    Proceedings of Joint ESEC/FSE ’11, Szeged, Hungary

    (2011)
  • N. Esfahani et al.

    Uncertainty in self-adaptive software systems

    Software Engineering for Self-Adaptive Systems II

    (2013)
  • A. Filieri et al.

    Reliability analysis in symbolic pathfinder

    Proceedings of ICSE ’13, San Francisco, CA, USA

    (2013)
  • A. Filieri et al.

    Probabilistic verification at runtime for self-adaptive systems

    Assurances for Self-Adaptive Systems

    (2013)
  • J.a.M. Franco et al.

    Improving self-adaptation planning through software architecture-based stochastic modeling

    J. Syst. Softw.

    (2016)
  • J. Geldenhuys et al.

    Probabilistic symbolic execution

    Proceedings of ISSTA ’12, Minneapolis, MN, USA

    (2012)
  • Cited by (0)

    Wenhua Yang obtained his Ph.D. degree in computer science and technology from the Department of Computer Science and Technology of Nanjing University. His research interests include self-adaptive application verification and validation.

    Chang Xu is a professor with the State Key Laboratory for Novel Software Technology and Department of Computer Science and Technology of Nanjing University. He received his Ph.D. degree in computer science and engineering from The Hong Kong University of Science and Technology. His research interests include big data software engineering, software testing and analysis, and adaptive and embedded system.

    Minxue Pan is an assistant professor with the State Key Laboratory for Novel Software Technology and Software Institute of Nanjing University. He received his Ph.D. degree in computer science and technology from Nanjing University. His research interests include formal methods, real-time and embedded systems, and hybrid systems.

    Chun Cao is an associate professor with the State Key Laboratory for Novel Software Technology and Department of Computer Science and Technology of Nanjing University. He received his Ph.D. degree in computer science and technology from Nanjing University. His research interests include software architecture, middleware systems and cloud computing.

    Xiaoxing Ma is a professor with the State Key Laboratory for Novel Software Technology and Department of Computer Science and Technology of Nanjing University. He received her Ph.D. degree in computer science and technology from Nanjing University. His research interests include self-adaptive software systems, software architecture and middleware systems.

    Jian Lu is a professor with the State Key Laboratory for Novel Software Technology and Department of Computer Science and Technology of Nanjing University. He received his Ph.D. degree in computer science and technology from Nanjing University. Her research interests include software engineering and methodology.

    View full text