Efficient validation of self-adaptive applications by counterexample probability maximization
Introduction
Self-adaptive applications have been capturing increasing attention recently because of their capabilities to adjust their behavior in response to their perception of environment (Cheng, de Lemos, Giese, et al., 2009, de Lemos, Giese, Müller, et al., 2013, Salehie, Tahvildari, 2009). However, developing high-assurance self-adaptive applications needs nontrivial effort, since developers have to guarantee the correctness of such applications’ predefined business logics while considering the applications’ interactions with environment in presence of uncertainty (de Lemos, Giese, Müller, et al., 2013, Esfahani, Malek, 2013, Giese, Bencomo, Pasquale, et al., 2014). As a result, self-adaptive applications are error-prone (Xu, Cheung, Ma, Cao, Lu, 2012, Kulkarni, Tripathi, 2010, Wang, Elbaum, Rosenblum, 2007), which makes checking the correctness of self-adaptive applications vitally important and significant. Verification, which is regarded as effective methods to achieve a dependable self-adaptive application, can provide evidence that the set of stated functional and non-functional properties are satisfied during the application’s operation (Weyns, Iftikhar, de la Iglesia, Ahmad, 2012, Baier, Katoen, et al., 2008). However, verification results themselves can be imprecise when uncertainty is not appropriately considered. This thus calls for further validation of the verification results.
There are many ways to obtain verification results for self-adaptive applications. Recent promising approaches, developed by others and us, have proposed to verify self-adaptive applications against various properties (e.g., stability and reliability) (Sama, Elbaum, Raimondi, Rosenblum, Wang, 2010, Bartels, Kleine, 2011, Camara, De Lemos, 2012, Filieri, Tamburrelli, 2013, Liu, Xu, Cheung, 2013, Yang, Xu, Liu, Cao, Ma, Lu, 2014). The verification techniques in article (Sama et al., 2010) explore an application’s state space to detect faults, yet they can report numerous false positives that can never happen in the application’s running environment, as the model may over-approximate the behavior of the application, while Liu et al. (2013) and Yang et al. (2014) eliminate the false positives by leveraging environmental constraints. However, most of them did not address the orthogonal but crucial issue of uncertainty in verifying self-adaptive applications. Uncertainty is the gap between certainty and what is currently known (Esfahani, Malek, 2013, Elbaum, Rosenblum, 2014, Aughenbaugh, 2006).
According to existing literature (Esfahani and Malek, 2013), uncertainty can be categorized as aleatory or epistemic. Aleatory uncertainty captures the uncertainty that is caused by randomness and is usually modeled using probabilities, while epistemic uncertainty corresponds to the lack of knowledge and it is usually not possible to represent this kind of uncertainty as a probability distribution. For example, oftentimes the user’s uncertainty in specifying system’s requirements and objectives can be considered as epistemic, since it is due to the lack of knowledge, and not randomness. In self-adaptive applications, uncertainty arises naturally when the applications interact with the environment through inevitably imprecise sensors or flawed actors (Ramirez, Jensen, Cheng, 2012, Esfahani, Kouroshfar, Malek, 2011). This type of uncertainty is deemed to be aleatory, since it is due to randomness. In paper Yang et al. (2014) we proposed to model a self-adaptive application with Interactive State Machine which is a variant of Finite State Machine and exploit an SMT solver (De Moura and Bjørner, 2008) to verify the State Machine with uncertainty modeled by approximated error ranges and distributions. Specifically, we depict the application’s failures by assertions (failure conditions) and collect the application’s behavior by enumerating each candidate path within a bound of path length in the State Machine to get a path condition, in which environment-related variables are augmented with error ranges to include uncertainty. The failure condition and the path condition are then joined and checked for satisfiability. If satisfied, it means the application could fail, and one concrete solution will be returned by the solver. The solution contains an assignment of values for the variables in the failure and path conditions and specifies the environment settings sensed by the application and the application’s internal states that causes the application failure. As this path and the corresponding solution give an example of an application failure which counters the quality requirement, they are together called a counterexample.
While the verification results of self-adaptive applications are informative, they can deviate from reality because of the inaccurate specification of uncertainty (Esfahani, Malek, 2013, D’Ippolito, Braberman, Kramer, Magee, Sykes, Uchitel, 2014). This stresses the need for further validation, especially for critical applications (Zelkowitz and Rus, 2001). The accuracy of verification results has a significant impact on the debugging process, since imprecise results would mislead developers’ attentions. Less important counterexamples could be falsely highlighted, or false counterexamples could be reported. To refine the verification results, validation is a natural and widely-adopted approach. Simulation has been a successful validation approach in many cases (Arrieta, Sagardui, Etxeberria, 2015, Arrieta, Sagardui, Etxeberria, 2014), however, its power is limited for self-adaptive applications, since an imprecise uncertainty specification can result in an inaccurate simulated environment. Thus, in order to validate the verification results (i.e., counterexamples), one has to run the application in real deployment under the environment setting realizing the conditions specified by the counterexample, and observe whether the application follows the path specified by the counterexample. This process is labor-intensive and time-consuming. It is often very difficult to force the application to execute as specified by the counterexample. When validating a self-adaptive application, sensing variables’ values are difficult to control because they depend on the environmental attributes sensed from its environment setting. It is practically infeasible to continually change the environment setting, when the application is running, to make the concerned variables take specific values. Furthermore, the sensing will be affected by uncertainty such as unpredictable noises in its environmental sensing. If one tried to directly set the error values caused by uncertainty, the validation would be meaningless, since these chosen values may not even exist in a real environment. Thus, in order to validate the verification results (i.e., counterexamples), one has to run the application in real deployment, and may have to repeatedly run the application until the counterexample is witnessed to be a real failure. In other words, there is a probability that a counterexample will happen in the real deployment.
The probability has a significant effect on the validation cost for a counterexample. For instance, if a counterexample’s probability is 0.001, in theory, we may need to run the application in real deployment under this counterexample’s setting for about 1000 times to observe an occurrence of the counterexample. This could require a considerable investment. For example, for our experiment subject of a robot-car application, to validate one behavior that only contains ten adaptation steps costs about one minute averagely. Besides, SMT solvers only give counterexamples that satisfy the failure conditions but not guarantee their probabilities, so it is possible many counterexamples’ probability values are small, according to the studies in Yang et al. (2014). It will definitely hinder efficient validation of self-adaptive applications. To expedite the process and mitigate the validation cost, in this paper we propose to efficiently validate a self-adaptive application’s counterexample by finding a path-equivalent counterexample with a higher probability. Counterexamples are path-equivalent if they comprise the same application path but different solutions of the path conditions. With a high probability counterexample, it will be more likely for the application to follow the counterexample’s specified path and fail. This can save much time in validation. As shown in our experiment, the searching for a high probability counterexample only costs several minutes, which is just a small fragment compared with the validation time saved by our approach.
For a self-adaptive application’s counterexample, the root cause of its probability is uncertainty, since uncertainty can affect the application’s sensing and adaptation, which further impact the application’s internal states. Uncertainty that arises from sensing and adaptation can be specified by error ranges and distributions (Esfahani, Malek, 2013, Yang, Xu, Liu, Cao, Ma, Lu, 2014), and with this information, we first propose to formulate a counterexample’s probability into a function. The inputs of the function include variables reflecting the environmental sensing, adaptation, application states and uncertainty. Apparently their values cannot be arbitrary, but on the contrary, subject to constraints (e.g., within some ranges). With this probability function, to find a higher probability counterexample, we just need to maximize the function, i.e., finding a set of inputs such that the probability function can reach its maximum value. Thus, the problem of finding a path-equivalent counterexample with a higher probability is reduced to a constrained optimization problem. To alleviate the cost of solving the optimization problem, we propose to separate the optimization problem into several sub-problems. These smaller problems require less solving cost; and what’s more, their results can be reused when they appear in other counterexamples’ optimization problems. Furthermore, for each (sub-)problem, we devise a method to simplify its objective function for easier computation. We then leverage the genetic algorithm to solve the problem. We show that by maximizing each counterexample’s probability, self-adaptive applications’ verification results can be validated in real deployment more efficiently. The efficiency is two-fold— counterexamples can be validated more efficiently provided that uncertainty is correctly modeled, and imprecise uncertainty modeling can also be exposed faster otherwise. We conducted a series of experiments on real applications and the results confirmed our approach’s effectiveness and efficiency. The main contributions of this paper are:
- •
We propose a novel approach to improving the efficiency of validating verification results for self-adaptive applications by finding high-probability counterexamples. By exploiting the probability function of the counterexample, the problem is reduced to a constrained optimization problem.
- •
We leverage multiple techniques to save the solving cost of the constrained optimization problem: a technique to reduce the problem into smaller sub-problems, a technique to simplify the objective function, and a genetic algorithm based technique to effectively solve the problem.
- •
We implement all concerned algorithms in a prototype tool and evaluate our validation approach on self-adaptive applications with both real and simulation experiments.
The remainder of this paper is organized as follows. Section 2 presents some basic notions concerning self-adaptive applications and uncertainty thereof. Section 3 uses a motivating example to explain the inadequacy of existing work and motivate our work. Section 4 presents our validation approach in detail. Section 5 evaluates our approach with self-adaptive applications. Section 6 discusses related work, and finally Section 7 concludes this paper.
Section snippets
Self-adaptive applications suffering uncertainty
Self-adaptation endows an application with the ability to satisfy certain objectives by automatically modifying its behavior based on the environment at runtime (Cheng, de Lemos, Giese, et al., 2009, Esfahani, Malek, 2013). However, the ever-growing complexity of applications is challenging the development of self-adaptive applications, since self-adaptation can manifest itself in different forms, such as web applications (Moreno et al., 2015) that can adapt to changing load or robotics
Motivating example
We now provide an example that illustrates how the verification is conducted. The verification results will present the challenges of validating self-adaptive applications, which motivates our work.
Consider the simplified code in Fig. 1(a). It is a fragment of a self-adaptive robot-car application that explores an unknown area by the car’s built-in ultrasonic sensors in four directions (Yang, Xu, Liu, Cao, Ma, Lu, 2014, Xu, Yang, Ma, Cao, Lu, 2013, Yang, Xu, Zhang, 2013). The application
Efficient validation approach
In this section we present our approach to validating self-adaptive applications. We begin with an overview of the approach, followed by a fundamental introduction to the optimization theory and a detailed presentation of our approach.
Experimental evaluation
We implemented our approach as a prototype, and our evaluation addresses the following research questions:
- •
RQ1: Can our approach effectively improve the efficiency of validating counterexamples for self-adaptive applications?
- •
RQ2: How does the precision of uncertainty model affect the effectiveness of our approach?
- •
RQ3: How does our genetic algorithm perform in solving the optimization problem compared with other algorithms?
Related work
In this section, we discuss some closely related work concerning quality assurance for self-adaptive applications, uncertainty handling in self-adaptive applications and probabilistic analysis.
Quality assurance. Developing high-quality self-adaptive applications is confronted with stiff challenges (Yang, Liu, Xu, Cheung, 2015, Cheng, Eder, Gogolla, et al., 2014), and many research efforts are made to assure their quality. Some studies address the problem of testing self-adaptive applications (
Conclusion
In this paper, we introduce a novel approach to improving the efficiency of validating counterexamples for self-adaptive applications by finding path-equivalent counterexamples of higher probabilities with respect to original ones. To achieve so, a counterexample’s probability is formulated into a probability function, and acts as the objective function to be maximized in a constrained optimization problem. With the proposed multiple optimization techniques, the optimization problem can be
Acknowledgments
This work was supported in part by National Key R&D Program (Grant No. 2017YFB1001801), National Natural Science Foundation (Grant Nos. 61690204, 61472174), and Jiangsu Natural Science Foundation (Grant No. BK20150589) of China. The authors thank the anonymous reviewers for valuable comments and suggestions for improving this article. The authors would also like to thank the support from the Collaborative Innovation Center of Novel Software Technology and Industrialization, Jiangsu, China.
Wenhua Yang obtained his Ph.D. degree in computer science and technology from the Department of Computer Science and Technology of Nanjing University. His research interests include self-adaptive application verification and validation.
References (79)
- et al.
Hope for the best, prepare for the worst: multi-tier control for adaptive systems
Proceedings of ICSE ’14, Hyderabad, India
(2014) - et al.
Known unknowns: testing in the presence of uncertainty
Proceedings of FSE ’14, Hong Kong, China
(2014) - et al.
Rainbow: architecture-based self-adaptation with reusable infrastructure
IEEE Comput.
(2004) - et al.
Exact and approximate probabilistic symbolic execution for nondeterministic programs
Proceedings of ASE ’14, Vasteras, Sweden
(2014) Combinatorial optimization, cross-entropy, ants and rare events
Stoch. Optimizat. Algo. Appl.
(2001)- et al.
Environment rematching: toward dependability improvement for self-adaptive applications
Proceedings of ASE ’13, Palo Alto, USA
(2013) - et al.
Static analysis of programs with imprecise probabilistic inputs
Verified Software: Theories, Tools, Experiments
(2014) - et al.
A practical guide for using statistical tests to assess randomized algorithms in software engineering
Proceedings of the 33rd International Conference on Software Engineering
(2011) - et al.
A configurable test architecture for the automatic validation of variability-intensive cyber-physical systems
Proceedings of. VALID ’14, Nice, France
(2014) - et al.
Test control algorithms for the validation of cyber-physical systems product lines
Proceedings of SPLC ’15, Nashville, Tennessee
(2015)
Managing Uncertainty in Engineering Design Using Imprecise Probabilities and Principles of Information Economics
Principles of Model Checking
Active sonar for obstacle localization using envelope shape information
Proceedingsof ICASSP 91: 1991 International Conference on Acoustics, Speech, and Signal Processing
A csp-based framework for the specification, verification, and implementation of adaptive systems
Proceedings of SEAMS ’11, Honolulu, USA
Compositional solution space quantification for probabilistic software analysis
Proceedings of PLDI ’14, Edinburgh, United Kingdom
Iterative distribution-aware sampling for probabilistic symbolic execution
Proceedings of Joint ESEC/FSE ’15, Bergamo, Italy
Engineering self-adaptive systems through feedback loops
Software Engineering for Self-Adaptive Systems
A trust region method based on interior point techniques for nonlinear programming
Math. Program.
Self-adaptive software needs quantitative verification at runtime
Commun. ACM
Evaluation of resilience in self-adaptive systems using probabilistic model-checking
Proceedings of SEAMS ’12, Zurich, Switzerland
Carisma: context-aware reflective middleware system for mobile applications
IEEE Trans. Softw. Eng.
Automated abstractions for contract validation
IEEE Trans. Softw. Eng.
Enabledness-based program abstractions for behavior validation
ACM Trans. Softw. Eng. Methodol.
Genetic algorithms in astronomy and astrophysics
Astrophys. J. Suppl. Ser.
Using models at runtime to address assurance for self-adaptive systems
[email protected]
Software engineering for self-adaptive systems: aresearch roadmap
Software Engineering for Self-Adaptive Systems
A goal-based modeling approach to develop requirements of an adaptive system with environmental uncertainty
MODELS
Satisfiability modulo the theory of costs: foundations and applications
Proceedings of TACAS ’10, Paphos, Cyprus
A globally convergent Lagrangian barrier algorithm for optimization with general inequality constraints and simple bounds
Math. Comput.
Z3: an efficient SMT solver
Proceedings of TACAS ’08/ETAPS ’08, Budapest, Hungary
An efficient constraint handling method for genetic algorithms
Comput Methods Appl. Mech. Eng.
A survey of autonomic communications
ACM Trans. Auton. Adapt. Syst.
On the analysis of the (1+1) evolutionary algorithm
Theor. Comput. Sci.
Taming uncertainty in self-adaptive software
Proceedings of Joint ESEC/FSE ’11, Szeged, Hungary
Uncertainty in self-adaptive software systems
Software Engineering for Self-Adaptive Systems II
Reliability analysis in symbolic pathfinder
Proceedings of ICSE ’13, San Francisco, CA, USA
Probabilistic verification at runtime for self-adaptive systems
Assurances for Self-Adaptive Systems
Improving self-adaptation planning through software architecture-based stochastic modeling
J. Syst. Softw.
Probabilistic symbolic execution
Proceedings of ISSTA ’12, Minneapolis, MN, USA
Cited by (0)
Wenhua Yang obtained his Ph.D. degree in computer science and technology from the Department of Computer Science and Technology of Nanjing University. His research interests include self-adaptive application verification and validation.
Chang Xu is a professor with the State Key Laboratory for Novel Software Technology and Department of Computer Science and Technology of Nanjing University. He received his Ph.D. degree in computer science and engineering from The Hong Kong University of Science and Technology. His research interests include big data software engineering, software testing and analysis, and adaptive and embedded system.
Minxue Pan is an assistant professor with the State Key Laboratory for Novel Software Technology and Software Institute of Nanjing University. He received his Ph.D. degree in computer science and technology from Nanjing University. His research interests include formal methods, real-time and embedded systems, and hybrid systems.
Chun Cao is an associate professor with the State Key Laboratory for Novel Software Technology and Department of Computer Science and Technology of Nanjing University. He received his Ph.D. degree in computer science and technology from Nanjing University. His research interests include software architecture, middleware systems and cloud computing.
Xiaoxing Ma is a professor with the State Key Laboratory for Novel Software Technology and Department of Computer Science and Technology of Nanjing University. He received her Ph.D. degree in computer science and technology from Nanjing University. His research interests include self-adaptive software systems, software architecture and middleware systems.
Jian Lu is a professor with the State Key Laboratory for Novel Software Technology and Department of Computer Science and Technology of Nanjing University. He received his Ph.D. degree in computer science and technology from Nanjing University. Her research interests include software engineering and methodology.