An effective sequential statistical test for probabilistic monitoring
Introduction
Regardless of the quality of a development process and the elegance of a system design, incorrect program statements will inevitably be introduced in software systems with a certain complexity. Failures may also occur when systems are dynamically composed at runtime as it is currently promoted with service-oriented systems. As a result, run-time verification and monitoring is fundamental to ensure the correctness of dynamically evolving software systems and receives increasing attention by the research community.
The main idea behind monitoring is to (a) formalize critical system properties that need to be maintained during the operation of a system with high-level property specification languages, (b) translate these properties into code fragments called monitors that observe the system execution and which are able to decide if the properties are fulfilled or not and (c) deploy the monitors together with the application. A considerable number of theoretical approaches have been developed for runtime monitoring of standard [1], [2], [3], [4], [5], [6], [7], [8] and real-time properties [9], [10], [11], [12]. Furthermore, tools like Java PathExplorer [13], Java-MaC [14], JavaMOP/MOP [15], [16] Tracematches [17] and PTQL [18] have been developed to practically support monitoring and runtime verification. However, quality of service (QoS) requirements or service level agreements (SLAs) [19], [20], [21] should also be monitorable. Most of these quality of service (QoS) requirements need to be defined as probabilistic properties [22].
Currently only a few approaches are published that specifically aim at monitoring probabilistic properties with so called approximate statistical monitors [20]. These approaches use different property specification languages and strategies to provide evidence that the system fulfils a probabilistic property. The first approach as proposed by Chan et al. [23] provides a platform for monitoring PCTL [24] properties in .NET applications. To provide statistical evidence that the probabilistic property holds, the approach calculates the ratio between the successful (or unsuccessful) monitoring results and the total number of observations. The second approach by Sammapun et al. [25] uses hypothesis testing based on probabilistic properties defined in a probabilistic extension of the Meta-Event Definition Language (MEDL). This approach is able to investigate properties with a single probabilistic operator and it provides p-values for the significance of the testing outcome. The third approach called probabilistic monitoring (ProMo) by Grunske and Zhang [26] uses the monitorable subset CSLMon of the continuous stochastic logic (CSL) [27], [28]. The sequential probability ratio test is used as a statistical hypothesis testing procedure to determine the outcome of a monitoring result with significants level of α and power of 1 − β.
When analyzing the different statistical decision procedures used in the probabilistic monitoring approaches, it becomes evident that each of the approaches has its own weaknesses. As an example, the first approach does not provide any evaluation of the statistical significance of the results. The second approach does not perform a power analysis. The third approach only tests the system once and does not support continuous monitoring. To overcome these problems, the aim of this article is to present an effective and generic statistical procedure for monitoring of probabilistic properties. This procedure should be applicable to any of the existing probabilistic monitoring approaches and further to real-time monitoring approaches, as long as they can provide a binary monitoring result (either success or failure) for a monitoring run. The test procedure should minimize the number of required samples and the runtime overhead to reach a decision.
The rest of the paper is organized as follows: Section 2 introduces the reader to the concepts of probabilistic monitoring and compares the existing approaches. Based on the comparison, requirements for an effective and generic statistical procedure for monitoring of probabilistic properties are derived. Section 3 describes a set of statistical algorithms for monitoring of probabilistic properties. The results of an experimental validation of the developed algorithms are presented in Section 4. Finally, Section 5 concludes the paper.
Section snippets
Fundamental idea
The idea behind monitoring of probabilistic properties is to use statistical hypothesis testing to give statistical evidence that the property is fulfilled or not. A statistical hypothesis test selects a number of samples from a large population and infers a population parameter (e.g. the correctness of a hypothesis) from these samples. Before the test, a null hypothesis H0 and an alternative hypothesis H1 need to be formulated. Both hypotheses need to be mutually exclusive so the rejection of
An effective sequential testing method for monitoring of probabilistic properties
Based on the requirements defined in Section 2.5 this section describes the development of an effective statistical testing method for probabilistic monitoring. Since the statistical testing procedure should minimize the number of required monitoring runs for a statistical test with a predefined confidence and power (REQ1), a sequential statistical testing procedure must be used [32] (please note that the SPRT is optimal for probabilities p at the boundary of the indifference region).
Empirical validation
To test the developed algorithms (Section 3) against the requirements a series of empirical, simulation-based experiments are used as a validation approach in this article. The interested reader is referred to [26], [25], for a practical and case-study-based validation of the original probabilistic monitoring with the ProMo and MaC approach. Requirement REQ1 is satisfied due to the selection of the sequential probability ratio test as a foundation for the probabilistic monitoring framework. The
Conclusion
In this article, an effective statistical testing method for monitoring of probabilistic properties has been introduced. This statistical testing method can be used in any monitoring framework where a single monitoring run has one of two outcomes “success” or “failure” and the monitoring outcomes follow the assumption of a Bernoulli trial. The method is based on the sequential probability ratio test, which is an effective test since in most relevant cases it reduces the number of required
References (32)
- et al.
A brief account of runtime verification
Journal of Logic and Algebraic Programming
(2009) - et al.
Timed property sequence chart
Journal of Systems and Software
(2010) - et al.
Statistical probabilistic model checking with a focus on time-bounded properties
Information and Computation
(2006) - et al.
Rule-based runtime verification
- et al.
Reducing the cost of path property monitoring through sampling
- et al.
Synthesizing monitors for safety properties
- et al.
Defining and monitoring service-level agreements for dynamic e-business
- et al.
Monitoring WS-agreements: an event calculus-based approach
- et al.
Monitoring temporal properties of continuous signals
- et al.
Rewriting-based techniques for runtime verification
Automated Software Engineering
(2005)
Monitoring of real-time properties
Runtime verification of timed LTL using disjunctive normalized equation systems
Electronic Notes in Theoretical Computer Science
Extending PSC for monitoring the timed properties in composite services
An overview of the runtime verification tool java pathexplorer
Formal Methods in System Design
Java-maC: A run-time assurance approach for java programs
Formal Methods in System Design
Java-MOP: A monitoring oriented programming environment for java
Cited by (21)
IgS-wBSRM: A time-aware Web Service QoS monitoring approach in dynamic environments
2018, Information and Software TechnologyCitation Excerpt :Some of these approaches are based on simple probability calculation [7]. Others are based on classical hypothesis testing SPRT (Sequential Probability Ratio Test [8]) [6,9,10] or on the Bayesian theory [11,12]. However, most existing probabilistic monitoring approaches are only suitable for traditional software or programs.
GODA: A goal-oriented requirements engineering framework for runtime dependability analysis
2016, Information and Software TechnologyCitation Excerpt :As previously stated, ways to obtain the probability values are various ones such as computing the system’s MTTF or following a runtime computation approach by Su et al. [45]. Although the monitoring and update procedure of probability values at runtime is out of scope of this work, a very suitable approach that could be incorporated into GODA in the future is the proposal by Grunske [20]. Finally, we plan to extend our GODA framework to further improve the design-time evaluation by supporting a cost-based analysis to achieve a solution that balances dependability and use of system resources, since these are often conflicting requirements.
Mobility-Aware Proactive QoS Monitoring for Mobile Edge Computing
2022, Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)M-BSRM: Multivariate BayeSian Runtime QoS Monitoring Using Point Mutual Information
2022, IEEE Transactions on Services ComputingMobility and Dependence-Aware QoS Monitoring in Mobile Edge Computing
2021, IEEE Transactions on Cloud Computing