An effective sequential statistical test for probabilistic monitoring

https://doi.org/10.1016/j.infsof.2010.10.003Get rights and content

Abstract

Context

A monitor checks if a system behaves according to a specified property at runtime. This is required for quality assurance purposes. Currently several approaches exist to monitor standard and real-time properties. However, a current challenge is to provide a comprehensive approach for monitoring probabilistic properties, as they are used to formulate quality of service requirements like performance, reliability, safety, and availability. The main problem of these probabilistic properties is that there is no binary acceptance condition.

Objective

To overcome this problem, this article presents an improved and generic statistical decision procedure based on acceptance sampling and sequential hypothesis testing.

Method

The developed decision procedure is validated using several experiments that determine the operating characteristic, runtime overhead as well as the expected sample sizes.

Results and conclusion

The experimental validation provides evidence that the developed testing procedure reduces the runtime overhead and improves the accuracy of classification. Thus, the statistical decision procedure is superior to the existing statistical tests currently used in probabilistic monitoring.

Introduction

Regardless of the quality of a development process and the elegance of a system design, incorrect program statements will inevitably be introduced in software systems with a certain complexity. Failures may also occur when systems are dynamically composed at runtime as it is currently promoted with service-oriented systems. As a result, run-time verification and monitoring is fundamental to ensure the correctness of dynamically evolving software systems and receives increasing attention by the research community.

The main idea behind monitoring is to (a) formalize critical system properties that need to be maintained during the operation of a system with high-level property specification languages, (b) translate these properties into code fragments called monitors that observe the system execution and which are able to decide if the properties are fulfilled or not and (c) deploy the monitors together with the application. A considerable number of theoretical approaches have been developed for runtime monitoring of standard [1], [2], [3], [4], [5], [6], [7], [8] and real-time properties [9], [10], [11], [12]. Furthermore, tools like Java PathExplorer [13], Java-MaC [14], JavaMOP/MOP [15], [16] Tracematches [17] and PTQL [18] have been developed to practically support monitoring and runtime verification. However, quality of service (QoS) requirements or service level agreements (SLAs) [19], [20], [21] should also be monitorable. Most of these quality of service (QoS) requirements need to be defined as probabilistic properties [22].

Currently only a few approaches are published that specifically aim at monitoring probabilistic properties with so called approximate statistical monitors [20]. These approaches use different property specification languages and strategies to provide evidence that the system fulfils a probabilistic property. The first approach as proposed by Chan et al. [23] provides a platform for monitoring PCTL [24] properties in .NET applications. To provide statistical evidence that the probabilistic property holds, the approach calculates the ratio between the successful (or unsuccessful) monitoring results and the total number of observations. The second approach by Sammapun et al. [25] uses hypothesis testing based on probabilistic properties defined in a probabilistic extension of the Meta-Event Definition Language (MEDL). This approach is able to investigate properties with a single probabilistic operator and it provides p-values for the significance of the testing outcome. The third approach called probabilistic monitoring (ProMo) by Grunske and Zhang [26] uses the monitorable subset CSLMon of the continuous stochastic logic (CSL) [27], [28]. The sequential probability ratio test is used as a statistical hypothesis testing procedure to determine the outcome of a monitoring result with significants level of α and power of 1  β.

When analyzing the different statistical decision procedures used in the probabilistic monitoring approaches, it becomes evident that each of the approaches has its own weaknesses. As an example, the first approach does not provide any evaluation of the statistical significance of the results. The second approach does not perform a power analysis. The third approach only tests the system once and does not support continuous monitoring. To overcome these problems, the aim of this article is to present an effective and generic statistical procedure for monitoring of probabilistic properties. This procedure should be applicable to any of the existing probabilistic monitoring approaches and further to real-time monitoring approaches, as long as they can provide a binary monitoring result (either success or failure) for a monitoring run. The test procedure should minimize the number of required samples and the runtime overhead to reach a decision.

The rest of the paper is organized as follows: Section 2 introduces the reader to the concepts of probabilistic monitoring and compares the existing approaches. Based on the comparison, requirements for an effective and generic statistical procedure for monitoring of probabilistic properties are derived. Section 3 describes a set of statistical algorithms for monitoring of probabilistic properties. The results of an experimental validation of the developed algorithms are presented in Section 4. Finally, Section 5 concludes the paper.

Section snippets

Fundamental idea

The idea behind monitoring of probabilistic properties is to use statistical hypothesis testing to give statistical evidence that the property is fulfilled or not. A statistical hypothesis test selects a number of samples from a large population and infers a population parameter (e.g. the correctness of a hypothesis) from these samples. Before the test, a null hypothesis H0 and an alternative hypothesis H1 need to be formulated. Both hypotheses need to be mutually exclusive so the rejection of

An effective sequential testing method for monitoring of probabilistic properties

Based on the requirements defined in Section 2.5 this section describes the development of an effective statistical testing method for probabilistic monitoring. Since the statistical testing procedure should minimize the number of required monitoring runs for a statistical test with a predefined confidence and power (REQ1), a sequential statistical testing procedure must be used [32] (please note that the SPRT is optimal for probabilities p at the boundary of the indifference region).

Empirical validation

To test the developed algorithms (Section 3) against the requirements a series of empirical, simulation-based experiments are used as a validation approach in this article. The interested reader is referred to [26], [25], for a practical and case-study-based validation of the original probabilistic monitoring with the ProMo and MaC approach. Requirement REQ1 is satisfied due to the selection of the sequential probability ratio test as a foundation for the probabilistic monitoring framework. The

Conclusion

In this article, an effective statistical testing method for monitoring of probabilistic properties has been introduced. This statistical testing method can be used in any monitoring framework where a single monitoring run has one of two outcomes “success” or “failure” and the monitoring outcomes follow the assumption of a Bernoulli trial. The method is based on the sequential probability ratio test, which is an effective test since in most relevant cases it reduces the number of required

References (32)

  • M. Leucker et al.

    A brief account of runtime verification

    Journal of Logic and Algebraic Programming

    (2009)
  • P. Zhang et al.

    Timed property sequence chart

    Journal of Systems and Software

    (2010)
  • H.L.S. Younes et al.

    Statistical probabilistic model checking with a focus on time-bounded properties

    Information and Computation

    (2006)
  • H. Barringer et al.

    Rule-based runtime verification

  • M.B. Dwyer et al.

    Reducing the cost of path property monitoring through sampling

  • K. Havelund et al.

    Synthesizing monitors for safety properties

  • A. Keller et al.

    Defining and monitoring service-level agreements for dynamic e-business

  • K. Mahbub et al.

    Monitoring WS-agreements: an event calculus-based approach

  • O. Maler et al.

    Monitoring temporal properties of continuous signals

  • G. Rosu et al.

    Rewriting-based techniques for runtime verification

    Automated Software Engineering

    (2005)
  • A. Bauer et al.

    Monitoring of real-time properties

  • K.J. Kristoffersen et al.

    Runtime verification of timed LTL using disjunctive normalized equation systems

    Electronic Notes in Theoretical Computer Science

    (2003)
  • P. Zhang et al.

    Extending PSC for monitoring the timed properties in composite services

  • K. Havelund et al.

    An overview of the runtime verification tool java pathexplorer

    Formal Methods in System Design

    (2004)
  • M. Kim et al.

    Java-maC: A run-time assurance approach for java programs

    Formal Methods in System Design

    (2004)
  • F. Chen et al.

    Java-MOP: A monitoring oriented programming environment for java

  • Cited by (21)

    • IgS-wBSRM: A time-aware Web Service QoS monitoring approach in dynamic environments

      2018, Information and Software Technology
      Citation Excerpt :

      Some of these approaches are based on simple probability calculation [7]. Others are based on classical hypothesis testing SPRT (Sequential Probability Ratio Test [8]) [6,9,10] or on the Bayesian theory [11,12]. However, most existing probabilistic monitoring approaches are only suitable for traditional software or programs.

    • GODA: A goal-oriented requirements engineering framework for runtime dependability analysis

      2016, Information and Software Technology
      Citation Excerpt :

      As previously stated, ways to obtain the probability values are various ones such as computing the system’s MTTF or following a runtime computation approach by Su et al. [45]. Although the monitoring and update procedure of probability values at runtime is out of scope of this work, a very suitable approach that could be incorporated into GODA in the future is the proposal by Grunske [20]. Finally, we plan to extend our GODA framework to further improve the design-time evaluation by supporting a cost-based analysis to achieve a solution that balances dependability and use of system resources, since these are often conflicting requirements.

    • Mobility-Aware Proactive QoS Monitoring for Mobile Edge Computing

      2022, Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
    View all citing articles on Scopus
    View full text