Application-based anomaly intrusion detection with dynamic information flow analysis

Abstract

This paper presents a new approach to detecting software security failures, whose primary goal is facilitating identification and repair of security vulnerabilities rather than permitting online response to attacks. The approach is based on online capture of executions and offline execution replay, profiling, and analysis. It employs fine-grained dynamic information flow analysis in conjunction with anomaly detection. This approach, which we call information flow anomaly detection, is capable of detecting a variety of security failures, including both ones that involve violations of confidentiality or integrity requirements and ones that do not. A prototype tool called DynFlow implementing the approach has been developed for use with Java byte code programs. To illustrate the potential of the approach, it is applied to detect security failures of four open source systems. Also, its effectiveness is compared to the effectiveness of an approach to anomaly detection that is based on analyzing method call stacks.

Introduction

Information flow control, which deals with restricting the flow of information between objects manipulated by a program, is a classical subject of computer security research (Fenton, 1974, Denning and Denning, 1977). Information flow control is necessary because controlling access to individual objects is not sufficient to prevent indirect propagation of information resulting in either leakage of information from sensitive objects to untrusted recipients or tampering with sensitive objects by untrusted agents. For example, a program that legitimately accesses confidential information associated with one user may, under certain conditions, disclose it to another user who is not authorized to access it.

Despite its long history, research on information flow control has until recently had relatively little impact on the field of intrusion detection (Denning, 1987), which aims at detecting and responding to intrusions/attacks against systems and applications. This is surprising because the purpose of certain sophisticated attacks against software is precisely to induce insecure information flows to or from sensitive objects, and such flows may be the only indication of such attacks.

Intrusion detection systems are of two principal types (Axelsson, March 2000): signature matching (or misuse detection) systems and anomaly detection systems. Signature matching systems look for intrusion signatures, which are characteristic indications of known attacks. Anomaly detection systems look for system/application behavior that is anomalous in the sense that it differs markedly from normal, safe behavior. Signature matching systems tend to issue fewer “false positive” alerts than anomaly detection systems, but the former cannot detect novel attacks, while the latter can. In practice, both kinds of systems suffer from problems of inaccuracy and imprecision.

Recently, Zimmermann et al., October 2003, Zimmermann et al., November 2003 proposed an intrusion detection model based on runtime enforcement of an information flow policy, which specifies the information flows that are permissible in a given system. They argued that their model detects confidentiality and integrity violations more reliably than either signature matching systems or anomaly detection systems do, because it focuses on policy violations rather than on ancillary events. Although enforcement of information flow policies is very desirable, it is not a complete solution to the problem of revealing security vulnerabilities in software. First, denial-of-service attacks need not cause confidentiality or integrity violations. Second, information flow policies may suffer from the same types of problems that affect other software specifications, such as incorrectness and incompleteness.

In this paper, we present evidence that dynamic information flow analysis (DIFA) has important applications to vulnerability detection besides the enforcement of information flow policies (which we addressed in Masri et al. (2004)). These applications are based on the fact that patterns of information flow that occur during execution of a program characterize its computation to a high degree. Information flows are fundamental aspects of program execution, because they indicate the direct and indirect interactions between program elements (e.g., variables, instructions, and procedures) and the associated dependences between those elements. Such interactions may span much program code and many program components. Information flows, which we assume are transitive in general,¹ also define the essential constraints on the order in which operations are executed, that is, those that determine a program's input/output behavior.²

We investigate the hypothesis that novel attacks often cause anomalous patterns of information flow, which sometimes are the only indication of a security failure. (Note that where an information flow policy exists, an anomalous information flow pattern associated with a security failure need not violate the policy, because the policy may be incomplete or the failure may not involve any illegal flows.) We show that these patterns can be detected by applying anomaly detection techniques to information flow profiles, that is, to execution profiles that characterize a program's dynamic information flows. This suggests that it is feasible to enhance anomaly intrusion detection systems with the capability to detect anomalous information flows associated with attacks. Note, however, that we do not claim that information flows alone are sufficient to characterize the full range of attacks that an IDS must confront.

Zimmermann et al., October 2003, Zimmermann et al., November 2003 policy-based model deals with information flows between entire objects. They assert that a finer-grained online analysis, which would involve analyzing: (1) data flows involving object fields and local variables and (2) control dependences, is unrealistic on a large-scale OS running third-party software (Zimmermann et al., October 2003). Our own prior work suggests that the higher overhead of fine-grained DIFA precludes its online application with processing-intensive applications (Masri, 2004). For use in corrective software maintenance, however, fine-grained DIFA is preferable to coarse-grained DIFA, because the former is more precise and because the analysis can be done offline with the aid of execution capture/replay (Steven et al., 2000, Orso and Kennedy, 2005). In contrast to most work on intrusion detection, including Zimmerman et al.'s, we emphasize offline identification and repair of security vulnerabilities in software rather than online detection and response to attacks. Our approach, which we call information flow anomaly detection (IFAD) calls for executions to be captured online and then replayed, profiled, analyzed, and audited (if suspicious) offline. This makes it feasible to employ fine-grained DIFA and to involve software developers in the analysis and decision-making processes.

In Section 2, we survey work related to dynamic information flow analysis. In Section 3, we discuss the assumptions behind our information flow anomaly detection approach. Section 4 describes our fine-grained DIFA tool DynFlow. Section 5 provides a detailed description of our approach. In Section 6, we illustrate the potential of our approach by applying it in several case studies involving security vulnerabilities in open source software; we also compare the effectiveness of our approach to another one based on analyzing method calls stacks. Section 7 presents our conclusions and future work.

The main contributions of this work are:

• A new semi-automated offline approach to detecting software security failures that combines fine-grained DIFA with anomaly detection techniques to facilitate the identification and repair of security vulnerabilities.

• A prototype implementation of the approach.

• Case studies in which the approach was employed to detect security failures of open source software and was compared to a form of anomaly detection based on analysis of method call stacks and to random sampling.