Protecting host-based intrusion detectors through virtual machines

Abstract

Intrusion detection systems continuously watch the activity on a network or computer, looking for attack and intrusion evidences. However, host-based intrusion detectors are particularly vulnerable, as they can be disabled or tampered by successful intruders. This work proposes and implements an architecture model aimed to protect host-based intrusion detectors, through the application of the virtual machine concept. Virtual machine environments are becoming an interesting alternative for several computing systems due to their advantages in terms of cost and portability. The architecture proposed here makes use of the execution spaces separation provided by a virtual machine monitor, in order to separate the intrusion detection system from the system under monitoring. As a consequence, the intrusion detector becomes invisible and inaccessible to intruders. The prototype implementation and the tests performed show the viability of this solution.

Introduction

Several tools contribute to improve the security of a computing system. Among them, intrusion detection systems (IDS) stand out. Such systems continuously watch the system activity, looking for attacks and intrusion evidences. Network-based intrusion detectors scans data collected from the network to detect malicious activity, and can be installed on dedicated, well-protected machines. On the other hand, host-based intrusion detectors analyze local data collected from computing hosts. Because they run as processes in the monitored system, they are particularly vulnerable to successful intruders. Once an intruder enters the system, it is able to defeat or modify the intrusion detector in order to hide its presence.

Virtual machines can be used to improve the security of a computing system against attacks to its services [6]. The virtual machine concept was defined in the 1960s: in the IBM VM/370 environment, virtual machines were used to create an exclusive environment for each user [12]. The use of virtual machines is becoming interesting also in modern computing systems due to their advantages in terms of cost and portability [5]. Examples of currently used virtual machines environments are VMware [26] and UML – User-Mode Linux [7]. A frequent use of virtual machine-based systems is the so-called server consolidation: instead of using several physical equipments, one uses a single (and more powerful) hardware equipment, in which several distinct, isolated virtual machines host distinct operating systems, applications, and services.

This work proposes and implements an architecture model aimed to protect host-based intrusion detectors, through the application of the virtual machine concept. The architecture proposed here makes use of the execution spaces separation provided by a virtual machine monitor, in order to separate the intrusion detection system from the system under monitoring. This separation protects the intrusion detector, as it becomes invisible and inaccessible to guest processes (and to eventual intruders). Through modifications on the virtual machine monitor, it is possible to transparently collect detailed information about the guest operating system activity. This data is then sent to an external intrusion detector, running on the underlying host operating system. Comparing it against a previously generated behavior database (created from previous executions), the intrusion detector can look for behavior deviations in guest processes. If an intrusion is suspected, a response system can act in order to prevent or defeat it. This feature is easily implemented by intercepting the system calls issued by guest processes.

This article is structured as follows: Section 2 recalls some virtual machine concepts used in this work, Section 3 reviews intrusion detection techniques, Section 4 details the proposal, Section 5 describes the current implementation, Section 6 presents experimental results, and Section 7 discusses related works.