Sabotaging the system boundary: A study of the inter-boundary vulnerability

Abstract

The hierarchy theory is the foundation of the modern computer system design. However, the interaction part between different system layers is usually the weak point of the system, which tends to have security flaws. When communicating across the system boundary, failure to enforce the required synchronization in the shared memory can cause data inconsistency of the communication partners. Especially when there is a privilege gap between different boundary sides, such data inconsistency can lead to security vulnerability and sabotage the trust boundary. In this paper, we propose the concept of inter-boundary vulnerability and give the first in-depth study of them. We investigate three typical boundaries in the system that inter-boundary vulnerabilities are prone to occur, including the kernel-user boundary, the hardware-OS boundary, and the VMM-guest OS boundary. Then, based on the investigation of 115 real-world vulnerability cases, we extract four vulnerability types and provide analysis for each type to illustrate the principle. Finally, we discuss the state-of-the-art techniques that are relevant to the detection, prevention, and exploitation of such vulnerabilities, aiming to light the future research on this topic.

Introduction

The hierarchical structure is the foundation of the computer system, which facilitates system design and implementation. However, owing to the complex functionality, such as data exchange, privilege isolation, and error disposal, the interaction part between different system layers is usually the weakness of the system. Besides, modern optimization schemes, such as concurrent processing and asynchrionization communication, also enlarge the security risks. Thus, communication between different system layers tends to have security flaws.

Shared memory is a fundamental and widespread communication scheme for the inter-domain communication of modern computer systems. The main reason for its popularity is the performance advantage compared to the other message-based communication mechanisms. However, the shared memory scheme is also vulnerable when used for cross-layer communication. Communication-based on shared memory usually requires additional synchronization, such as locks and mutexes. Otherwise, concurrent access to shared data can cause memory corruption errors. These synchronization methods require all communication partners to participate, otherwise, the synchronization cannot be enforced. This usually is not a problem when all communication participants operate on the same privilege level, such as communication between normal processes or between threads. However, when one side of the communication is less privileged, the shared memory interface becomes a trust boundary, and the situation becomes complicated [1]. Since high-level synchronization methods are not enforced in shared memory interfaces, they can simply be ignored, causing data inconsistency. Such data inconsistency can sabotage the trust boundary and cause security vulnerability, which can lead to severe consequences, such as memory corruption errors and sensitive information disclosure. We call it the “inter-boundary vulnerability”.

The inter-boundary vulnerability exists in the communication between different system layers (or domains). Different from the concurrency bugs, the inter-boundary vulnerability usually occurs where there is a privilege gap, known as the trust boundary. Consequently, the misuse of synchronization primitives by one communication partner (usually the privileged one) gives a chance to the less privileged (malicious) partner to cause harmful results to the privileged one. Although there is a large amount of research on the safe use of shared resources, such as the race conditions [2], [3], [4], [5], [6], [7], [8] and concurrency bugs [9], [10], [11], [12], [13], [14], [15], they usually focus on the insecure behavior caused by the misuse of synchronization primitives. They do not take the existence of a malicious communication partner into account, thus, hardly applicable to the detection of security vulnerabilities in the thrust boundary. Therefore, inter-boundary vulnerability is a new research point that supplements these researches.

Previous research has raised the awareness of the double-fetch vulnerability [16], [17], [18], [19], [20]. The double-fetch vulnerability is a subclass of the inter-boundary vulnerability, which is caused by the violation of the read-after-read data dependency between the kernel address space and user address space. However, as we have mentioned above, such vulnerability is theoretically not limited to the kernel-user boundary nor the read-after-read data dependency. Thus, in this paper, we propose the concept of inter-boundary vulnerability and give the first in-depth study of it. We broaden the scope of this topic by studying more system boundary types and analyzing more data dependency types, aiming to light the future research on this topic. In summary, we make the following contributions.

• - We extract three system boundaries that inter-boundary vulnerabilities are prone to occur, including the kernel-user boundary, the hardware-OS boundary, and the VMM-guest OS boundary.

• - We investigate 115 real-world inter-boundary vulnerability cases and categorize four inter-boundary vulnerability types based on the analysis of these cases. We have made the collected vulnerabilities available online for the security community for further research.

• - We review the state-of-the-art techniques that are relevant to the detection, exploitation, and prevention of the inter-boundary vulnerability, and point out challenges for the future work.

The rest of the paper is organized as follows: Section 2 reviews the background knowledge of this topic and introduces the vulnerability-prone boundaries in the system. Section 3 classifies the inter-boundary vulnerabilities and analyzes related works on detection, exploitation, and prevention. Section 4 gives an in-depth analysis of this topic. Section 5 discusses the perspectives and challenges for the future work, followed by conclusions.