Elsevier

Computers & Security

Volume 31, Issue 2, March 2012, Pages 176-191
Computers & Security

Toward a general defense against kernel queue hooking attacks

https://doi.org/10.1016/j.cose.2011.12.007Get rights and content

Abstract

Kernel queue hooking (KQH) attacks achieve stealthy malicious function execution by embedding malicious hooks in dynamic kernel schedulable queues (K-Queues). Because they keep kernel code and persistent hooks intact, they can evade detection of state-of-the-art kernel integrity monitors. Moreover, they have been used by advanced malware such as the Rustock spam bot to achieve malicious goals. In this paper, we present a systematic defense against such novel attacks. We propose the Precise Lookahead Checking of function Pointers approach that checks the legitimacy of pending K-Queue callback requests by proactively checking function pointers that may be invoked by the callback function. To facilitate the derivation of specifications for any K-Queue, we build a unified static analysis framework and a toolset that can derive from kernel source code properties of legitimate K-Queue requests and turn them into source code for the runtime checker. We implement proof-of-concept runtime checkers for four K-Queues in Linux and perform a comprehensive experimental evaluation of these checkers, which shows that our defense is effective against KQH attacks.

Introduction

Rootkits have become one of the most dangerous threats to systems security. Because they run at the same privilege level as the operating systems kernel, they can modify the OS behavior in arbitrary ways. For example, they can tamper with existing code and/or data of the OS to conceal the runtime state of the system in terms of running processes, network connections, and files. Besides, they can add new functionalities to the OS to carry out malicious activities such as key logging and sensitive information collection. In recent years, significant work has been proposed to detect (Baliga et al., 2008; Carbone et al., 2009; Yin et al., 2010; Garfinkel and Rosenblum, 2003; Jiang et al., 2007; Petroni et al., 2004; Petroni and Hicks, 2007; Rutkowska, 2005; Wang et al., 2005) analyze (Lanzi et al., 2009; Riley et al., 2009; Wang et al., 2008; Yin et al., 2007, 2008), or defend against (Litty et al., 2008; Riley et al., 2008; Seshadri et al., 2007; Wang et al., 2009) rootkits.

However, existing work on rootkits mainly focuses on attacks that change legitimate kernel code (Petroni et al., 2004; Litty et al., 2008; Riley et al., 2008; Seshadri et al., 2007) or change legitimate kernel hooks (locations in kernel space that hold function pointers) (Petroni and Hicks, 2007; Wang et al., 2009), but falls short of attacks that create malicious hooks in dynamically allocated kernel objects, as demonstrated by kernel queue hooking (KQH) attacks (Section 2). Briefly speaking, KQH rootkits leverage various callback mechanisms of the kernel, which enable the rootkits to direct kernel control flow as effectively as exploiting buffer overflows, and by reusing legitimate kernel code, these rootkits can successfully hijack control flow of the victim kernel, yet remain invisible to state-of-the-art defense techniques. Specifically, KQH attacks are unique in three important ways:

  • They do not hijack existing, legitimate kernel hooks; instead, they create their own malicious kernel hooks.

  • They leverage kernel data structures that can have multiple instances at the same time. For example, although the Linux kernel allows each type of interrupt to have only one handler registered in the interrupt descriptor table (IDT), it allows multiple IRQ action handlers for the same interrupt to coexist.

  • They leverage dynamic kernel data structures. Again take IRQ action handlers as example, the kernel uses a queue to keep track of currently registered IRQ action handlers and this queue can grow or shrink at runtime depending on which entities are interested in a particular IRQ, including rootkits.

Therefore, it is much harder to detect malicious manipulations of the IRQ action queue than those of the IDT because it is non-trivial to find the known-good values for the IRQ action queue; and it is also much harder to defend against KQH attacks on the IRQ action queue. For example, we cannot simply make the embedded hooks immutable (as proposed by HookSafe (Wang et al., 2009)), because we do not know whether a hook is benign or malicious to start with.

For ease of presentation, we call data structures such as the IRQ action queue kernel schedulable queues (or K-Queues for short), and attacks that insert malicious requests to such queue-like data structures K-Queue hooking (KQH) attacks. We elaborate on the difference between KQH attacks and other attacks in Section 3.2.

Moreover, as we will discuss in Section 2.2, advanced and real malware is already misusing K-Queues to their advantage; examples of such malware include major spam bots such as Rustock, Pushdo/Cutwail, and Storm/Peacomm.

Therefore, the technical novelty and the realistic threat of KQH attacks call for new defense approaches. The challenge is that malicious data objects share the same K-Queues with legitimate data objects, and both kinds can be created and destroyed at runtime. So a reasonable defense has to check the legitimacy of each object each time the kernel uses it for control transfer decisions (because such objects may not be persistent). To this end, we design and implement a hypervisor-supported reference monitor that intercepts and rejects malicious callback requests while allowing legitimate kernel callback requests to proceed. Secondly, it is very subtle and tricky to develop a specification for legitimate kernel data objects. Manually doing this for a code base as large as the Linux kernel is hopeless and error-prone at best. To address this issue, we build automated tools to derive such specifications more efficiently. Moreover, we employ code generation techniques to automatically translate the inferred specifications into runtime check code.

Specifically, we employ static program analysis (e.g., points-to analysis and transitive closure analysis) of the kernel source code to derive legitimacy specifications. To facilitate the generation of specifications for the entire class of K-Queues, we build a unified static analysis framework and a set of tools that can not only derive specifications from kernel source code, but also generate the corresponding checking code for different K-Queue instances. Furthermore, we design and implement a proof-of-concept runtime reference monitor that checks pending requests for four types of K-Queues (IRQ action queue, tasklet queue, soft timer queue, and task queue) in the Linux kernel and guards the check result against tampering. Finally, we perform a comprehensive experimental evaluation of our tools as well as a detailed performance overhead analysis of the reference monitor, which shows that our approach is able to detect all sample KQH rootkits that we have.

The rest of this paper is organized as follows. Section 2 introduces KQH attacks and the threats they pose to systems security. Section 3 presents our modeling of KQH attacks and a high level description of our defense. Section 4 discusses our design of a unified static analysis framework and tools that can generate specifications of legitimate K-Queue requests and turn them into checking code. Section 5 presents our implementation of the defense. Section 6 discusses evaluation of our defense, Section 7 talks about related work, and Section 8 concludes the paper.

Section snippets

Kernel queue hooking

Kernel Queue Hooking (KQH) is a rootkit technique that leverages extensible kernel data structures, such as queues, to inject malicious control flows into the victim kernel. The queues contain instances of dynamic kernel data structures and they can be hooked because they are organized in linked lists. Finally, running malicious logic is possible because such data structures contain function pointers.

Threat modeling

We adopt the standard queuing model for K-Queues. That is, each K-Queue contains a request queue and a server. A request is inserted into the queue by an enqueue operation and it is removed from the queue by a dequeue operation by the server. This model works well for the K-Queues. For example, all instances of K-Queues provide APIs (the enqueue operations) for submitting an execution request for some callback function, and they all have a dispatch engine (the server) that takes the request

Overview

We build a unified static analysis framework (Fig. 8) and a set of tools that can be used to derive security specifications for legitimate K-Queue requests. We assume that source code of the kernel is available for a static analysis, and such code is the only one that can be trusted. We note that although details of different K-Queues may vary, their specifications can be derived by a common set of analysis tasks. For example, the top-level legitimate K-Queue callback functions can be derived

The K-queue analyzers

We implement the static analyzers for the IRQ action queue, the tasklet queue, the task queue, and the soft timer queue based on our static analysis framework, using CIL (C Intermediate Language) (Necula et al., 2002). We implement the analysis engine in Shell scripts, which invokes our CIL modules that implement the basic analysis tools (Section 4.5). CIL provides an implementation of OLF (Das, 2000), but it is not field-sensitive, so we improve it to satisfy our needs. All of our CIL modules

Effectiveness against KQH attacks

We test the effectiveness of our K-Queue defense by running synthetic proof-of-concept rootkits in the guest kernel, including the keylogger and the CPU cycle stealer described in (Wei et al., 2008), and the keylogger described in Section 2. These rootkits leverage the Linux K-Queues in Table 1 and employ type 1 or type 2 attacks (Section 3.2.3). We observe that our K-Queue guards can immediately detect such rootkits. This shows that our defense is effective again KQH attacks.

Our K-Queue

Related work

Rootkits have attracted a lot of attention in the research community. Existing work can be classified into three areas: detection, defense, and analysis.

Related work in detecting rootkit execution includes integrity-based approaches such as Gibraltar (Baliga et al., 2008), HookScout (Yin et al., 2010), Livewire (Garfinkel and Rosenblum, 2003), Copilot (Petroni et al., 2004), SBCFI (Petroni and Hicks, 2007), Patagonix (Litty et al., 2008), and System Virginity Verifier (Rutkowska, 2005), and

Conclusion

We present a solution to kernel queue hooking (KQH) attacks that manipulate K-Queues to achieve stealthy and continual malicious function execution. Such attacks are actively used by advanced malware such as the Rustock spam bot, but they remain invisible to state-of-the-art kernel integrity monitors. We propose the PLCP (Precise Lookahead Checking of function Pointers) approach that checks the legitimacy of pending K-Queue requests by proactively checking function pointers that may be invoked

Jinpeng Wei received a PhD in Computer Science from Georgia Institute of Technology, Atlanta, GA in 2009. He is currently an assistant professor at the School of Computing and Information Sciences, Florida International University, Miami, FL. His research interests include malware detection, information flow security in distributed systems, could computing security, and file-based race condition vulnerabilities. He is a member of the IEEE and the ACM.

References (39)

  • Abadi M, Budiu M, Erlingsson U, Ligatti J. Control-flow integrity. In: Proceedings of the 12th ACM conference on...
  • Anderson LO. Program analysis and specialization for the C programming language. PhD thesis: University of Copenhagen;...
  • Arbaugh WA, Farber DJ, Smith JM. A secure and reliable bootstrap architecture. In: Proceedings of IEEE symposium on...
  • Baliga A, Ganapathy V, Iftode L. Automatic inference and enforcement of kernel data structure invariants. In:...
  • Barham P, Dragovic B, Fraser K, Hand S, Harris T, Ho A, et al. Xen and the art of virtualization. In: Proceedings of...
  • Becher M, Dornseif M, Klein CN. FireWire: all your memory are belong to us. In: Proceedings of CanSecWest;...
  • M. Bishop et al.

    Checking for race conditions in file accesses

    Computing Systems

    (Spring 1996)
  • F. Boldewin

    Peacomm.C – cracking the nutshell

    (September 2007)
  • Carbone M, Cui W, Lu L, Lee W, Peinado M, Jiang X. Mapping kernel objects to enable systematic integrity checking. In:...
  • Das M. Unification-based pointer analysis with directional assignments. In: Proceedings of the 2000 ACM SIGPLAN...
  • A. Decker et al.

    Pushdo/Cutwail: a study of the Pushdo/Cutwail botnet

    (May 2009)
  • Garfinkel T, Rosenblum M. A virtual machine introspection based architecture for intrusion detection. In: Proceedings...
  • Hind M. Pointer analysis: haven't we solved this problem yet? In: Proceedings of the 2001 ACM SIGPLAN-SIGSOFT workshop...
  • Hund R, Holz T, Freiling FC. Return-oriented rootkits: bypassing kernel code integrity protection mechanisms. In:...
  • Jiang X, Wang X, Xu D. Stealthy malware detection through VMM-based “Out-Of-the-Box” semantic view reconstruction. In:...
  • Kiriansky V, Bruening D, Amarasinghe S. Secure execution via program shepherding. In: Proceedings of the 11th USENIX...
  • L. Kwiatek et al.

    Yet another Rustock analysis

    Virus Bulletin

    (August 2008)
  • Lanzi A, Sharif M, Lee W. K-Tracer: a system for extracting kernel malware behavior. In: Proceedings of the 16th annual...
  • Litty L, Lagar-Cavilla HA, Lie D. Hypervisor support for identifying covertly executing binaries. In: Proceedings of...
  • Cited by (5)

    Jinpeng Wei received a PhD in Computer Science from Georgia Institute of Technology, Atlanta, GA in 2009. He is currently an assistant professor at the School of Computing and Information Sciences, Florida International University, Miami, FL. His research interests include malware detection, information flow security in distributed systems, could computing security, and file-based race condition vulnerabilities. He is a member of the IEEE and the ACM.

    Calton Pu received the PhD degree from the University of Washington in 1986. He is a professor and the John P. Imlay Jr. chair in software at Georgia Institute of Technology, Atlanta, GA. He has published more than 60 journal papers and book chapters, 170 refereed workshop and conference papers in operating systems, transaction processing, systems reliability and security, and Internet data management. He has served on more than 100 program committees for more than 50 international conferences and workshops. He is a member of the ACM, a senior member of the IEEE, and a fellow of the AAAS.

    View full text