Using a bioinformatics approach to generate accurate exploit-based signatures for polymorphic worms

Abstract

In this paper, we propose Simplified Regular Expression (SRE) signature, which uses multiple sequence alignment techniques, drawn from bioinformatics, in a novel approach to generating more accurate exploit-based signatures. We also provide formal definitions of what is “a more specific” and what is “the most specific” signature for a polymorphic worm and show that the most specific exploit-based signature generation is NP-hard. The approach involves three steps: multiple sequence alignment to reward consecutive substring extractions, noise elimination to remove noise effects, and signature transformation to make the SRE signature compatible with current IDSs. Experiments on a range of polymorphic worms and real-world polymorphic shellcodes show that our bioinformatics approach is noise-tolerant and as that because it extracts more polymorphic worm characters, like one-byte invariants and distance restrictions between invariant bytes, the signatures it generates are more accurate and precise than those generated by some other exploit-based signature generation schemes.

Introduction

Internet worms propagate over the Internet through infections in which a worm sends a payload (such as a shellcode or a copy of itself) to the target host by exploiting operation system or network service vulnerabilities (Tang et al., 2009). A polymorphic worm is a worm that changes its appearance at each infection, making detection and prevention much harder. One of the most popular and effective ways to detect worms is signature-based detection (also called content-based filtering). The generated signature can be either exploit-based or vulnerability-based: an exploit-based signature describes the characteristics of one or a number of exploits; a vulnerability-based signature describes properties of one vulnerability and can detect all possible exploits utilizing this vulnerability.

Although vulnerability-based signature could be more effective, so far both exploit-based and vulnerability-based signatures have equal importance for worm detection in current IDSs (Vulnerability, 2005) for the following reasons. First, the real vulnerability-based signature¹ can be generated only if a vulnerability is disclosed (Brumley et al., 2006); however an exploit-based signature can be fast and timely generated to detect zero-day exploits of an undisclosed vulnerability. Second, most IDS/anti-virus vendors generate both types of signatures. Through them, we not only know when we were attacked, but how we were attacked by investigating whether an exploit is new or a variant of a well-known exploit. Third, some vulnerability-based signatures cannot be adopted in current IDSs since they require that IDSs are able to perform profound protocol analysis (Crandall et al., 2005). However, this obviously overestimates the capability of current IDSs. On the contrary, it has no such requirement to generate exploit-based signatures and hence exploit-based signature generation systems are more compatible to current IDSs. Compared with manual signature generation, the recently studied automatic signature generation approaches can generate more accurate signatures for worms much faster, especially for polymorphic worms. Thus, this paper will only focus on automatic exploit-based signature generation for polymorphic worms.

We find that currently available exploit-based signature generation approaches may fail to create accurate signatures from collections of exploit samples of polymorphic worms. To understand why, consider a sample of a polymorphic worm (an infection flow) as a string sequence consisting of invariant and wildcard bytes. Invariant bytes have fixed values and are present in every worm sample. In contrast, wildcard bytes change their values in each sample. Fig. 1 shows an example of the polymorphic Code Red II worm, which contains a sequence of seven invariant parts: “GET”, “.ida?”, “XX”, “%u”, “%u780”, “=”, and “HTTP/1.0 r n”. Typically, we would try to extract the invariant parts of polymorphic worms as their signatures, since invariant bytes in a worm flow are composed of a number of invariant parts which are crucial to the exploitation of a vulnerable server (Crandall et al., 2005, Newsome et al., 2005). But this raises two difficulties. First, some invariant parts in polymorphic worms cannot be extracted. Earlier approaches (Kreibich and Crowcroft, 2003, Kim and Karp, 2004, Singh et al., 2004) were able to generate only a single invariant part. More up-to-date approaches (Polygraph (Newsome et al., 2005) and Hamsa (Li et al., 2006)) can extract most invariant parts except for one-byte invariant parts (such as “=” in the Code Red II worm). Second, no approach takes into account all distance restrictions between invariant parts (like ‘‘‘%u780’ is 4 bytes after ‘%u”’ in the Code Red II worm). Yet we might surmise that whether one-byte invariant (signature) parts and distance restrictions are valuable in worm detection. Consider Table 1, which shows part of rules (a rule represents a signature) of two well-known IDSs, Snort and Bro. It can be seen that 40.6% of rules in Snort's exploit.rules file consist of multiple invariant parts, 38.7% contain distance restrictions, and 22% contain one-byte signature parts. These figures suggest that distance restrictions could play a role in worm infections and that signatures generated by previous NSG systems may not be accurate enough to identify worms, resulting in a high rate of false positives.

In this paper, we model the problem of generating accurate exploit-based signature given some zero-day exploits of a new polymorphic worm. We propose a signature type – Simplified Regular Expression (SRE) signature. SRE can be easily transformed to rules in current IDSs to accurately capture polymorphic worms. Based on SRE, we provide formal definitions of what is “a more specific” and what is “the most specific” signature of a polymorphic worm such that we can compare the accuracy of two signatures. We show that it is an NP-hard problem to generate the most specific SRE signature of a polymorphic worm. Our approach proposed in the paper is a network-based and exploit-based scheme to generate SRE signature for a single polymorphic worm. It is a bioinformatics approach, inspired by the multiple sequence alignment techniques (used to identify motifs and domains preserved by evolution) in bioinformatics. The approach consists of three steps: multiple sequence alignment, noise elimination and signature transformation. The multiple sequence alignment step is based on the T-coffee algorithm (Notredame et al., 2000) and our newly proposed CSR (Consecutive Substrings Rewarded) algorithm. CSR is a pairwise alignment algorithm that captures contiguous invariant parts in worm samples. It is an extension of the Needleman–Wunsch algorithm and is based on rewarding consecutive matches. We minimize the impact of noise samples by using a noise elimination algorithm that is relevant to a noise tolerance rate θ. A signature transformation step is used to derive accurate SRE signatures that can be conveniently used in current IDSs. We analyze the complexity of the approach and the impact of its noise tolerance rate θ.

Experiments on a range of polymorphic worms and real-world polymorphic shellcodes show that the SRE signatures generated by our approach are more accurate than those of previous methods, extracting more polymorphic worm characters, including the one-byte invariant and distance restriction, and distinguishes between benign traffic and worm traffic. Our approach is resilient to a small portion of noise samples. Our experiments show that our approach (combining the CSR and T-coffee algorithms) is more accurate than other sequence alignment algorithms with regard to accuracy.

The rest of the paper is organized as follows. We first review the related work in Section 2. In Section 3, we propose the SRE signature and raise the accurate exploit-based signature generation problem. In Section 4, we present a bioinformatics approach that generates accurate SRE signatures given some exploit samples from a polymorphic worm. Section 5 evaluates our approach with and without noise sample interference. We discuss the complexity, the scope and limitation of our approach in Section 6. Finally, Section 7 offers our conclusion.