TCP Reassembler for Layer7-Aware Network Intrusion Detection/Prevention Systems

Miyuki HANAOKA
Makoto SHIMAMURA
Kenji KONO

Publication
IEICE TRANSACTIONS on Information and Systems   Vol.E90-D    No.12    pp.2019-2032
Publication Date: 2007/12/01
Online ISSN: 1745-1361
DOI: 10.1093/ietisy/e90-d.12.2019
Print ISSN: 0916-8532
Type of Manuscript: PAPER
Category: Dependable Computing
Keyword: 
network security,  network intrusion detection/prevention systems,  layer7-aware NIDS/NIPS,  TCP reassembler,  

Full Text: PDF(741.9KB)>>
Buy this Article



Summary: 
Exploiting layer7 context is an effective approach to improving the accuracy of detecting malicious messages in network intrusion detection/prevention systems (NIDS/NIPSs). Layer7 context enables us to inspect message formats and the message exchanged order. Unfortunately, layer7-aware NIDS/NIPSs pose crucial implementation issues because they require full TCP and IP reassembly without losing 1) complete prevention, 2) performance, 3) application transparency, or 4) transport transparency. Complete prevention means that the NIDS/NIPS should prevent malicious messages from reaching target applications. Application transparency means not requiring any modifications to and/or reconfiguration of server and client applications. Transport transparency is not to disrupt the end-to-end semantics of TCP/IP. To the best of our knowledge, none of the existing approaches meet all of these requirements. We have developed an efficient mechanism for layer7-aware NIDS/NIPSs that does meet the above requirements. Our store-through does this by forwarding each out-of-order or IP-fragmented packet immediately after copying the packet even if it has not been checked yet by an NIDS/NIPS sensor. Although the forwarded packet might turn out to be a part of an attack message, the store-through mechanism can successfully defend against the attack by blocking one of the subsequent packets that contain another part of attack message. Testing of a prototype in Linux kernel 2.4.30 demonstrated that the overhead of our mechanism is negligible compared with that of a simple IP forwarder even with the presence of out-of-order and IP-fragmented packets. In addition, the experimental results suggest that the CPU and memory usage incurred by our store-through is not significant.


open access publishing via