Skip to main content
SpringerLink
Log in

Detecting zero-day attacks using context-aware anomaly detection at the application-layer

  • Regular Contribution
  • Published:
International Journal of Information Security Aims and scope Submit manuscript

Abstract

Anomaly detection allows for the identification of unknown and novel attacks in network traffic. However, current approaches for anomaly detection of network packet payloads are limited to the analysis of plain byte sequences. Experiments have shown that application-layer attacks become difficult to detect in the presence of attack obfuscation using payload customization. The ability to incorporate syntactic context into anomaly detection provides valuable information and increases detection accuracy. In this contribution, we address the issue of incorporating protocol context into payload-based anomaly detection. We present a new data representation, called \({c}_n\)-grams, that allows to integrate syntactic and sequential features of payloads in an unified feature space and provides the basis for context-aware detection of network intrusions. We conduct experiments on both text-based and binary application-layer protocols which demonstrate superior accuracy on the detection of various types of attacks over regular anomaly detection methods. Furthermore, we show how \({c}_n\)-grams can be used to interpret detected anomalies and thus, provide explainable decisions in practice.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9

Similar content being viewed by others

Notes

  1. http://www.metasploit.com.

References

  1. Borisov, N., Brumley, D.J., Wang, H., Dunagan, J., Joshi, P., Guo, C.: Generic application-level protocol analyzer and its language. In: Proceedings of Network and Distributed System Security Symposium (NDSS) (2007)

  2. Cretu, G., Stavrou, A., Locasto, M., Stolfo, S.J., Keromytis, A.D.: Casting out demons: sanitizing training data for anomaly sensors. In: ieeesp (2008)

  3. Cui, W., Kannan, J., Wang. H.J.: Discoverer: automatic protocol reverse engineering from network traces. In: SS’07: Proceedings of 16th USENIX Security Symposium on USENIX Security Symposium, pp. 1–14 (2007)

  4. Detristan, T., Ulenspiegel, T., Malcom, Y., Underduk, M.: Polymorphic shellcode engine using spectrum analysis. Phrack. 11(61) (2003)

  5. Düssel, P., Gehl, C., Laskov, P., Rieck, K.: Incorporation of application layer protocol syntax into anomaly detection. In: ICISS, pp. 188–202 (2008)

  6. Folga, P., Sharif, M., Perdisci, R., Kolesnikov, O., Lee, W.: Polymorphic blending attacks. In: Proceedings of USENIX Security Symposium, pp. 241–256 (2006)

  7. Forrest, S., Hofmeyr, S.A., Somayaji, A., Longstaff, T.A.: A sense of self for unix processes. In: Proceedings of IEEE Symposium on Security and Privacy, pp. 120–128. Oakland (1996)

  8. Gao, D., Reiter, M.K., Song, D.: Behavioral distance measurement using hidden markov models. In: Recent Adances in Intrusion Detection (RAID), pp. 19–40 (2006)

  9. Ingham, K.L., Somayaji, A., Burge, J., Forrest, S.: Learning DFA representations of HTTP for protecting web applications. Comput. Netw. 51(5), 1239–1255 (2007)

    Article  MATH  Google Scholar 

  10. Kloft, M., Laskov, P.: Security analysis of online centroid anomaly detection. Technical report UCB/EECS-2010-22. EECS Department, University of California, Berkeley (2010)

  11. Kolesnikov, O., Dagon, D., Lee, W.: Advanced polymorphic worms: evading IDS by blending with normal traffic. In: Proceedings of USENIX Security Symposium (2004)

  12. Kruegel, C., Vigna, G.: Anomaly detection of web-based attacks. In: Proceedings of 10th ACM Conference on Computer and Communications Security, pp. 251–261 (2003)

  13. Kruegel, C., Toth, T., Kirda, E.: Service specific anomaly detection for network intrusion detection. In: Proceedings of ACM Symposium on Applied, Computing, pp. 201–208 (2002)

  14. Kruegel, C., Vigna, G., Robertson, W.: A multi-model approach to the detection of web-based attacks. Comput. Netw. 48(5), 717–738 (2005)

    Article  Google Scholar 

  15. Lee, W., Stolfo, S.J.: A framework for constructing features and models for intrusion detection systems. ACM Trans. Inf. Syst. Secur. 3, 227–261 (2000)

    Article  Google Scholar 

  16. Lodhi, H., Saunders, C., Shawe-Taylor, J., Cristianini, N., Watkins, C.: Text classification using string kernels. J. Mach. Learn. Res. 2, 419–444 (2002)

    MATH  Google Scholar 

  17. Mahoney, M.V., Chan, P.K.: PHAD: packet header anomaly detection for identifying hostile network traffic. Technical Report CS-2001-2, Florida Institute of Technology (2001)

  18. Mahoney, M.V., Chan, P.K.: Learning nonstationary models of normal network traffic for detecting novel attacks. In: Proceedings of ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD), pp. 376–385 (2002)

  19. Mahoney, M.V., Chan, P.K.: An analysis of the 1999 DARPA/Lincoln Laboratory evaluation data for network anomaly detection. In: Giovanni V, Kruegel, Christopher, Erland J (eds) Recent Advances in Intrusion Detection. Springer, Berlin, Heidelberg, pp. 220–237 (2003)

  20. Müller, K.-R., Mika, S., Rätsch, G., Tsuda, K., Schölkopf, B.: An introduction to kernel-based learning algorithms. IEEE Neural Netw. 12(2), 181–201 (2001)

    Article  Google Scholar 

  21. Pang, R., Paxson, V., Sommer, R., Peterson, L.L.: binpac: A yacc for writing application protocol parsers. In: Proceedings of ACM Internet Measurement Conference, pp. 289–300 (2006)

  22. Paxson, V.: Bro: a system for detecting network intruders in real-time. In: Proceedings of USENIX Security Symposium, pp. 31–51 (1998)

  23. Paxson, V.: The bro 0.8 user manual. Lawrence Berkeley National Laboratory and ICSI Center for Internet Research (2004)

  24. Perdisci, R., Ariu, D., Fogla, P., Giacinto, G., Lee, W.: McPAD: a multiple classifier system for accurate payload-based anomaly detection. Comput. Netw. 53(6), 864–881 (2009)

  25. Rieck, K., Laskov, P.: Detecting unknown network attacks using language models. In: Detection of Intrusions and Malware, and Vulnerability Assessment, Proceedings of 3rd DIMVA Conference, LNCS, pp. 74–90 (2006)

  26. Rieck, K., Laskov, P.: Language models for detection of unknown attacks in network traffic. J. Comput. Virol. 2(4), 243–256 (2007)

  27. Rieck, K., Laskov, P.: Visualization and explanation of payload-based anomaly detection. In: Proceedings of European Conference on Computer Network Defense (EC2ND) (2009)

  28. Roesch, M.: Snort: lightweight intrusion detection for networks. In: Proceedings of USENIX Large Installation System Administration Conference LISA, pp. 229–238 (1999)

  29. Shawe-Taylor, J., Cristianini, N.: Kernel Methods for Pattern Analysis. Cambridge University Press, Cambridge (2004)

    Book  MATH  Google Scholar 

  30. Song, Y., Keromytis, A.D., Stolfo, S.J.: Spectrogram: a mixture-of-markov-chains model for anomaly detection in web traffic. In: Proceedings of Network and Distributed System Security Symposium (NDSS) (2009)

  31. Tax, D., Duin, R.: Data domain description by support vectors. In: Verleysen, M. (ed.) Proceedings ESANN, pp. 251–256. D. Facto Press, Brussels (1999)

  32. Vishwanathan, S.V.N., Smola, A.J.: Fast kernels for string and tree matching. In: Tsuda, K., Schölkopf, B., Vert, J.F. (eds.) Kernels and Bioinformatics, pp. 113–130. MIT Press, Cambridge (2004)

    Google Scholar 

  33. Wang, K., Stolfo, S.J.: Anomalous payload-based network intrusion detection. In: Recent Adances in Intrusion Detection (RAID), pp. 203–222 (2004)

  34. Wang, K., Parekh, J.J., Stolfo, S.J.: Anagram: a content anomaly detector resistant to mimicry attack. In: Recent Adances in Intrusion Detection (RAID), pp. 226–248 (2006)

  35. Wireshark. Wireshark: network protocol analyzer. http://www.wireshark.org (2010)

  36. Wondracek, G., Milani, C.P., Kruegel, C., Kirda, E.: Automatic network protocol analysis. In: 15th Symposium on Network and Distributed System Security (NDSS) (2008)

Download references

Author information

Authors and Affiliations

  1. University of Bonn, Institute of Computer Science 4, Friedrich-Ebert-Allee 144, 53113, Bonn, Germany

    Patrick Duessel & Michael Meier

  2. Trifense GmbH - Intelligent Network Defense, Germendorfer Strasse 79, 16727, Velten, Germany

    Christian Gehl

  3. Infineon Technologies AG, Am Campeon 1-12, 86579, Neubiberg, Germany

    Ulrich Flegel

  4. CUNY John Jay College of Criminal Justice, Mathematics and Computer Science Department, 524 West 59th St, New York, NY, 10019, USA

    Sven Dietrich

Authors
  1. Patrick Duessel
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Christian Gehl
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Ulrich Flegel
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Sven Dietrich
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Michael Meier
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Patrick Duessel.

Appendix: Experiment results details

Appendix: Experiment results details

See Tables 6, 7 and 8.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Duessel, P., Gehl, C., Flegel, U. et al. Detecting zero-day attacks using context-aware anomaly detection at the application-layer. Int. J. Inf. Secur. 16, 475–490 (2017). https://doi.org/10.1007/s10207-016-0344-y

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10207-016-0344-y

Keywords

Search

Navigation