Skip to main content
Log in

Data warehousing and data mining techniques for intrusion detection systems

Distributed and Parallel Databases Aims and scope Submit manuscript

Abstract

This paper describes data mining and data warehousing techniques that can improve the performance and usability of Intrusion Detection Systems (IDS). Current IDS do not provide support for historical data analysis and data summarization. This paper presents techniques to model network traffic and alerts using a multi-dimensional data model and star schemas. This data model was used to perform network security analysis and detect denial of service attacks. Our data model can also be used to handle heterogeneous data sources (e.g. firewall logs, system calls, net-flow data) and enable up to two orders of magnitude faster query response times for analysts as compared to the current state of the art. We have used our techniques to implement a prototype system that is being successfully used at Army Research Labs. Our system has helped the security analyst in detecting intrusions and in historical data analysis for generating reports on trend analysis.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

References

  1. S. Chaudhuri and U. Dayal, “An overview of data warehousing and OLAP technology,” SIGMOD Record, March 1997.

  2. W.H. Inmon, Building the Data Warehouse, 2nd edn. John Wiley, 1996.

  3. R. Kimball, The Data Warehouse, John Wiley, Toolkit, 1996.

    Google Scholar 

  4. J. Han and M. Kamber, Data Mining Concepts and Techniques, Morgan Kaufmann, August 2000.

  5. H.G. Molina, J.D. Ullman, and J. Widom, Database Systems the Complete Book, Prentice Hall, 2002.

  6. R. Agrawal and R. Srikant, “Fast algorithms for mining association rules,” in Proc. 1994 International Conference on Very Large Databases (VLDB ’94), Santiago, Chile, 1994, pp. 487–499.

  7. A. Singhal and S. Jajodia, “Data mining for intrusion detection,” Published as a chapter in Data Mining Handbook, Kluwer, December 2004.

  8. H. Debar, M. Beker, and D. Siboni, “A neural network component for an intrusion detection system,” in Proceedings of the IEEE Computer Society Symposium on Research in Security and Privacy, Oakland, CA, May 1992, pp. 240–250.

  9. W. Lee, S.J. Stolfo, and K.W. Kwok, “Mining audit data to build intrusion detection models,” in Proc. Fourth International Conference on Knowledge Discovery and Data Mining, NewYork, 1998.

  10. W. Lee and S.J. Stolfo, “Data Mining approaches for intrusion detection,” in Proc. Seventh USENIX Security Symposium, San Antonio, TX, 1998.

  11. D. Bararba, N. Wu, and S. Jajodia, “Detecting novel network intrusions using bayes estimators,” in Proc. First SIAM Conference on Data Mining, Chicago, IL, April 2001.

  12. D. Barbara, J. Couto, S. Jajodia, and N. Wu, “Adam: Detecting intrusions by data mining,” in Proc. 2nd Annual IEEE Information Assurance Workshop, West Point, NY, June 2001.

  13. L. Ertoz, E. Eilertson, A. Lazarevic, P. Tan, P. Dokes, V. Kumar, and J. Srivastava, “Detection of novel attacks using data mining,” Proc. IEEE Workshop on Data Mining and Computer Security, November 2003.

  14. V. Kumar, A. Lazarevic, L. Ertoz, A. Ozgur, and J. Srivastava, “A Comparative study of anomaly detection schemes in network intrusion detection,” in Proc.Third SIAM International Conference on Data Mining, San Francisco, May 2003.

  15. L. Portnoy, E. Eskin, and S.J. Stolfo, “Intrusion detection with unlabeled data using clustering,” in Proceedings of ACM Workshop on Data Mining Applied to Security, 2001.

  16. T. Abraham, “IDDM: Intrusion detection using data mining techniques,” Technical Report DSTO-GD-0286, DSTO Electronics and Surveillance Research Laboratory, 2001.

  17. A. Valdes and K. Skinner, “Adaptive, model based monitoring for cyber attack detection,” in Recent Advances on Intrusion Detection, France, Springer Verlag, 2000, pp 80–93.

  18. P. Ning, Y. Cui, and D.S. Reeves, “Constructing attack scenarios through correlation of intrusion alerts,” in Proc ACM Computer and Communications Security Conf., 2002.

  19. P. Ning and D. Xu, “Learning attack strategies from intrusion alerts,” in Proc ACM Computer and Communications Security Conf., 2003.

  20. F. Cuppens and A. Miege, “Alert correlation in a cooperative intrusion detection framework,” in Proc. IEEE Symposium on Security and Privacy, May 2002.

  21. A. Singhal, “ANSWER: Network monitoring using object oriented rules, in Proceedings of the Tenth Conference on Innovative Application of Artificial Intelligence, G. Weiss and J. Ros, Madison, Wisconsin, July 1998

  22. A. Singhal, “Data modeling and data warehousing techniques to improve IDS,” Work in Progress Paper presented at the 21st Annual Computer Security Applications Conference (ACSAC), December 2005.

  23. A. Singhal, “Design of data warehouse for network/Web Services,” in Proceedings of Conference on Information and Knowledge Management (CIKM), November 2004.

  24. DARPA, DARPA Intrusion Detection Evaluation, 1998, http://ideval.ll.mit.edu/1998_index.html.

  25. SNORT, SNORT Intrusion Detection System, http://www.snort.org

  26. RealSecure IDS, http://www.iss.net

  27. KDD Cup 1999, http://www.kdd.ics.uci.edu/databases/kddcup99/task.html

  28. GraphViz, Graph layout and drawing software, http://www.research.att.com/sw/tools/graphviz

  29. X. Qin and W. Lee, “Statistical causality analysis of INFOSEC alert data,” in Proceedings of 6th International Symposium on Recent Advances in Intrusion Detection (RAID 2003), September 2003.

  30. X. Qin and W. Lee, “Discovering novel attack strategies from INFOSEC alerts,” in Proceedings of the 9th European Symposium on Research in Computer Security (ESORICS 2004), September 2004.

  31. S, Noel, E. Robertson, and S. Jajodia, “Correlating intrusion events and building attack scenarios through attack graph distances,” in Proceedings of the 20th Annual Computer Security Applications Conference, Tucson, Arizona, December 2004.

  32. J. Koziol, Intrusion Detection with Snort, Published by Sam Publishing, 2003.

Download references

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Anoop Singhal.

Additional information

Recommended by: Ashfaq Khokhar

Rights and permissions

Reprints and permissions

About this article

Cite this article

Singhal, A., Jajodia, S. Data warehousing and data mining techniques for intrusion detection systems. Distrib Parallel Databases 20, 149–166 (2006). https://doi.org/10.1007/s10619-006-9496-5

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10619-006-9496-5

Keywords

Navigation