A new data normalization method for unsupervised anomaly intrusion detection

Abstract

Unsupervised anomaly detection can detect attacks without the need for clean or labeled training data. This paper studies the application of clustering to unsupervised anomaly detection (ACUAD). Data records are mapped to a feature space. Anomalies are detected by determining which points lie in the sparse regions of the feature space. A critical element for this method to be effective is the definition of the distance function between data records. We propose a unified normalization distance framework for records with numeric and nominal features mixed data. A heuristic method that computes the distance for nominal features is proposed, taking advantage of an important characteristic of nominal features—their probability distribution. Then, robust methods are proposed for mapping numeric features and computing their distance, these being able to tolerate the impact of the value difference in scale and diversification among features, and outliers introduced by intrusions. Empirical experiments with the KDD 1999 dataset showed that ACUAD can detect intrusions with relatively low false alarm rates compared with other approaches.