Abstract
As cyber attacks increase in volume and complexity, it becomes more and more difficult for existing analytical tools to detect previously unseen malware. This paper proposes a cooperative framework to leverage the robustness of big data analytics and the power of ensemble learning techniques to detect the abnormal behavior. In addition to this proposal, we implement a large scale network abnormal traffic behavior detection system performed by the framework. The proposed model detects the abnormal behavior from large scale network traffic data using a combination of a balanced decomposition algorithm and an ensemble SVM. First, the collected dataset is divided into k subsets based on the similarity between patterns using a parallel map reduce k-means algorithm. Then, patterns are randomly selected from each cluster and balanced training sub datasets are formed. Next, the subsets are fed into the mappers to build an SVM model. The construction of the ensemble is achieved in the reduce phase. The proposed structure closely delivers a high accuracy as the number of iterations increases. Experimental results show a promising gain in detection rate and false alarm compared with other existing models.
This is a preview of subscription content, log in via an institution.
Buying options
Tax calculation will be finalised at checkout
Purchases are for personal use only
Learn about institutional subscriptionsReferences
Tsai, C.F., Hsu, Y.F., Lin, C.Y., Lin, W.Y.: Intrusion detection by machine learning: a review. Expert Syst. Appl. 36(10), 11994–12000 (2009)
Janssen, T., Grady, N.: Big data for combating cyber attacks. In: CEUR Workshop Proceedings, Fairfax, vol. 1097, pp. 151–158 (2013)
Cardenas, A.A., Manadhata, P.K., Rajan, S.P.: Big data analytics for security. IEEE Secur. Priv. 11(6), 74–76 (2013)
Scarfone, K.A., Mell, P.M.: Guide to intrusion detection and prevention systems (IDPS), Special Publication (NIST SP), pp. 800–894 (2007)
Jones, A.K., Sielken, R.S.: Computer system intrusion detection: a survey. Technical report, Computer Science Department, University of Virginia (2000)
Zamani, M., Movahedi, M.: Machine learning techniques for intrusion detection (2013). arXiv preprint arXiv:1312-2177
Suthaharan, S.: Big data classification: problems and challenges in network intrusion prediction with machine learning. ACM SIGMETRICS Perform. Eval. Rev. 41(4), 70–73 (2014)
Lee, Y., Lee, Y.: Toward scalable internet traffic measurement and analysis with Hadoop. SIGCOMM Comput. Commun. Rev. 43(1), 5–13 (2012)
Ahn, S.H., Kim, N.U., Chung, T.M.: Big data analysis system concept for detecting unknown attacks. In: 16th International Conference on Advanced Communication Technology (ICACT), South Korea, pp. 16–19 (2014)
Marchal, S., Jiang, X., State, R., Engel, T.: A big data architecture for large scale security monitoring. In: Proceedings of IEEE International Congress Big Data, Anchorage, pp. 56–63 (2010)
Rathore, M.M., Ahmad, A., Paul, A.: Real time intrusion detection system for ultra-high-speed big data environments. J. Supercomput. 72(9), 3489–3510 (2016)
Dos Santos, E.M.: Static and dynamic overproduction and selection of classifier ensembles with genetic algorithms. Ecole de Technologie Superieure, Canada (2008)
Aburomman, A.A., Ibne Reaz, M.B.: A survey of intrusion detection systems based on ensemble and hybrid classifiers. Comput. Secur. 65, 135–152 (2017)
Gaikwad, D.P., Thool, R.C.: Intrusion detection system using bagging ensemble method of machine learning. In: International Conference on Computing Communication Control and Automation, pp. 291–295. IEEE (2015)
Folino, G., Pisani, F.S.: Combining ensemble of classifiers by using genetic programming for cyber security applications. In: Mora, A.M., Squillero, G. (eds.) EvoApplications 2015. LNCS, vol. 9028, pp. 54–66. Springer, Cham (2015). doi:10.1007/978-3-319-16549-3_5
Aburomman, A.A., Ibne Reaz, M.B.: A novel SVM-kNN-PSO ensemble method for intrusion detection system. Appl. Soft Comput. 38, 360–372 (2016)
Vapnik, V.: Statistical Learning Theory. Wiley-Interscience, New York (1998)
Acknowledgments
The National Natural Science Foundation of China under Grant Nos. 61370212, 61402127, 61502118; the Natural Science Foundation of Heilongjiang Province under Grant Nos. F2015029, F2016009; the Fundamental Research Fund for the Central Universities under Grant No. HEUCF100601.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2017 Springer Nature Singapore Pte Ltd.
About this paper
Cite this paper
Marir, N., Wang, H. (2017). A Cooperative Abnormal Behavior Detection Framework Based on Big Data Analytics. In: Zou, B., Li, M., Wang, H., Song, X., Xie, W., Lu, Z. (eds) Data Science. ICPCSEE 2017. Communications in Computer and Information Science, vol 727. Springer, Singapore. https://doi.org/10.1007/978-981-10-6385-5_17
Download citation
DOI: https://doi.org/10.1007/978-981-10-6385-5_17
Published:
Publisher Name: Springer, Singapore
Print ISBN: 978-981-10-6384-8
Online ISBN: 978-981-10-6385-5
eBook Packages: Computer ScienceComputer Science (R0)