Abstract
Anomaly detection in networks is an important cybersecurity threat detection capability. Anomalies in networks are often not localized to a single point, but are spread over a range of nodes. In this case of distributed anomalies, the anomalies are typically too subtle to detect at an individual-node level, and so require anomaly detection over groups of nodes. But it is usually not known a priori on which subset of nodes to focus, and it is infeasible to check all 2N subsets of nodes in a network. This renders distributed anomaly detection extremely challenging. An emerging strategy for detecting such anomalies is to apply a detection technique to a hierarchy of clusters of nodes in the network. However, developing such a hierarchy is challenging in large, decentralized networks with no central controller. In this work, we present Multilevel Autonomous Clustering (MAC), a novel local algorithm for self-organized, hierarchical clustering in distributed networks. MAC enables individual devices in a distributed system to determine their cluster membership at multiple levels using only local information, without centralized computation or information about the entire network. The result is an approach to hierarchical graph clustering that is both practical to use in large, real-world systems, as well as effective for distributed anomaly detection. The algorithm is evaluated on both synthetic and real-world networks. Its effectiveness for anomaly detection is demonstrated on various test problems.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Chatzigiannakis, V., Papavassiliou, S., Grammatikou, M., Maglaris, B.: Hierarchical anomaly detection in distributed large-scale sensor networks. In: 11th IEEE Symposium on Computers and Communications, pp. 761–767 (2006)
Palpanas, T., Papadopoulos, D., Kalogeraki, V., Gunopulos, D.: Distributed deviation detection in sensor networks. ACM SIGMOD Rec. 32(4), 77–82 (2003)
Rajasegarar, S., Leckie, C., Palaniswami, M., Bezdek, J.C.: Distributed anomaly detection in wireless sensor networks. In: 10th IEEE Singapore International Conference on Communication Systems, pp. 1–5 (2006)
Tanachaiwiwat, S., Helmy, A.: Correlation analysis for alleviating effects of inserted data in wireless sensor networks. In: The Second Annual International Conference on Mobile and Ubiquitous Systems: Networking and Services, pp. 97–108 (2005)
De Paola, A., Gaglio, S., Re, G.L., Milazzo, F., Ortolani, M.: Adaptive distributed outlier detection for WSNs. IEEE Trans. Cybern. 45(5), 902–913 (2015)
Robinson, J., Lonergan, M., Singh, L., Candido, A., Sayal, M.: SHARD: a framework for sequential, hierarchical anomaly ranking and detection. In: Tan, P.N., Chawla, S., Ho, C.K., Bailey, J. (eds.) PAKDD. LNCS, vol. 7302, pp. 243–255. Springer, Heidelberg (2012)
Rassam, M., Zainal, A., Maarof, M.: An efficient distributed anomaly detection model for wireless sensor networks. AASRI Procedia 5, 9–14 (2013)
Bridges, R.A., Collins, J.P., Ferragut, E.M., Laska, J.A., Sullivan, B.D.: Multi-level anomaly detection on time-varying graph data. In: ASONAM 2015, pp. 579–583 (2015)
Bagrow, J.P., Bollt, E.M.: Local method for detecting communities. Phys. Rev. E 72(4), 046108 (2005)
Clauset, A.: Finding local community structure in networks. Phys. Rev. E 72(2), 026132 (2005)
Schaeffer, S.E.: Stochastic local clustering for massive graphs. In: Ho, T.B., Cheung, D., Liu, H. (eds.) PAKDD. LNCS, vol. 3518, pp. 354–360. Springer, Heidelberg (2005)
Schaeffer, S.E.: Graph clustering. Comput. Sci. Rev. 1, 27–64 (2007)
Schaeffer, S.E., Marinoni, S.: Dynamic local clustering for hierarchical Ad Hoc networks. In: 3rd Annual IEEE Communications Society on Sensor and Ad Hoc Communications and Networks, vol. 2, pp. 667–672 (2006)
Blondel, V.D., Guillaume, J.-L., Lambiotte, R., Lefebvre, E.: Fast unfolding of communities in large networks. J. Stat. Mech.: Theory Exp. 2008(10), P10008 (2008)
Zimmerman, R.D., Murillo-Sanchez, C.E., Thomas, R.J.: MATPOWER: steady-state operations, planning and analysis tools for power systems research and education. IEEE Trans. Power Syst. 26(1), 12–19 (2011)
Pedregosa, F., et al.: Scikit-learn: machine learning in Python. J. Mach. Learn. Res. 12, 2825–2830 (2011)
Clauset, A., Newman, M.E.J., Moore, C.: Finding community structure in very large networks. Phys. Rev. E 70(6), 066111 (2004)
Leskovec, J., Krevl, A.: SNAP datasets: stanford large network dataset collection (2014). http://snap.stanford.edu/data
Hong, J., Liu, C., Govindarasu, M.: Integrated anomaly detection for cyber security of the substations. IEEE Trans. Smart Grid 5(4), 1643–1653 (2014)
Bollobás, B.: 6. The evolution of random graphs – the giant component. Random Graphs, Cambridge Studies in Advanced Mathematics, vol. 73, 2nd edn., pp. 130–159. Cambridge University Press, Cambridge (2001)
Acknowledgements
Dr. Alyson Fox, Dr. Eisha Nathan, and Dr. Tim La Fond, of Lawrence Livermore National Laboratory (LLNL), provided significant feedback and suggestions for improving the MAC algorithm. Professor Marija Ilic (MIT LIDS, MIT Lincoln Laboratory) provided valuable support and guidance. Within LLNL, Global Security’s E Program, Critical Infrastructure Cybersecurity Group and Engineering’s CED, Computational Electromagnetics Group both provided programmatic support. This work was performed under the auspices of the U.S. Department of Energy by Lawrence Livermore National Laboratory under Contract DE-AC52-07NA27344.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2020 Springer Nature Switzerland AG
About this paper
Cite this paper
Partha, M.A., Ponce, C.V. (2020). MAC: Multilevel Autonomous Clustering for Topologically Distributed Anomaly Detection. In: Cherifi, H., Gaito, S., Mendes, J., Moro, E., Rocha, L. (eds) Complex Networks and Their Applications VIII. COMPLEX NETWORKS 2019. Studies in Computational Intelligence, vol 881. Springer, Cham. https://doi.org/10.1007/978-3-030-36687-2_62
Download citation
DOI: https://doi.org/10.1007/978-3-030-36687-2_62
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-36686-5
Online ISBN: 978-3-030-36687-2
eBook Packages: Intelligent Technologies and RoboticsIntelligent Technologies and Robotics (R0)