Abstract
Understanding flow changes in network traffic has great importance in designing and building robust networking infrastructure. Recent efforts from industry and academia have led to the development of monitoring tools that are capable of collecting real-time flow data, predicting future traffic patterns, and mirroring packet headers. These monitoring tools, however, require offline analysis of the data to understand the big versus small flows and recognize congestion hot spots in the network, which is still an unfilled gap in research. In this study, we proposed an innovative unsupervised clustering approach, DynamicDeepFlow, for network traffic pattern clustering. The DynamicDeepFlow can recognize unseen network traffic patterns based on the analysis of the rapid flow changes from the historical data. The proposed method consists of a deep learning model, variational autoencoder, and a shallow learning model, k-means++. The variational autoencoder is used to compress and extract the most useful features from the flow inputs. The compressed and extracted features then serve as input-output pairs to k-means++. The k-means++ explores the structure hidden in these features and then uses them to cluster the network traffic patterns. To the best of our knowledge, this is one of the first attempts to apply a real-time network clustering approach to monitor network operations. The real-world network flow data from Energy Sciences Network (a network serving the U.S. Department of Energy to support U.S. scientific research) was utilized to verify the performance of the proposed approach in network traffic pattern clustering. The verification results show that the proposed method is able to distinguish anomalous network traffic patterns from normal patterns, and thereby trigger an anomaly flag.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Akyildiz, I.F., Lee, A., Wang, P., Luo, M., Chou, W.: Research challenges for traffic engineering in software defined networks. IEEE Netw. 30(3), 52–58 (2016)
Fukuda, K., Takayasu, H., Takayasu, M.: Spatial and temporal behavior of congestion in internet traffic. Fractals 7(01), 23–31 (1999)
Mallick, T., Kiran, M., Mohammed, B., Balaprakash, P.: Dynamic graph neural network for traffic forecasting in wide area networks. arXiv preprint arXiv:2008.12767 (2020)
Joshi, M., Hadi, T.H.: A review of network traffic analysis and prediction techniques. arXiv preprint arXiv:1507.05722 (2015)
Welzl, M.: Network Congestion Control: Managing Internet Traffic. Wiley, Hoboken (2005)
Kettimuthu, R., Vardoyan, G., Agrawal, G., Sadayappan, P.: Modeling and optimizing large-scale wide-area data transfers. In: 14th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing, pp. 196–205. IEEE (2014)
Uceda, V., RodrÃguez, M., Ramos, J., GarcÃa-Dorado, J.L., Aracil, J.: Selective capping of packet payloads for network analysis and management. In: Steiner, M., Barlet-Ros, P., Bonaventure, O. (eds.) TMA 2015. LNCS, vol. 9053, pp. 3–16. Springer, Cham (2015). https://doi.org/10.1007/978-3-319-17172-2_1
Singh, H.: Performance analysis of unsupervised machine learning techniques for network traffic classification. In: 2015 Fifth International Conference on Advanced Computing & Communication Technologies, pp. 401–404. IEEE (2015)
Soule, A., Salamatia, K., Taft, N., Emilion, R., Papagiannaki, K.: Flow classification by histograms: or how to go on safari in the internet. In: Proceedings of the Joint International Conference on Measurement and Modeling of Computer Systems, pp. 49–60 (2004)
Auld, T., Moore, A.W., Gull, S.F.: Bayesian neural networks for internet traffic classification. IEEE Trans. Neural Netw. 18(1), 223–239 (2007)
Roughan, M., Sen, S., Spatscheck, O., Duffield, N.: Class-of-service mapping for QoS: a statistical signature-based approach to IP traffic classification. In: Proceedings of the 4th ACM SIGCOMM conference on Internet measurement, pp. 135–148 (2004)
Ahmed, M., Mahmood, A.N.: Novel approach for network traffic pattern analysis using clustering-based collective anomaly detection. Ann. Data Sci. 2(1), 111–130 (2015)
Benesty, J., Chen, J., Huang, Y., Cohen, I.: Pearson correlation coefficient. In: Cohen, I., Huang, Y., Chen, J., Benesty J. (eds.) Noise Reduction in Speech Processing. STSP, vol. 2, pp. 1–4. Springer, Heidelberg (2009). https://doi.org/10.1007/978-3-642-00296-0_5
Shen, S., Sadoughi, M., Li, M., Wang, Z., Hu, C.: Deep convolutional neural networks with ensemble learning and transfer learning for capacity estimation of lithium-ion batteries. Appl. Energy 260, 114296 (2020)
Shen, S., Sadoughi, M., Chen, X., Hong, M., Hu, C.: A deep learning method for online capacity estimation of lithium-ion batteries. J. Energy Storage 25, 100817 (2019)
Li, T., Pasternack, G.B.: Revealing the diversity of hydropeaking patterns by time-series data mining. AGU Fall Meet. Abstr. 2020, H049–03 (2020)
Li, T., Pasternack, G.B.: Revealing the diversity of hydropeaking flow regimes. J. Hydrol. 598, 126392 (2021)
Shen, S., Sadoughi, M., Hu, C.: Online estimation of lithium-ion battery capacity using transfer learning. In: IEEE Transportation Electrification Conference and Expo (ITEC), pp. 1–4. IEEE (2019)
Shen, S., Sadoughi, M., Chen, X., Hong, M., Hu, C.: Online estimation of lithium-ion battery capacity using deep convolutional neural networks. In: International Design Engineering Technical Conferences and Computers and Information in Engineering Conference, vol. 51753, p. V02AT03A058. American Society of Mechanical Engineers (2018)
Shen, S., et al.: A physics-informed deep learning approach for bearing fault detection. Eng. Appl. Artif. Intell. 103, 104295 (2021)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2022 Springer Nature Switzerland AG
About this paper
Cite this paper
Shen, S., Kiran, M., Mohammed, B. (2022). DynamicDeepFlow: An Approach for Identifying Changes in Network Traffic Flow Using Unsupervised Clustering. In: Renault, É., Boumerdassi, S., Mühlethaler, P. (eds) Machine Learning for Networking. MLN 2021. Lecture Notes in Computer Science, vol 13175. Springer, Cham. https://doi.org/10.1007/978-3-030-98978-1_7
Download citation
DOI: https://doi.org/10.1007/978-3-030-98978-1_7
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-98977-4
Online ISBN: 978-3-030-98978-1
eBook Packages: Computer ScienceComputer Science (R0)