Haojie Ji
Skip Abstract Section
Abstract
Network telemetry, characterized by its efficient push model and high-performance communication protocol (gRPC), offers a new avenue for collecting fine-grained real-time data. Despite its advantages, existing network telemetry systems lack a theoretical basis for setting measurement frequency, struggle to capture informative samples, and face challenges in setting a uniform frequency for multi-metric monitoring. We introduce FineMon, an innovative adaptive network telemetry scheme for precise, fine-grained, multi-metric data monitoring. FineMon leverages a novel Two-sided Frequency Adjustment (TFA) to dynamically adjust the measurement frequency on the Network Management System (NMS) and infrastructure sides. On the NMS side, we provide a theoretical basis for frequency determination, drawing on changes in the rank of multi-metric data to minimize monitoring overhead. On the infrastructure side, we adjust the frequency in real-time to capture significant data fluctuations. We propose a robust Enhanced-Subspace-based Tensor Completion (ESTC) to ensure accurate recovery of fine-grained data, even with noise or outliers. Through extensive experimentation with three real datasets, we demonstrate FineMon's superiority over existing schemes in reduced measurement overhead, enhanced accuracy, and effective capture of intricate temporal features.
- Hiroshi Akima. 1970. A new method of interpolation and smooth curve fitting based on local procedures. Journal of the ACM (JACM) 17, 4 (1970), 589--602.Google Scholar
Digital Library
- Hiroshi Akima. 1974. A method of bivariate interpolation and smooth surface fitting based on local procedures. Commun. ACM 17, 1 (1974), 18--20.Google ScholarDigital Library
- Maria-Florina F Balcan and Hongyang Zhang. 2016. Noise-tolerant life-long matrix completion via adaptive sampling. Advances in Neural Information Processing Systems 29 (2016).Google Scholar
- Thierry Blu, Philippe Thévenaz, and Michael Unser. 2004. Linear interpolation revitalized. IEEE Transactions on Image Processing 13, 5 (2004), 710--719.Google ScholarDigital Library
- CAIDA. 2018. The CAIDA UCSD Anonymized Internet Traces. [Online]. Available: https://www.caida.org/catalog/datasets/passive_dataset.Google Scholar
- Emmanuel Candes and Benjamin Recht. 2012. Exact matrix completion via convex optimization. Commun. ACM 55, 6 (2012), 111--119.Google ScholarDigital Library
- Emmanuel J Candès, Justin Romberg, and Terence Tao. 2006. Robust uncertainty principles: Exact signal reconstruction from highly incomplete frequency information. IEEE Transactions on information theory 52, 2 (2006), 489--509.Google ScholarDigital Library
- Emmanuel J Candès and Terence Tao. 2010. The power of convex relaxation: Near-optimal matrix completion. IEEE Transactions on Information Theory 56, 5 (2010), 2053--2080.Google ScholarDigital Library
- Wei Cao, Yusong Gao, Feifei Li, Sheng Wang, Bingchen Lin, Ke Xu, Xiaojie Feng, Yucong Wang, Zhenjun Liu, and Gejin Zhang. 2020. Timon: A timestamped event database for efficient telemetry data processing and analytics. In Proceedings of the 2020 ACM SIGMOD International Conference on Management of Data. 739--753.Google ScholarDigital Library
- Jeffrey D Case, Mark Fedor, Martin L Schoffstall, and James Davin. 1989. Simple network management protocol (SNMP). Technical Report.Google Scholar
- Kenjiro Cho, Koushirou Mitsuya, and Akira Kato. 2000. Traffic data repository at the {WIDE} project. In 2000 USENIX Annual Technical Conference (USENIX ATC 00).Google Scholar
- Lei Deng, Xiao-Yang Liu, Haifeng Zheng, Xinxin Feng, and Youjia Chen. 2021. Graph spectral regularized tensor completion for traffic data imputation. IEEE Transactions on Intelligent Transportation Systems 23, 8 (2021), 10996--11010.Google ScholarCross Ref
- Lei Deng, Haifeng Zheng, Xiao-Yang Liu, Xinxin Feng, and Zhizhang David Chen. 2020. Network latency estimation with leverage sampling for personal devices: An adaptive tensor completion approach. IEEE/ACM Transactions on Networking 28, 6 (2020), 2797--2808.Google ScholarDigital Library
- Frederick N Fritsch and Ralph E Carlson. 1980. Monotone piecewise cubic interpolation. SIAM J. Numer. Anal. 17, 2 (1980), 238--246.Google ScholarDigital Library
- Hancheng Ge, Kai Zhang, Majid Alfifi, Xia Hu, and James Caverlee. 2018. DisTenC: A distributed algorithm for scalable tensor completion on spark. In 2018 IEEE 34th International Conference on Data Engineering (ICDE). IEEE, 137--148.Google ScholarCross Ref
- Google. 2015. gRPC. https://grpc.io/.Google Scholar
- Arpit Gupta, Rob Harrison, Marco Canini, Nick Feamster, Jennifer Rexford, and Walter Willinger. 2018. Sonata: Query-driven streaming network telemetry. In Proceedings of the 2018 conference of the ACM special interest group on data communication. 357--371.Google ScholarDigital Library
- Richard A Harshman et al. 1970. Foundations of the PARAFAC procedure: Models and conditions for an" explanatory" multimodal factor analysis. (1970).Google Scholar
- Guanghui He and Jennifer C Hou. 2006. On sampling self-similar Internet traffic. Computer Networks 50, 16 (2006), 2919--2936.Google ScholarCross Ref
- Zilong He, Pengfei Chen, Xiaoyun Li, Yongfeng Wang, Guangba Yu, Cailin Chen, Xinrui Li, and Zibin Zheng. 2020. A spatiotemporal deep learning approach for unsupervised anomaly detection in cloud systems. IEEE Transactions on Neural Networks and Learning Systems (2020).Google ScholarCross Ref
- Rick Hofstede, Pavel Celeda, Brian Trammell, Idilio Drago, Ramin Sadre, Anna Sperotto, and Aiko Pras. 2014. Flow monitoring explained: From packet capture to data analysis with netflow and ipfix. IEEE Communications Surveys & Tutorials 16, 4 (2014), 2037--2064.Google ScholarCross Ref
- Bo Huang, Cun Mu, Donald Goldfarb, and John Wright. 2014. Provable low-rank tensor recovery. Optimization-Online 4252, 2 (2014), 455--500.Google Scholar
- Bo Hui, Da Yan, Haiquan Chen, and Wei-Shinn Ku. 2022. Time-sensitive POI Recommendation by Tensor Completion with Side Information. In 2022 IEEE 38th International Conference on Data Engineering (ICDE). IEEE, 205--217.Google ScholarCross Ref
- Liuwei Huo, Dingde Jiang, Xiangnan Zhu, and Huibin Jia. 2019. An Adaptive Measurement Method for Flow Traffic in Software Defined Networking. In Simulation Tools and Techniques: 11th International Conference, SIMUtools 2019, Chengdu, China, July 8--10, 2019, Proceedings 11. Springer, 115--124.Google ScholarCross Ref
- Zoom Video Communications Inc. 2020. Zoom Client Connection Process Whitepaper. https://explore.zoom.us/docs/doc/Zoom_Client_Connection%20Process_Whitepaper.pdf.Google Scholar
- Yuting Jiang, Yifan Xiong, Lei Qu, Cheng Luo, Chen Tian, Peng Cheng, and Yongqiang Xiong. 2022. Moneo: Non-intrusive Fine-grained Monitor for AI Infrastructure. In ICC 2022-IEEE International Conference on Communications. IEEE, 2586--2591.Google ScholarDigital Library
- Misha E Kilmer, Karen Braman, Ning Hao, and Randy C Hoover. 2013. Third-order tensors as operators on matrices: A theoretical and computational framework with applications in imaging. SIAM J. Matrix Anal. Appl. 34, 1 (2013), 148--172.Google ScholarDigital Library
- Akshay Krishnamurthy and Aarti Singh. 2013. Low-rank matrix and tensor completion via adaptive sampling. Advances in neural information processing systems 26 (2013).Google Scholar
- Akshay Krishnamurthy and Aarti Singh. 2014. On the power of adaptivity in matrix completion and approximation. arXiv preprint arXiv:1407.3619 (2014).Google Scholar
- Sue E Leurgans, Robert T Ross, and Rebecca B Abel. 1993. A decomposition for three-way arrays. SIAM J. Matrix Anal. Appl. 14, 4 (1993), 1064--1083.Google ScholarDigital Library
- Yuliang Li, Rui Miao, Hongqiang Harry Liu, Yan Zhuang, Fei Feng, Lingbo Tang, Zheng Cao, Ming Zhang, Frank Kelly, Mohammad Alizadeh, et al. 2019. HPCC: High precision congestion control. In Proceedings of the ACM Special Interest Group on Data Communication. 44--58.Google Scholar
- Ji Liu, Przemyslaw Musialski, Peter Wonka, and Jieping Ye. 2012. Tensor completion for estimating missing values in visual data. IEEE transactions on pattern analysis and machine intelligence 35, 1 (2012), 208--220.Google Scholar
- Xiao-Yang Liu, Shuchin Aeron, Vaneet Aggarwal, Xiaodong Wang, and Min-You Wu. 2015. Adaptive sampling of RF fingerprints for fine-grained indoor localization. IEEE Transactions on Mobile Computing 15, 10 (2015), 2411--2423.Google ScholarDigital Library
- Yipeng Liu, Zhen Long, Huyan Huang, and Ce Zhu. 2019. Low CP rank and tucker rank tensor completion for estimating missing components in image data. IEEE Transactions on Circuits and Systems for Video Technology 30, 4 (2019), 944--954.Google ScholarCross Ref
- Yuanyuan Liu, Fanhua Shang, Wei Fan, James Cheng, and Hong Cheng. 2015. Generalized higher order orthogonal iteration for tensor learning and decomposition. IEEE transactions on neural networks and learning systems 27, 12 (2015), 2551--2563.Google Scholar
- Yuanyuan Liu, Fanhua Shang, Licheng Jiao, James Cheng, and Hong Cheng. 2014. Trace norm regularized CANDE-COMP/PARAFAC decomposition with missing data. IEEE transactions on cybernetics 45, 11 (2014), 2437--2448.Google Scholar
- Nick McKeown, Tom Anderson, Hari Balakrishnan, Guru Parulkar, Larry Peterson, Jennifer Rexford, Scott Shenker, and Jonathan Turner. 2008. OpenFlow: enabling innovation in campus networks. ACM SIGCOMM computer communication review 38, 2 (2008), 69--74.Google ScholarDigital Library
- Sky McKinley and Megan Levine. 1998. Cubic spline interpolation. College of the Redwoods 45, 1 (1998), 1049--1060.Google Scholar
- Xiaonan Nie, Xupeng Miao, Zhi Yang, and Bin Cui. 2022. Tsplit: Fine-grained gpu memory management for efficient dnn training via tensor splitting. In 2022 IEEE 38th International Conference on Data Engineering (ICDE). IEEE, 2615--2628.Google ScholarCross Ref
- Bernardino Romera-Paredes and Massimiliano Pontil. 2013. A new convex relaxation for tensor completion. Advances in neural information processing systems 26 (2013).Google Scholar
- Josep Sanjuas-Cuxart, Pere Barlet-Ros, Nick Duffield, and Ramana Kompella. 2012. Cuckoo sampling: Robust collection of flow aggregates under a fixed memory budget. In 2012 Proceedings IEEE INFOCOM. IEEE, 2751--2755.Google ScholarCross Ref
- Robin Sommer and Anja Feldmann. 2002. NetFlow: Information loss or win?. In Proceedings of the 2nd ACM SIGCOMM Workshop on Internet measurment. 173--174.Google ScholarDigital Library
- Ryota Tomioka, Kohei Hayashi, and Hisashi Kashima. 2010. Estimation of low-rank tensors via convex optimization. arXiv preprint arXiv:1010.0789 (2010).Google Scholar
- Ledyard R Tucker. 1966. Some mathematical notes on three-mode factor analysis. Psychometrika 31, 3 (1966), 279--311.Google ScholarCross Ref
- Mea Wang, Baochun Li, and Zongpeng Li. 2004. sFlow: Towards resource-efficient and agile service federation in service overlay networks. In 24th International Conference on Distributed Computing Systems, 2004. Proceedings. IEEE, 628--635.Google ScholarCross Ref
- Canhua Wu, Nengwen Zhao, Lixin Wang, Xiaoqin Yang, Shining Li, Ming Zhang, Xing Jin, Xidao Wen, Xiaohui Nie, Wenchi Zhang, et al . 2021. Identifying root-cause metrics for incident diagnosis in online service systems. In 2021 IEEE 32nd International Symposium on Software Reliability Engineering (ISSRE). IEEE, 91--102.Google ScholarCross Ref
- Qin Wu, John Strassner, Adrian Farrel, and Liang Zhang. 2016. Network Telemetry and Big Data Analysis. Internet-Draft draft-wu-t2trg-network-telemetry-00. Internet Engineering Task Force. https://datatracker.ietf.org/doc/draft-wu-t2trg-network-telemetry/00/ Work in Progress.Google Scholar
- Kun Xie, Yuxiang Chen, Xin Wang, Gaogang Xie, Jiannong Cao, Jigang Wen, Guangming Yang, and Jiaqi Sun. 2020. Accurate and fast recovery of network monitoring data with GPU-accelerated tensor completion. IEEE/ACM Transactions on Networking 28, 4 (2020), 1601--1614.Google ScholarDigital Library
- Yangyang Xu and Wotao Yin. 2013. A block coordinate descent method for regularized multiconvex optimization with applications to nonnegative tensor factorization and completion. SIAM Journal on imaging sciences 6, 3 (2013), 1758--1789.Google Scholar
- Zemin Zhang and Shuchin Aeron. 2016. Exact tensor completion using t-SVD. IEEE Transactions on Signal Processing 65, 6 (2016), 1511--1526.Google ScholarDigital Library
Index Terms
- FineMon: An Innovative Adaptive Network Telemetry Scheme for Fine-Grained, Multi-Metric Data Monitoring with Dynamic Frequency Adjustment and Enhanced Data Recovery
Recommendations
Multi-dimensional imaging data recovery via minimizing the partial sum of tubal nuclear norm
AbstractIn this paper, we investigate tensor recovery problems within the tensor singular value decomposition (t-SVD) framework. We propose the partial sum of the tubal nuclear norm (PSTNN) of a tensor. The PSTNN is a surrogate of the tensor ...
Highlights- We propose the partial sum of the tubal nuclear norm (PSTNN) for tensor recovery.
Network telemetry: towards a top-down approach
Network telemetry is about understanding what is happening in the current network. It serves as the basis for making a variety of management decisions for improving the performance, availability, security, and efficiency of networks. However, it is ...
Tensor Factorization with Total Variation and Tikhonov Regularization for Low-Rank Tensor Completion in Imaging Data
AbstractThe main aim of this paper is to study tensor factorization for low-rank tensor completion in imaging data. Due to the underlying redundancy of real-world imaging data, the low-tubal-rank tensor factorization (the tensor–tensor product of two ...
Comments