Abstract
The internet traffic data completion is an important and challenging task in network engineering. Due to the multi-dimensionality of internet traffic data, we introduce two tensor train (TT) based optimization models with temporal regularization to recover the data from an incomplete observation. Moreover, we propose two easily implementable algorithms by following the spirit of alternating minimization. It is remarkable that our algorithms have closed-form solutions and one algorithm can be implemented in a parallel way for large-scale problems. Some numerical experiments on real-world datasets show that our approaches perform better than some existing state-of-the-art matrix- and tensor-based completion methods in terms of achieving higher accuracy and taking much less computing time for some datasets.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
Notes
A fiber is defined by fixing every index but one, which is analogue of matrix row or column, we refer to Kolda and Bader (2009) for more details.
References
Acar, E., Dunlavy, D., Kolda, T., & Morup, M. (2011). Scalable tensor factorizations for incomplete data. Chemometrics and Intelligent Laboratory Systems, 106(1), 41–56.
Aggarwal, C. C. (2016). Model-based collaborative filtering. In: Recommender systems, pp. 71–138. Springer.
Bader, B. W., Kolda, T. G., et al. (2015). MATLAB Tensor Toolbox Version 2.6. Available online. http://www.sandia.gov/~tgkolda/TensorToolbox/
Bengua, J. A., Phien, H., Tuan, H. D., & Do, M. N. (2017). Efficient tensor completion for color image and video recovery: Low-rank tensor train. IEEE Transactions on Image Processing, 26(5), 2466–2479.
Bolte, J., Sabach, S., & Teboulle, M. (2014). Proximal alternating linearized minimization for nonconvex and nonsmooth problems. Mathematical Programming, 146, 459–494.
Cahn, R. (1998). Wide Area Network Design: Concepts and Tools for Optimization. San Francisco: Morgan Kaufmann Publishers.
Candès, E. J., & Recht, B. (2009). Exact matrix completion via convex optimization. Foundations of Computational Mathematics, 9(6), 717–772.
Carroll, J. D., & Chang, J. J. (1970). Analysis of individual differences in multidimensional scaling via an n-way generalization of Eckart-Young decomposition. Psychometrika, 35(3), 283–319.
Chen, X., Yang, J., & Sun, L. (2020). A nonconvex low-rank tensor completion model for spatio-temporal traffic data imputation. Transportation Research Part C Emerging Technologies, 117, 102673.
Chen, Y. C., Qiu, L., Zhang, Y., Xue, G., & Hu, Z. (2014) Robust network compressive sensing. In: Proceedings of the 20th Annual International Conference on Mobile Computing and Networking, pp. 545–556.
Da Silva, C., & Herrmann, F. (2015). Optimization on the hierarchical Tucker manifold applications to tensor completion. Linear Algebra and Its Applications, 481, 131–173.
De Lathauwer, L., Moor, B. D., & Vandewalle, J. (2000). A multilinear singular value decomposition. SIAM Journal on Matrix Analysis and Applications, 21(4), 1253–1278.
De Silva, V., & Lim, L. H. (2008). Tensor rank and the ill-posedness of the best low-rank approximation problem. SIAM Journal on Matrix Analysis and Applications, 30, 1084–1127.
Du, R., Chen, C., Yang, B., & Guan, X. (2013) VANET based traffic estimation: A matrix completion approach. In: Proceedings of IEEE GLOBECOM, pp. 30–35.
Dunlavy, D. M., Kolda, T. G., & Acar, E. (2010) Poblano v1.0: A matlab toolbox for gradient-based optimization. Sandia National Laboratories, Technical Report SAND2010-1422.
Eckart, C., & Young, G. (1936). The approximation of one matrix by another of lower rank. Psychometrika, 1, 211–218.
Gürsun, G., & Crovella, M. (2012) On traffic matrix completion in the internet. In: Proceedings of the 2012 Internet Measurement Conference, pp. 399–412.
Harshman, R. (1970). Foundations of the PARAFAC procedure: Models and methods for an “explanatory” multi-mode factor analysis. UCLA Working Papers in Phonetics 16, pp. 1–84.
Hillar, C., & Lim, L. H. (2013). Most tensor problems are NP-hard. Journal of the ACM, 60(45), 1–39.
Jain, P., Meka, R., & Dhillon, I.S. (2010). Guaranteed rank minimization via singular value projection. In: Advances in Neural Information Processing Systems, pp. 937–945.
Jiang, X., Zhong, Z., Liu, X., & So, H. C. (2017). Robust matrix completion via alternating projection. IEEE Signal Processing Letters, 24(5), 579–583.
Ko, C., Batselier, K., Daniel, L., Yu, W., & Wong, N. (2020). Fast and accurate tensor completion with total variation regularized tensor trains. IEEE Transactions on Image Processing, 29, 6918–6931.
Kolda, T. G., & Bader, B. W. (2009). Tensor decompositions and applications. SIAM Review, 51, 455–500.
Lakhina, A., Crovella, M., & Diot, C. (2004). Diagnosing network-wide traffic anomalies. In: Proceedings of ACM SIGCOMM, pp. 219–230.
Lakhina, A., Papagiannaki, K., Crovella, M., Diot, C., Kolaczyk, E., & Taft, N. (2004). Structural analysis of network traffic flows. ACM SIGMETRICS Performance Evaluation Review, 32, 61–72.
Li, J., Cai, J. F., & Zhao, H. (2020). Robust inexact alternating optimization for matrix completion with outliers. Journal of Computational Mathematics, 38(2), 337–354.
Liu, X., Wen, Z., & Zhang, Y. (2013). Limited memory block Krylov subspace optimization for computing dominant singular value decompositions. SIAM Journal on Scientific Computing, 35(3), A1641–A1668.
Majumdar, A. (2020). Matrix completion via thresholding. https://ww2.mathworks.cn/matlabcentral/fileexchange/26395-matrix-completion-via-thresholding
Mardani, M., & Giannakis, G. (2013). Robust network traffic estimation via sparsity and low rank. In: Proceedings of IEEE ICASSP, pp. 4529–4533.
Oseledets, I. (2011). Tensor-train decomposition. SIAM Journal on Scientific Computing, 33(5), 2295–2317.
Oseledets, I., & Tyrtyshnikov, E. (2010). TT-cross approximation for multidimensional arrays. Linear Algebra and Its Applications, 432(1), 70–88.
Pan, C., Ling, C., He, H., Qi, L., & Xu, Y. (2020). Low-rank and sparse enhanced Tucker decomposition for tensor completion. ArXiv:2010.00359v1
Ringberg, H., Soule, A., Rexford, J., & Diot, C. (2007) Sensitivity of PCA for traffic anomaly detection. In: Proceedings of ACM SIGMETRICS, pp. 109–120. San Diego, CA.
Roughan, M., Thorup, M., & Zhang, Y. (2003). Traffic engineering with estimated traffic matrices. In: Proceedings of ACM IMC, pp. 248–258.
Roughan, M., Zhang, Y., Willinger, W., & Qiu, L. (2012). Spatio-temporal compressive sensing and internet traffic matrices (extended version). IEEE/ACM Transactions on Networking, 20(3), 662–676.
Shang, K., Li, Y. F., & Huang, Z. H. (2019). Iterative p-shrinkage thresholding algorithm for low Tucker rank tensor recovery. Information Sciences, 482, 374–391.
Tucker, L. R. (1966). Some mathematical notes on three-mode factor analysis. Psychometrika, 31(3), 279–311.
Tune, P., & Roughan, M. (2015) Internet traffic matrices: A primer. In: H. haddadi, O. Bonaventure (eds.) Recent Advances in Networking, pp. 1–56.
Wang, Y., Zhang, Y., Piao, X., Liu, H., & Zhang, K. (2018). Traffic data reconstruction via adaptive spatial-temporal correlations. IEEE Transactions on Intelligent Transportation Systems, 20(4), 1531–1543.
Wei, K., Cai, J. F., Chan, T. F., & Leung, S. (2016). Guarantees of riemannian optimization for low rank matrix recovery. SIAM Journal of Matrix Analysis and Application, 37(3), 1198–1222.
Xie, K., Peng, C., Wang, X., Xie, G., Wen, J., Cao, J., et al. (2018). Accurate recovery of internet traffic data under variable rate measurements. IEEE/ACM Transactions on Networking, 26(3), 1137–1150.
Xie, K., Wang, L., Wang, X., Xie, G., Wen, J., & Zhang, G. (2016). Accurate recovery of internet traffic data: A tensor completion approach. In: IEEE INFOCOM 2016 - The 35th Annual IEEE International Conference on Computer Communications, pp. 1–9.
Xie, K., Wang, X., Wang, X., Chen, Y., Xie, G., Ouyang, Y., et al. (2019). Accurate recovery of missing network measurement data with localized tensor completion. IEEE/ACM Transactions on Networking, 27(6), 2222–2235.
Yu, X., Luo, Z., Qi, L., & Xu, Y. (2021). Slrta: A sparse and low-rank tensor-based approach to internet traffic anomaly detection. Neurocomputing, 434, 295–314.
Yuan, M., & Zhang, C. H. (2016). On tensor completion via nuclear norm minimization. Foundations of Computational Mathematics, 16, 1031–1068.
Zhao, Q., Ge, Z., Wang, J., & Xu, J. (2006). Robust taffic matrix estimation with imperfect information: making use of multiple data sources. ACM SIGMETRICS Performance Evaluation Review, 34, 133–144.
Zhou, H., Zhang, D., Xie, K., & Chen, Y.: Spatio-temporal tensor completion for imputing missing internet traffic data. In: 2015 IEEE 34th International Performance Computing and Communications Conference (IPCCC), pp. 1–7.
Acknowledgements
The authors would like to thank the two reviewers for their careful reading and valuable comments, which helped us improve the presentation of this paper. Also, they are grateful to Prof. Kun Xie and Dr. Huibin Zhou [two authors of Zhou et al. (2015)] for their kind help on the numerical experiments. Moreover, many thanks go to the authors who shared their code and data on websites. H. He and C. Ling were supported in part by National Natural Science Foundation of China (Nos. 11771113 and 11971138) and Zhejiang Provincial Natural Science Foundation (Nos. LY19A010019, LY20A010018, and LD19A010002).
Author information
Authors and Affiliations
Corresponding author
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
About this article
Cite this article
Zhang, Z., Ling, C., He, H. et al. A tensor train approach for internet traffic data completion. Ann Oper Res 339, 1461–1479 (2024). https://doi.org/10.1007/s10479-021-04147-4
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10479-021-04147-4