Abstract
Modeling spatiotemporal interactions in multivariate time series is key to their effective processing, but challenging because of their irregular and often unknown structure. Statistical properties of the data provide useful biases to model interdependencies and are leveraged by correlation and covariance-based networks as well as by processing pipelines relying on principal component analysis (PCA). However, PCA and its temporal extensions suffer instabilities in the covariance eigenvectors when the corresponding eigenvalues are close to each other, making their application to dynamic and streaming data settings challenging. To address these issues, we exploit the analogy between PCA and graph convolutional filters to introduce the SpatioTemporal coVariance Neural Network (STVNN), a relational learning model that operates on the sample covariance matrix of the time series and leverages joint spatiotemporal convolutions to model the data. To account for the streaming and non-stationary setting, we consider an online update of the parameters and sample covariance matrix. We prove the STVNN is stable to the uncertainties introduced by these online estimations, thus improving over temporal PCA-based methods. Experimental results corroborate our theoretical findings and show that STVNN is competitive for multivariate time series processing, it adapts to changes in the data distribution, and it is orders of magnitude more stable than online temporal PCA.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
Notes
- 1.
Note that in Equation (13) we kept explicitly the parameter suboptimality term outside the notation \(\mathcal {O}(1/t)\) to highlight the role of filter updates in the stability. The other terms \(\mathcal {O}(1/t)\) include perturbations due to the covariance uncertainty.
References
Arguez, A., et al.: NOAA’s 1981–2010 U.S. climate normals: an overview. Bull. Am. Meteorological Soc. 93(11), 1687 – 1697 (2012)
Bessadok, A., Mahjoub, M.A., Rekik, I.: Graph neural networks in network neuroscience. IEEE Trans. Pattern Anal. Mach. Intell. 45(5), 5833–5848 (2022)
Bien, J., Tibshirani, R.J.: Sparse estimation of a covariance matrix. Biometrika 98(4), 807–820 (2011)
Brand, M.: Incremental singular value decomposition of uncertain data with missing values. In: Heyden, A., Sparr, G., Nielsen, M., Johansen, P. (eds.) Computer Vision – ECCV 2002, pp. 707–720. Springer, Berlin, Heidelberg (2002)
Brooks, D.: Deep learning and information geometry for time-series classification, Ph.D. thesis, Sorbonne Université (2020)
Cao, D., et al.: Spectral temporal graph neural network for multivariate time-series forecasting. Adv. Neural. Inf. Process. Syst. 33, 17766–17778 (2020)
Cardoso, J.V.D.M., Ying, J., Palomar, D.P.: Algorithms for learning graphs in financial markets. arXiv preprint arXiv:2012.15410 (2020)
Cardot, H., Degras, D.: Online principal component analysis in high dimension: which algorithm to choose? Int. Stat. Rev. 86(1), 29–50 (2018)
Friedman, J., Hastie, T., Tibshirani, R.: Sparse inverse covariance estimation with the graphical lasso. Biostatistics 9(3), 432–441 (2007)
Gama, F., Bruna, J., Ribeiro, A.: Stability properties of graph neural networks. IEEE Trans. Signal Process. 68, 5680–5695 (2020)
Garrigos, G., Gower, R.M.: Handbook of convergence theorems for (stochastic) gradient methods (2023)
Girault, B.: Stationary graph signals using an isometric graph translation. In: 2015 23rd European Signal Processing Conference (EUSIPCO), pp. 1516–1520 (2015)
Golub, G.: Matrix Computations. JHU Press (2013). https://doi.org/10.56021/9781421407944
Guo, T., Xu, Z., Yao, X., Chen, H., Aberer, K., Funaya, K.: Robust online time series prediction with recurrent neural networks. In: 2016 IEEE International Conference on Data Science and Advanced Analytics (DSAA), pp. 816–825 (2016)
Habib, B., Isufi, E., van Breda, W., Jongepier, A., Cremer, J.L.: Deep statistical solver for distribution system state estimation. IEEE Trans. Power Syst. 39(2), 4039–4050 (2024)
Isufi, E., Gama, F., Shuman, D.I., Segarra, S.: Graph filters for signal processing and machine learning on graphs. IEEE Trans. Signal Process., 1–32 (2024). https://doi.org/10.1109/TSP.2024.3349788
Isufi, E., Loukas, A., Perraudin, N., Leus, G.: Forecasting time series with Varma recursions on graphs. IEEE Trans. Signal Process. 67(18), 4870–4885 (2019)
Jiang, W., Luo, J.: Graph neural network for traffic forecasting: a survey. Expert Syst. Appl. 207, 117921 (2022)
Jin, M., Zheng, Yu., Li, Y.F., Chen, S., Yang, B., Pan, S.: Multivariate time series forecasting with dynamic graph neural ODEs. IEEE Trans. Knowl. Data Eng. 35(9), 9168–9180 (2023). https://doi.org/10.1109/TKDE.2022.3221989
Jolliffe, I.T.: Rotation of ill-defined principal components. J. Royal Stat. Soc. Ser. C Appl. Stat. 38(1), 139–147 (1989)
Jolliffe, I.T.: Principal Component Analysis. Springer, New York (2002)
Jolliffe, I.T., Cadima, J.: Principal component analysis: a review and recent developments. Philos. Trans. Royal Soc. A Math. Phys. Eng. Sci. 374 (2016)
Kerimov, B., et al.: Assessing the performances and transferability of graph neural network metamodels for water distribution systems. J. Hydroinf. 25(6), 2223–2234 (2023)
Kolaczyk, E.D.: Statistical Analysis of Network Data: Methods and Models. Springer, New York, NY (2009). https://doi.org/10.1007/978-0-387-88146-1
Loukas, A.: How close are the eigenvectors of the sample and actual covariance matrices? In: Precup, D., Teh, Y.W. (eds.) Proceedings of the 34th International Conference on Machine Learning. Proceedings of Machine Learning Research, vol. 70, pp. 2228–2237. PMLR (2017)
Marques, A.G., Segarra, S., Leus, G., Ribeiro, A.: Stationary graph processes and spectral estimation. IEEE Trans. Signal Process. 65(22), 5911–5926 (2017)
Perraudin, N., Vandergheynst, P.: Stationary signal processing on graphs. IEEE Trans. Signal Process. 65(13), 3462–3477 (2017)
Pham, Q., Liu, C., Sahoo, D., Hoi, S.C.: Learning fast and slow for online time series forecasting. arXiv preprint arXiv:2202.11672 (2022)
Pourahmadi, M., Noorbaloochi, S.: Multivariate time series analysis of neuroscience data: some challenges and opportunities. Curr. Opin. Neurobiol. 37, 12–15 (2016)
Rui, L., Nejati, H., Cheung, N.M.: Dimensionality reduction of brain imaging data using graph signal processing. In: 2016 IEEE International Conference on Image Processing (ICIP), pp. 1329–1333 (2016)
Saadallah, A., Mykula, H., Morik, K.: Online adaptive multivariate time series forecasting. In: Amini, M.R., Canu, S., Fischer, A., Guns, T., Kralj Novak, P., Tsoumakas, G. (eds.) Machine Learning and Knowledge Discovery in Databases, pp. 19–35. Springer, Cham (2023)
Sanhudo, L., Rodrigues, J.: Enio Vasconcelos Filho: multivariate time series clustering and forecasting for building energy analysis: application to weather data quality control. J. Build. Eng. 35, 101996 (2021)
Shahid, N., Perraudin, N., Kalofolias, V., Puy, G., Vandergheynst, P.: Fast robust PCA on graphs. IEEE J. Sel. Top. Signal Process. 10(4), 740–756 (2016)
Sihag, S., Mateos, G., McMillan, C., Ribeiro, A.: Covariance neural networks. Adv. Neural. Inf. Process. Syst. 35, 17003–17016 (2022)
Sihag, S., Mateos, G., McMillan, C., Ribeiro, A.: Explainable brain age prediction using covariance neural networks. Adv. Neural Inf. Process. Syst. 36 (2024)
Sihag, S., Mateos, G., McMillan, C.T., Ribeiro, A.: Transferablility of covariance neural networks and application to interpretable brain age prediction using anatomical features. arXiv preprint arXiv:2305.01807 (2023)
Vershynin, R.: High-dimensional probability: an introduction with applications in data science. In: Cambridge Series in Statistical and Probabilistic Mathematics. Cambridge University Press (2018)
Wu, Z., Pan, S., Long, G., Jiang, J., Chang, X., Zhang, C.: Connecting the dots: multivariate time series forecasting with graph neural networks. In: Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 753–763 (2020)
Wu, Z., Pan, S., Long, G., Jiang, J., Chang, X., Zhang, C.: MTGNN (2020). https://github.com/nnzhan/MTGNN
Yi, K., et al.: FourierGNN: rethinking multivariate time series forecasting from a pure graph perspective. arXivpreprint arXiv:2311.06190 (2023)
Yi, K., et al.: Frequency-domain MLPS are more effective learners in time series forecasting. arXiv preprint arXiv:2311.06184 (2023)
Acknowledgments
The study was supported by the TU Delft AI Labs program and the NWO OTP GraSPA proposal #19497.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Ethics declarations
Disclosure of Interests
The authors have no competing interests to declare that are relevant to the content of this article.
1 Electronic supplementary material
Below is the link to the electronic supplementary material.
Rights and permissions
Copyright information
© 2024 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Cavallo, A., Sabbaqi, M., Isufi, E. (2024). Spatiotemporal Covariance Neural Networks. In: Bifet, A., Davis, J., Krilavičius, T., Kull, M., Ntoutsi, E., Žliobaitė, I. (eds) Machine Learning and Knowledge Discovery in Databases. Research Track. ECML PKDD 2024. Lecture Notes in Computer Science(), vol 14942. Springer, Cham. https://doi.org/10.1007/978-3-031-70344-7_2
Download citation
DOI: https://doi.org/10.1007/978-3-031-70344-7_2
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-70343-0
Online ISBN: 978-3-031-70344-7
eBook Packages: Computer ScienceComputer Science (R0)
