Abstract
Multivariate Time Series (MTS) is ubiquitous in the real world, and its prediction plays a vital role in a wide range of applications. Recently, many researchers have made persistent efforts to design powerful models. For example, Spatial-Temporal Graph Neural Networks (STGNNs) have become increasingly popular MTS prediction methods due to their state-of-the-art performance. However, we found there exists much unfairness in the comparison of the performance of existing models, which may prevent researchers from making correct judgments. Meanwhile, researchers usually have to build training pipelines that are complex and error-prone when designing new models, which further obstacles the quick and deep innovation in the MTS prediction field. In this paper, we first analyze the sources of unfairness and then propose a fair and easy-to-use benchmark, BasicTS, to address the above two issues. On the one hand, for a given MTS prediction model, BasicTS evaluates its ability based on rich datasets and standard pipelines. On the other hand, BasicTS provides users with flexible and extensible interfaces to facilitate convenient designing and exhaustive evaluation of new models. In addition, based on BasicTS, we provide performance revisits of several popular MTS prediction models. The proposed benchmark is publicly available at https://github.com/zezhishao/BasicTS.
Z. Shao—Project leader.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Cao, D., et al.: Spectral temporal graph neural network for multivariate time-series forecasting. In: Advances in Neural Information Processing Systems, vol. 33, pp. 17766–17778 (2020)
Li, M., Zhu, Z.: Spatial-temporal fusion graph neural networks for traffic flow forecasting. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 35, pp. 4189–4196 (2021)
Li, Y., Yu, R., Shahabi, C., Liu, Y.: Diffusion convolutional recurrent neural network: data-driven traffic forecasting. arXiv preprint arXiv:1707.01926 (2017)
Wu, Z., Pan, S., Long, G., Jiang, J., Zhang, C.: Graph wavenet for deep spatial-temporal graph modeling. arXiv preprint arXiv:1906.00121 (2019)
Shang, C., Chen, J., Bi, J.: Discrete graph structure learning for forecasting multiple time series. arXiv preprint arXiv:2101.06861 (2021)
Alexandrov, A., et al.: GluonTS: probabilistic and neural time series modeling in Python. J. Mach. Learn. Res. 21(116), 1–6 (2020)
Wang, J., Jiang, J., Jiang, W., Li, C., Zhao, W.X.: Libcity: an open library for traffic prediction. In: Proceedings of the 29th International Conference on Advances in Geographic Information Systems, pp. 145–148 (2021)
Cui, Y., Xie, J., Zheng, K.: Historical inertia: a neglected but powerful baseline for long sequence time-series forecasting. In: Proceedings of the 30th ACM International Conference on Information & Knowledge Management, pp. 2965–2969 (2021)
Hochreiter, S., Schmidhuber, J.: Long short-term memory. Neural Comput. 9(8), 1735–1780 (1997)
Yu, B., Yin, H., Zhu, Z.: Spatio-temporal graph convolutional networks: a deep learning framework for traffic forecasting. arXiv preprint arXiv:1709.04875 (2017)
Gao, J., et al.: MTGNN: multi-task graph neural network based few-shot learning for disease similarity measurement. Methods 198, 88–95 (2022)
Li, F., et al.: Dynamic graph convolutional recurrent network for traffic prediction: Benchmark and solution. ACM Trans. Knowl. Discov. Data (TKDD) 17, 1–21 (2021)
Bai, L., Yao, L., Li, C., Wang, X., Wang, C.: Adaptive graph convolutional recurrent network for traffic forecasting. In: Advances in Neural Information Processing Systems, vol. 33, pp. 17804–17815 (2020)
Deng, J., Chen, X., Jiang, R., Song, X., Tsang, I.W.: ST-Norm: spatial and temporal normalization for multi-variate time series forecasting. In: Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining, pp. 269–278 (2021)
Shao, Z., et al.: Decoupled dynamic spatial-temporal graph neural network for traffic forecasting. arXiv e-prints, pp. arXiv-2206 (2022)
Kingma, D.P., Ba, J.: Adam: a method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014)
Lydia, A., Francis, S.: Adagrad-an optimizer for stochastic gradient descent. Int. J. Inf. Comput. Sci. 6(5), 566–568 (2019)
Loshchilov, I., Hutter, F.: Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101 (2017)
Smith, S.L., Kindermans, P.-J., Ying, C., Le, Q.V.: Don’t decay the learning rate, increase the batch size. arXiv preprint arXiv:1711.00489 (2017)
Radiuk, P.M.: Impact of training set batch size on the performance of convolutional neural networks for diverse datasets (2017)
Amari, S.-I.: Backpropagation and stochastic gradient descent method. Neurocomputing 5(4–5), 185–196 (1993)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Liang, Y., Shao, Z., Wang, F., Zhang, Z., Sun, T., Xu, Y. (2023). BasicTS: An Open Source Fair Multivariate Time Series Prediction Benchmark. In: Gainaru, A., Zhang, C., Luo, C. (eds) Benchmarking, Measuring, and Optimizing. Bench 2022. Lecture Notes in Computer Science, vol 13852. Springer, Cham. https://doi.org/10.1007/978-3-031-31180-2_6
Download citation
DOI: https://doi.org/10.1007/978-3-031-31180-2_6
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-31179-6
Online ISBN: 978-3-031-31180-2
eBook Packages: Computer ScienceComputer Science (R0)