Abstract
In case only a limited amount of data is available, deep learning models often do not generalize well. We propose a novel deep learning architecture to deal with this problem and achieve high prediction accuracy. To this end, we combine four different concepts: greedy layer-wise pretraining, attention via performers, residual connections, and LSTM autoencoder stacking. We present the application of the method in geodetic data science, for the prediction of length-of-day and GNSS station position time series, two of the most important problems in the field of geodesy. In these particular cases, where we have only relatively short time series, we achieve state-of-the-art performance compared to other statistical and machine learning methods.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Notes
References
Altamimi, Z., Rebischung, P., Métivier, L., Collilieux, X.: ITRF2014: a new release of the International Terrestrial Reference Frame modeling nonlinear station motions. J. Geophys. Res. Solid Earth 121(8), 6109–6131 (2016)
Amiri-Simkooei, A.A., Tiberius, C.C.J.M., Teunissen, P.J.G.: Assessment of noise in GPS coordinate time series: methodology and results. J. Geophys. Res. Solid Earth 117(B7) (2007). https://doi.org/10.1029/2006JB004913
Amiri-Simkooei, A.R.: Noise in multivariate GPS position time-series. J. Geodesy 83, 175–187 (2009)
Blewitt, G., Hammond, W.C., Kreemer, C.: Harnessing the GPS data explosion for interdisciplinary science. Eos, Science News by AGU (2018). https://doi.org/10.1029/2018EO104623
Bahdanau, D., Cho, K., Bengio, Y.: Neural machine translation by jointly learning to align and translate. arXiv:1409.0473 (2014)
Bengio, Y., Lamblin, P., Popovici, D., Larochelle, H.: Greedy layer-wise training of deep networks. In: Proceedings of the Twentieth Annual Conference on Neural Information Processing Systems, Vancouver, British Columbia, Canada (2006)
Bizouard, C., Lambert, S., Gattano, C., Becker, O., Richard, J.-Y.: The IERS EOP 14C04 solution for Earth orientation parameters consistent with ITRF 2014. J. Geodesy 93(5), 621–633 (2018). https://doi.org/10.1007/s00190-018-1186-3
Choromanski, K., et al.: Rethinking attention with performers. In: The Ninth International Conference on Learning Representations, Virtual (2021)
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., Fei-Fei, L.: ImageNet: a large-scale hierarchical image database. In: 2009 IEEE Conference on Computer Vision and Pattern Recognition, Miami, Florida, pp. 248–255. IEEE (2009)
Dobslaw, H., Dill, R.: Predicting earth orientation changes from global forecasts of atmosphere-hydrosphere dynamics. Adv. Space Res. 61(4), 1047–1054 (2018)
Earth Orientation Center. https://hpiers.obspm.fr/eop-pc
Egger, J., Weickmann, K., Honika, K.-P.: Angular momentum in the global atmospheric circulation. Rev. Geophys. 45(4007) (2007). https://doi.org/10.1029/2006RG000213
Goodfellow, I., Bengio, Y., Courville, A.: Deep Learning. MIT Press, Cambridge (2017)
He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, USA, pp. 770–778. IEEE (2016)
Hochreiter, S., Schmidhuber, J.: Long short-term memory. Neural Comput. 9(8), 1735–1780 (1997)
International Earth Rotation and Reference Systems Service. https://www.iers.org/IERS/EN/DataProducts/EarthOrientationData/eop.html
Jiang, P., Chen, C., Liu, X.: Time series prediction for evolutions of complex systems: a deep learning approach. In: 2016 IEEE International Conference on Control and Robotics Engineering, Singapore (2016)
Kiani Shahvandi, M., Soja, B.: Modified deep transformers for geodetic GNSS time series prediction. In: IEEE International Geoscience and Remote Sensing Symposium, Belgium (2021)
Kingma, D.P., Ba, J.: Adam: a method for stochastic optimization. In: International Conference on Learning Representations, San Diego, California, USA (2015)
Kim, J., El-Khamy, M., Lee, J.: Residual LSTM: design of a deep recurrent architecture for distant speech recognition. In: Proceedings of the Interspeech, pp. 1591–1595 (2017)
Krizhevsky, A., Sutskever, I., Hinton, G.: ImageNet classification with deep convolutional neural networks. Commun. ACM 60, 84–90 (2017)
LeCunn, Y., Bengio, Y., Hinton, G.: Deep learning. Nature 521, 436–444 (2015)
Lin, K., Lin, Q., Zhou, C., Yao, J.: Time series prediction based on linear regression and SVR. In: Third International Conference on Natural Computation (2007)
Liu, J., Gong, X.: Attention mechanism enhanced LSTM with residual architecture and its application for protein-protein interaction residue pairs prediction. BMC Bioinform. 20 (2019). https://doi.org/10.1186/s12859-019-3199-1
Loffe, S., Szegedy, C.: Batch Normalization: accelerating deep network training by reducing internal covariate shift. In: Proceedings of the 32nd International Conference on Machine Learning, pp. 448–456 (2015)
Makridakis, S., Spiliotis, E., Assimakopoulos, V.: Statistical and machine learning forecasting methods: concerns and ways forward. Plos One 13, e0194889 (2018)
Modiri, S., Belda, S., Hoseini, M., Heinkelmann, R., Ferrándiz, J.M., Schuh, H.: A new hybrid method to improve the ultra-short-term prediction of LOD. J. Geodesy 94(2), 1–14 (2020). https://doi.org/10.1007/s00190-020-01354-y
Nair, V., Hinton, G.: Rectified linear units improve restricted Boltzmann machines. In: Proceedings of the 27th International Conference on Machine Learning, Haifa, Israel (2010)
Petit, G., Luzum, B.: IERS Conventions. International Earth Rotation and Reference Systems Service (2010)
Piccolomini, E.L., Gandolfi, S., Poluzzi, L., Tavasci, L., Cascarano, P., Pascucci, A.: Recurrent neural networks applied to GNSS time series for denoising and prediction. In: 26th International Symposium on Temporal Representation and Reasoning (2019)
Qiao, H., Wang, T., Wang, P., Qiao, S., Zhang, L.: A time-distributed spatiotemporal feature learning method for machine health monitoring with multi-sensor time series. Sensors 18(9), 2932 (2018)
Sagheer, A., Kotb, M.: Unsupervised pre-training of a deep LSTM-based stacked autoencoder for multivariate time series forecasting problems. Nat. Sci. Rep. 9, 19038 (2019)
Sutskever, I., Vinyals, O., Le, Q.-V.: Sequence to sequence learning with neural networks. In: Proceedings of the 27th International Conference on Neural Information Processing Systems, pp. 3104–3112 (2014)
Vaswani, A., et al.: Attention is all you need. In: 31st Conference on Neural Information Processing Systems (NIPS 2017), Long Beach, CA, USA (2017)
Veerman, M.-A., Heerwaarden, C.C.-v.: Trends in and closure of the atmospheric angular momentum budget in the 20th century in ERA-20C. Q. J. R. Meteorol. Soc. 145(724), 2990–3003 (2019)
Wang, J., Nie, G., Gao, S., Wu, S., Li, H., Ren, X.: Landslide deformation prediction based on a GNSS time series analysis and recurrent neural network model. Remote Sens. 13, 1055 (2021)
Zhai, X., Kolesnikov, A., Houlsby, N., Beyer, L.: Scaling vision transformers. arXiv (2021)
Zhao, H., Jia, J., Koltun, V.: Exploring self-attention for image recognition. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10076–10085, Virtual (2020)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2022 Springer Nature Switzerland AG
About this paper
Cite this paper
Kiani Shahvandi, M., Soja, B. (2022). Small Geodetic Datasets and Deep Networks: Attention-Based Residual LSTM Autoencoder Stacking for Geodetic Time Series. In: Nicosia, G., et al. Machine Learning, Optimization, and Data Science. LOD 2021. Lecture Notes in Computer Science(), vol 13163. Springer, Cham. https://doi.org/10.1007/978-3-030-95467-3_22
Download citation
DOI: https://doi.org/10.1007/978-3-030-95467-3_22
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-95466-6
Online ISBN: 978-3-030-95467-3
eBook Packages: Computer ScienceComputer Science (R0)