Abstract
Satellite cloud images can help meteorologists characterize the weather patterns, such as identifying tropical cyclone (TC) intensity, detecting climate anomaly regions and predicting rain effects, which makes satellite cloud image forecasting become an important task. In recent years, an increasing number of deep learning models have demonstrated their ability to predict spatiotemporal types of data. However, applying these methods directly to predict satellite cloud images does not consider the chaotic nature of the atmosphere and the satellite images of large scale observation areas, especially in extreme climate events, such as tropical cyclones. Hence, we propose both a novel deep learning model called “Satellite Cloud Spatio-Temporal sequence” (SCSTque) and a dataset benchmark called “Tropical Cyclone Cloudage Map Dataset” (TCCMD) for tropical cyclone satellite images to address these problems. Specifically, the SCSTque model is a sequence-to-sequence (Seq2Seq) autoencoder architecture to use the previous images to predict the next few images, which can fully extract spatial appearance features and temporal dynamics. The TCCMD dataset is a real-world large-scale dataset, which contains satellite image sequences for each corresponding tropical cyclone event from 2010 to 2020. The novel method is validated using the TCCMD dataset and the Moving MNIST dataset. The experimental results proved the SCSTque method outperforms the baseline Seq2Seq-based methods, including a ConvLSTM model, a PredRNN model and a FCLSTM model.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
Data Availability
Not applicable
References
Arking A, Lo RC, Rosenfeld A (1978) A fourier approach to cloud motion estimation. J Appl Meteorol (1962-1982), pp 735–744
Ballas N, Yao L, Pal C, Courville A (2015) Delving deeper into convolutional networks for learning video representations. arXiv:1511.06432
Berthomier L, Pradel B, Perez L (2020) Cloud cover nowcasting with deep learning. In: 2020 10th international conference on image processing theory, tools and applications (IPTA). IEEE, pp 1–6
Carvalho LM, Jones C (2001) A satellite method to identify structural properties of mesoscale convective systems based on the maximum spatial correlation tracking technique (mascotte). J Appl Meteorol Climatol 40 (10):1683–1701
Cho K, Van Merriënboer B, Gulcehre C, Bahdanau D, Bougares F, Schwenk H, Bengio Y (2014) Learning phrase representations using RNN encoder-decoder for statistical machine translation. arXiv:1406.1078
ECMWF (2023) https://cds.climate.copernicus.eu
Endlich R, Wolf D, Hall D, Brain A (1971) Use of a pattern recognition technique for determining cloud motions from sequences of satellite photographs. J Appl Meteorol Climatol 10(1):105–117
Evans AN (2006) Cloud motion analysis using multichannel correlation-relaxation labeling. IEEE Geosci Remote Sens Lett 3(3):392–396
Fujita T, Bradbury DL, Murino C (1968) A study of mesoscale cloud motions computed from ATS-I and terrestrial photographs, Department of the Geophysical Sciences University of Chicago
Ham Y-G, Kim J-H, Luo J-J (2019) Deep learning for multi-year enso forecasts. Nature 573(7775):568–572
Himawari-8 Information (2023) http://agora.ex.nii.ac.jp/digitaltyphoon/himawari-3g/
Hong S, Kim S, Joh M, Song S-K (2017) Psique: next sequence prediction of satellite images using a convolutional sequence-to-sequence network. arXiv:1711.10644
IBTrACS (2023) https://www.ncei.noaa.gov/products/international-best-track-archive
Lee J-H, Lee SS, Kim HG, Song S-K, Kim S, Ro YM (2019) Mcsip net: multichannel satellite image prediction via deep neural network. IEEE Trans Geosci Remote Sens 58(3):2212–2224
Liu K, Zhang R, Li W, Zhao Z, Jiang H (2008) Cloud cluster movement forecast technique of satellite cloud pictures based on singular value decomposition and artificial neural networks. J PLA Univ Sci Tech 3:298–302
Luo C, Li X, Ye Y (2020) Pfst-lstm: a spatiotemporal lstm model with pseudoflow prediction for precipitation nowcasting. IEEE J Sel Top Appl Earth Obs Remote Sens 14:843–857
Mao X, Shen C, Yang Y-B (2016) Image restoration using very deep convolutional encoder-decoder networks with symmetric skip connections. Adv Neural Inf Process Syst 29
Marzano FS, Rivolta G, Coppola E, Tomassetti B, Verdecchia M (2007) Rainfall nowcasting from multisatellite passive-sensor images using a recurrent neural network. IEEE Trans Geosci Remote Sens 45(11):3800–3812
Mathieu M, Couprie C, LeCun Y (2015) Deep multi-scale video prediction beyond mean square error. arXiv:1511.05440
Papin C, Bouthemy P, Mémin E, Rochard G (2000) Tracking and characterization of highly deformable cloud structures. In: European conference on computer vision. Springer, pp 428–442
Pradhan R, Aygun RS, Maskey M, Ramachandran R, Cecil DJ (2017) Tropical cyclone intensity estimation using a deep convolutional neural network. IEEE Trans Image Process 27(2):692–702
Reda FA, Liu G, Shih KJ, Kirby R, Barker J, Tarjan D, Tao A, Catanzaro B (2018) Sdc-net: video prediction using spatially-displaced convolution. In: Proceedings of the european conference on computer vision (ECCV), pp 718–733
Rivolta G, Marzano F, Coppola E, Verdecchia M (2006) Artificial neural-network technique for precipitation nowcasting from satellite imagery. Adv Geosci 7:97–103
Shi X, Chen Z, Wang H, Yeung D-Y, Wong W-K, Woo W-C (2015) Convolutional lstm network: a machine learning approach for precipitation nowcasting. Adv Neural Inf Process Syst 28
Shi X, Gao Z, Lausen L, Wang H, Yeung D-Y, Wong W-K, Woo W-C (2017) Deep learning for precipitation nowcasting: a benchmark and a new model. Adv Neural Inf Process Syst 30
Smith EA (1975) The mcidas system. IEEE Trans Geosci Electron 13(3):123–136
Sutskever I, Vinyals O, Le QV (2014) Sequence to sequence learning with neural networks. Adv Neural Inf Process Syst 27
Synthetic dataset with moving digits (2023) http://yann.lecun.com/exdb/mnist/
Tan C, Feng X, Long J, Geng L (2018) Forecast-clstm: A new convolutional lstm network for cloudage nowcasting. In: 2018 IEEE visual communications and image processing (VCIP). IEEE, pp 1–4
Villegas R, Yang J, Hong S, Lin X, Lee H (2017) Decomposing motion and content for natural video sequence prediction. arXiv:1706.08033
Wang G, Liu L (2007) A multiscale identifying algorithm for heavy rainfall and application in nowcasting. Chin J Atmos Sci 31(3):400–409
Wang J, Zhang R, Yu W (2007) Non-linear forecast model of cloud clusters movement based on parameters retrieval of historical satellite cloud pictures time series. J-Natl Univ Def Technol 29(5):41
Wang Y, Long M, Wang J, Gao Z, Yu PS (2017) Predrnn: recurrent neural networks for predictive learning using spatiotemporal lstms. Adv Neural Inf Process Syst 30
Wang Y, Gao Z, Long M, Wang J, Philip SY (2018) Predrnn++: towards a resolution of the deep-in-time dilemma in spatiotemporal predictive learning. In: International conference on machine learning. PMLR, pp 5123–5132
Wang R, Teng D, Yu W, Zhang X, et al. (2022) Improvement and application of gan models for time series image prediction—a case study of time series satellite nephograms
Western North Pacific Satellite Images (2023) http://agora.ex.nii.ac.jp/digital-typhoon/region/pacific/4/
Xu Z, Du J, Wang J, Jiang C, Ren Y (2019) Satellite image prediction relying on gan and lstm neural networks. In: ICC 2019-2019 IEEE international conference on communications (ICC). IEEE, pp 1–6
Yang J, LV W, Ma Y (2010) An automatic groundbased cloud detection method based on local threshold interpolation. Acta Meteor Sin 68(6):1007–1017
Yu X, Chen Z, Chen G, Zhang H, Zhou J (2019) A tensor network for tropical cyclone wind speed estimation. In: IGARSS 2019-2019 IEEE international geoscience and remote sensing symposium. IEEE, pp 10007–10010
Yuan S, Wang C, Mu B, Zhou F, Duan W (2021) Typhoon intensity forecasting based on lstm using the rolling forecast method. Algorithms 14(3):83
Zhou L, Kambhamettu C, Goldgof DB, Palaniappan K, Hasler A (2001) Tracking nonrigid motion and structure from 2d satellite cloud images without correspondences. IEEE Transactions On Pattern Analysis And Machine Intelligence 23(11):1330–1336
Zahera HM, Sherif MA, Ngonga A (2020) Semantic-based end-to-end learning for typhoon intensity prediction. arXiv:2003.13779
Funding
This work was funded by National Natural Science Foundation of China (No.U2142206), Shanghai Sailing Program (No.19YF1436900) and Natural Science Foundation of Shanghai (No.20ZR1440900).
Author information
Authors and Affiliations
Contributions
Jie Lian and Ruirong Chen collected and analyzed data, and wrote original manuscript. Jie Lian and Ruirong Chen reviewed the manuscript. All authors read and approved the final manuscript.
Corresponding author
Ethics declarations
Conflict of Interests
Not applicable
Additional information
Communicated by: H. Babaie
Publisher’s note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.
About this article
Cite this article
Lian, J., Chen, R. A sequence-to-sequence based multi-scale deep learning model for satellite cloud image prediction. Earth Sci Inform 16, 1207–1225 (2023). https://doi.org/10.1007/s12145-023-00945-5
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s12145-023-00945-5