Skip to main content
SpringerLink
Log in

Two-stream convolutional LSTM for precipitation nowcasting

  • S.I. : Deep Learning for Time Series Data
  • Published:
Neural Computing and Applications Aims and scope Submit manuscript

Abstract

Reliable precipitation nowcasting is essential to many fields, which can guide people to reasonably carry out production activities and respond to rainstorm disasters. However, precipitation nowcasting is a very challenging task because of correlation and heterogeneity both in space and in time. Most previous studies have not adequately captured the long-term and long-range spatiotemporal dependencies in the data, leading to insufficient modeling and poor prediction performance. To make more accurate prediction, we propose a novel deep learning model for precipitation nowcasting, called two-stream convolutional LSTM which includes short-term sub-network and long-term sub-network. The two sub-networks, respectively, make predictions on inputs at different time intervals to capture the heterogeneity of rainfall data. On this basis, an innovative recombination module is proposed to fuse the outputs of two sub-networks. In addition, we embed the 3D convolutions and self-attention mechanism to construct a new memory cell, named 3D-SA-LSTM, to extract the spatiotemporal feature. Two-stream convolutional LSTM achieves the state-of-the-art prediction performance on a real-world large-scale dataset and is a more flexible framework that can be conveniently applied to other similarly time series prediction tasks: traffic forecasting and planning, financial analysis and management, actions recognition and prediction, etc.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7

Similar content being viewed by others

References

  1. Yuehong S, Wanchang Z, Yonghe L (2009) Application of back-propagation neural network in precipitation estimation with doppler radar. Plateau Meteor 28(4):846–853

    Google Scholar 

  2. Heuvelink D, Berenguer M, Brauer CC et al (2020) Hydrological application of radar rainfall nowcasting in the Netherlands. Environ Int 136:105431. https://doi.org/10.1016/j.envint.2019.105431

    Article  Google Scholar 

  3. Sun J, Xue M, Wilson JW et al (2014) Use of NWP for nowcasting convective precipitation: Recent progress and challenges. Bull Am Meteor Soc 95(3):409–426. https://doi.org/10.1175/BAMS-D-11-00263.1

    Article  Google Scholar 

  4. Wang G, Wong W, Liu L et al (2013) Application of multi-scale tracking radar echoes scheme in quantitative precipitation nowcasting. Adv Atmos Sci 30(2):448–460. https://doi.org/10.1175/BAMS-D-11-00263.1

    Article  Google Scholar 

  5. Li H, Wang X, Wu S et al (2021) A new method for determining an optimal diurnal threshold of GNSS precipitable water vapor for precipitation forecasting. Remote Sens 13(7):1390. https://doi.org/10.3390/rs13071390

    Article  Google Scholar 

  6. Chen M, Bica B, Tüchler L et al (2017) Statistically extrapolated nowcasting of summertime precipitation over the Eastern Alps. Adv Atmos Sci 34:925–938. https://doi.org/10.1007/s00376-017-6185-4

    Article  Google Scholar 

  7. LeCun Y, Bengio Y, Hinton G (2015) Deep learning. Nature 521(7553):436–444. https://doi.org/10.1038/nature14539

    Article  Google Scholar 

  8. Shi X, Chen Z, Wang H, Yeung DY, Wong WK, Woo WC (2015) Convolutional LSTM network: a machine learning approach for precipitation nowcasting. In: Advances in neural information processing systems, p 802–810.

  9. Kong Y, Fu Y (2018) Human action recognition and prediction: A survey. arXiv:1806.11230

  10. Sahin C, Garcia-Hernando G, Sock J et al (2020) A review on object pose recovery: from 3d bounding box detectors to full 6d pose estimators. Image Vis Comput 96:103898. https://doi.org/10.1016/j.imavis.2020.103898

    Article  Google Scholar 

  11. Sun H, Tang M, Peng W et al (2021) Interval prediction of short-term building electrical load via a novel multi-objective optimized distributed fuzzy model. Neural Comput & Applic. https://doi.org/10.1007/s00521-021-06162-9

    Article  Google Scholar 

  12. Boogaard MV, Pickkers P, Hoeven HV et al (2010) PREDICT, Prediction of Delirium in ICU Patients: development and validation of a prediction model. Crit Care 14:P498. https://doi.org/10.1186/cc8730

    Article  Google Scholar 

  13. Chen Z, Xiao X, Li C et al (2016) Erratum to: Real-time transient stability status prediction using cost-sensitive extreme learning machine. Neural Comput & Applic 27:333. https://doi.org/10.1007/s00521-015-1926-8

    Article  Google Scholar 

  14. Baehr J, Fröhlich K, Botzet M et al (2015) The prediction of surface temperature in the new seasonal prediction system based on the MPI-ESM coupled climate model. Clim Dyn 44(9–10):2723–2735. https://doi.org/10.1007/s00382-014-2399-7

    Article  Google Scholar 

  15. Kosek W, McCarthy DD, Luzum BJ (2001) El Niño impact on polar motion prediction errors. Stud Geophys Geod 45(4):347–361. https://doi.org/10.1023/A:1022073503034

    Article  Google Scholar 

  16. Chiu PC, Selamat A, Krejcar O et al (2021) Imputation of rainfall data using the sine cosine function fitting neural network. Int J Interact Multimed and Art Intellig. https://doi.org/10.9781/ijimai.2021.08.013

    Article  Google Scholar 

  17. Sutskever I, Vinyals O, Le Q V (2014) Sequence to sequence learning with neural networks. arXiv:1409.3215.

  18. Schmidhuber J (1992) Learning complex, extended sequences using the principle of history compression. Neural Comput 4(2):234–242. https://doi.org/10.1162/neco.1997.9.8.1735

    Article  Google Scholar 

  19. Cho, K., van Merriënboer, B., Gulcehre, C., et al. (2014). Learning phrase representations using RNN encoder-decoder for statistical machine translation. arXiv:1406.1078

  20. Srivastava N, Mansimov E, Salakhutdinov R (2015) Unsupervised learning of video representations using LSTMS. In: International conference on machine learning, p 843–852

  21. Ranzato MA, Huang FJ, Boureau YL et al (2007) Unsupervised learning of invariant feature hierarchies with applications to object recognition[C]. IEEE Conf Comput Vis Pattern Recogn 2007:1–8

    Google Scholar 

  22. Finn C, Goodfellow I, Levine S (2016) Unsupervised learning for physical interaction through video prediction. arXiv:1605.07157.

  23. Kalchbrenner N, van den Oord A, Simonyan K, Danihelka I, Vinyals O, Graves A, Kavukcuoglu K (2017) Video pixel networks. In: Proceedings of the international conference on machine learning. p 1771–1779

  24. Vondrick C, Pirsiavash H, Torralba A (2016) Generating videos with scene dynamics. arXiv:1609.02612

  25. Mathieu M, Couprie C, LeCun Y (2015) Deep multi-scale video prediction beyond mean square error. arXiv:1511.05440

  26. Goodfellow IJ, Pouget-Abadie J, Mirza M, Xu B, Warde-Farley D, Ozairy S, Courville A, Bengio Y (2014) Generative adversarial nets. arXiv:1406.2661v1

  27. Tran D, Bourdev L, Fergus R, Torresani L, Paluri M (2015) Learning spatiotemporal features with 3d convolutional networks. In: Proceedings of the IEEE International conference on computer vision, p 4489–4497. https://doi.org/10.1109/ICCV.2015.510.

  28. Ji S, Xu W, Yang M, Yu K (2013) 3D convolutional neural networks for human action recognition. IEEE TransPattern Anal Mach Intell 35(1):221–231. https://doi.org/10.1109/TPAMI.2012.59

    Article  Google Scholar 

  29. Wang X, Girshick R, Gupta A, He K (2018) Non-local neural networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, p 7794–7803. https://doi.org/10.1109/CVPR.2018.00813

  30. Vaswani A, Shazeer N, Parmar N et al (2017) Attention is all you need. arXiv:1706.03762

  31. Huddar MG, Sannakki SS, Rajpurohit VS (2021) Attention-based multi-modal sentiment analysis and emotion detection in conversation using RNN. Int J Inter Multimed and Art Intellig. https://doi.org/10.9781/ijimai.2020.07.004

    Article  Google Scholar 

  32. Klein B, Wolf L, Afek Y (2015) A dynamic convolutional layer for short range weather prediction. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, p 4840–4848. https://doi.org/10.1109/CVPR.2015.7299117

  33. Woo W, Wong W (2017) Operational application of optical flow techniques to radar-based rainfall nowcasting. Atmos 8:48. https://doi.org/10.3390/ATMOS8030048

    Article  Google Scholar 

  34. Shi X, Gao Z, Lausen L et al (2017) Deep learning for precipitation nowcasting: a benchmark and a new model. arXiv:1706.03458

  35. WangY, Long M, Wang J, Gao Z, Yu P S (2017) PredRNN: recurrent neural networks for predictive learning using spatiotemporal LSTMs. In: Proceedings of the Neural Information Processing Systems, p 879–888.

  36. Wang Y, Jiang L, Yang M H, Li L J, Long M, Li F (2019) Eidetic 3d lstm: A model for video prediction and beyond. In: Proceedings of the International Conference on Learning Representations

  37. Wang Y, Zhang J, Zhu H, Long M, Wang J, Yu P S (2019) Memory in memory: a predictive neural network for learning higher-order non-stationarity from spatiotemporal dynamics. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, p 9154–9162. https://doi.org/10.1109/CVPR.2019.00937.

  38. Guen, V L, Thome N (2020) Disentangling physical dynamics from unknown factors for unsupervised video prediction. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 11474–11484. https://doi.org/10.1109/CVPR42600.2020.01149

  39. Luo W, Li Y, Urtasun R, Zemel R (2016) Understanding the effective receptive field in deep convolutional neural networks. In: Proceedings of the 30th International Conference on Neural Information Processing Systems (NIPS), p 4905–4913

  40. Hu J, Shen L, Sun G (2018) Squeeze-and-excitation networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, p 7132–7141. https://doi.org/10.1109/TPAMI.2019.2913372

  41. Simonyan K, Zisserman A (2014) Two-stream convolutional networks for action recognition in videos. arXiv:1406.2199

  42. He K, Zhang X, Ren S et al (2016) Deep residual learning for image recognition, In: Proceedings of the 2016 IEEE conference on computer vision and pattern recognition, p 770–778. https://doi.org/10.1109/cvpr.2016.90

  43. Kingma D P, Ba J (2014) Adam: A method for stochastic optimization. arXiv:1412.6980

  44. Ba J L, Kiros J R, Hinton G E (2016) Layer normalization. arXiv:1607.06450

  45. Hogan RJ, Ferro CAT, Jolliffe IT et al (2010) Equitability revisited: why the “equitable threat score” is not equitable. Weather Forecast 25:710–726. https://doi.org/10.1175/2009WAF2222350.1

    Article  Google Scholar 

Download references

Funding

This work was supported by the National Natural Science Foundation of China (Nos. 61906097) and a project funded by the Priority Academic Program Development of Jiangsu Higher Education Institutions (PAPD).

Author information

Authors and Affiliations

  1. Jiangsu Collaborative Innovation Center of Atmospheric Environment and Equipment Technology(CICAEET), Nanjing University of Information Science & Technology, Nanjing, 210044, China

    Suting Chen, Xin Xu, Yanyan Zhang, Dongwei Shao & Song Zhang

  2. Nanjing University of Information Science & Technology, Wuxi Institute of Technology, Wuxi, 214100, China

    Suting Chen

  3. Jiangsu Institute of Meteorological Sciences, NanJing, 210009, China

    Mingjian Zeng

Authors
  1. Suting Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Xin Xu
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Yanyan Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Dongwei Shao
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Song Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Mingjian Zeng
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

All authors contributed to the study conception and design. Material preparation, data collection and analysis were performed by Suting Chen and Xin Xu. The first draft of the manuscript was written by Xin Xu, and all authors commented on previous versions of the manuscript. All authors read and approved the final manuscript.

Corresponding author

Correspondence to Suting Chen.

Ethics declarations

Conflicts of interest

The authors declare that they have no conflict of interests.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Below is the link to the electronic supplementary material.

Supplementary file1 (PDF 1001 kb)

Supplementary file2 (PDF 1003 kb)

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Chen, S., Xu, X., Zhang, Y. et al. Two-stream convolutional LSTM for precipitation nowcasting. Neural Comput & Applic 34, 13281–13290 (2022). https://doi.org/10.1007/s00521-021-06877-9

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00521-021-06877-9

Keywords

Search

Navigation