Skip to main content

Deformable Multi-Scale Network for Snow Removal in Video

  • Conference paper
  • First Online:
Pattern Recognition (ICPR 2024)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 15332))

Included in the following conference series:

  • 83 Accesses

Abstract

Snowfall severely degrades outdoor video visibility while reducing the performance of subsequent vision tasks. Although video recovery methods based on deep learning have achieved amazing accomplishments, video snow removal still faces problems such as varying scales and intricate trajectories of snowflakes, which makes it difficult to remove snowflakes and easy to create artifacts on moving objects. To address these issues, we propose a deformable multi-scale video desnowing network. Specifically, we design a multi-scale pseudo-3D residual block(MSRB-P3D) that can effectively remove snowflakes of different scales. Furthermore, a deformable large kernel attention 3Dblock(D-LKA 3Dblock) is introduced to capture the inter-frame dynamic information and reduce the artifacts. Due to the scarcity of dataset, we proposed a new dataset named Synthetic and Real Snowy Video Dataset(SRSVD). Extensive experiments have proven that our proposed method not only outperforms other state-of-the-art methods on both synthetic and real snowy videos, but also effectively improves performance on subsequent vision task.

G. Zhou—This work was supported by National Natural Science Foundation of China (No.62166040, No.62261053) and Tianshan Talent Training Project - Xinjiang Science and Technology Innovation Team Program (2023TSYCTD0012).

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

References

  1. Bossu, J., Hautiere, N., Tarel, J.P.: Rain or snow detection in image sequences through use of a histogram of orientation of streaks. Int. J. Comput. Vision 93, 348–367 (2011)

    Article  Google Scholar 

  2. Kim, J.H., Sim, J.Y., Kim, C.S.: Video deraining and desnowing using temporal correlation and low-rank matrix completion. IEEE Trans. Image Process. 24(9), 2658–2670 (2015)

    Article  MathSciNet  Google Scholar 

  3. Li, M., et al.: Video rain streak removal by multiscale convolutional sparse coding. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 6644–6653 (2018)

    Google Scholar 

  4. Li, M., Cao, X., Zhao, Q., Zhang, L., Meng, D.: Online rain/snow removal from surveillance videos. IEEE Trans. Image Process. 30, 2029–2044 (2021)

    Article  Google Scholar 

  5. Ren, W., Tian, J., Han, Z., Chan, A., Tang, Y.: Video desnowing and deraining based on matrix decomposition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4210–4219 (2017)

    Google Scholar 

  6. Yue, Z., Xie, J., Zhao, Q., Meng, D.: Semi-supervised video deraining with dynamical rain generator. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 642–652 (2021)

    Google Scholar 

  7. Zhang, K., Li, D., Luo, W., Ren, W., Liu, W.: Enhanced spatio-temporal interaction learning for video deraining: faster and better. IEEE Trans. Pattern Anal. Mach. Intell. 45(1), 1287–1293 (2022)

    Article  Google Scholar 

  8. Chen, H., et al.: Snow removal in video: a new dataset and a novel method. In: 2023 IEEE/CVF International Conference on Computer Vision (ICCV), pp. 13165–13176. IEEE (2023)

    Google Scholar 

  9. Liu, Y.F., Jaw, D.W., Huang, S.C., Hwang, J.N.: DesnowNet: context-aware deep network for snow removal. IEEE Trans. Image Process. 27(6), 3064–3073 (2018)

    Article  MathSciNet  Google Scholar 

  10. Chen, W.T., et al.: All snow removed: single image desnowing algorithm using hierarchical dual-tree complex wavelet representation and contradict channel loss. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 4196–4205 (2021)

    Google Scholar 

  11. Zhang, K., Li, R., Yu, Y., Luo, W., Li, C.: Deep dense multi-scale network for snow removal using semantic and depth priors. IEEE Trans. Image Process. 30, 7419–7431 (2021)

    Article  Google Scholar 

  12. Chen, X., et al.: Unpaired deep image deraining using dual contrastive learning. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2017–2026 (2022)

    Google Scholar 

  13. Chen, W.-T., Fang, H.-Y., Ding, J.-J., Tsai, C.-C., Kuo, S.-Y.: JSTASR: joint size and transparency-aware snow removal algorithm based on modified partial convolution and veiling effect removal. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, J.-M. (eds.) ECCV 2020. LNCS, vol. 12366, pp. 754–770. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-58589-1_45

    Chapter  Google Scholar 

  14. Yang, W., Tan, R.T., Wang, S., Liu, J.: Self-learning video rain streak removal: when cyclic consistency meets temporal correspondence. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1720–1729 (2020)

    Google Scholar 

  15. Wang, S., Zhu, L., Fu, H., Qin, J., Schönlieb, C.B., Feng, W., Wang, S.: Rethinking video rain streak removal: a new synthesis model and a deraining network with video rain prior. In: European Conference on Computer Vision, pp. 565–582. Springer (2022)

    Google Scholar 

  16. Yang, Y., Aviles-Rivero, A.I., Fu, H., Liu, Y., Wang, W., Zhu, L.: Video adverse-weather-component suppression network via weather messenger and adversarial backpropagation. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 13200–13210 (2023)

    Google Scholar 

  17. Xue, T., Zhou, G., He, R., Wang, Z., Chen, J., Jia, Z.: RVDNet: a two-stage network for real-world video desnowing with domain adaptation. In: ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 3305–3309. IEEE (2024)

    Google Scholar 

  18. Christiansen, M.: Adobe after effects CC visual effects and compositing studio techniques. Adobe Press (2013)

    Google Scholar 

  19. Azad, R., et al.: Beyond self-attention: deformable large kernel attention for medical image segmentation. In: Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 1287–1297 (2024)

    Google Scholar 

  20. Tong, Z., Song, Y., Wang, J., Wang, L.: VideoMAE: masked autoencoders are data-efficient learners for self-supervised video pre-training. Adv. Neural. Inf. Process. Syst. 35, 10078–10093 (2022)

    Google Scholar 

  21. Ying, X., Wang, L., Wang, Y., Sheng, W., An, W., Guo, Y.: Deformable 3d convolution for video super-resolution. IEEE Signal Process. Lett. 27, 1500–1504 (2020)

    Article  Google Scholar 

  22. Dempster, A.P., Laird, N.M., Rubin, D.B.: Maximum likelihood from incomplete data via the EM algorithm. J. Roy. Stat. Soc.: Ser. B (Methodol.) 39(1), 1–22 (1977)

    Article  MathSciNet  Google Scholar 

  23. Uhlenbeck, G.E., Ornstein, L.S.: On the theory of the Brownian motion. Phys. Rev. 36(5), 823 (1930)

    Article  Google Scholar 

  24. Jonathon Luiten, A.H.: Trackeval (2020). https://github.com/JonathonLuiten/TrackEval

  25. Chan, K.C., Zhou, S., Xu, X., Loy, C.C.: BasicVSR++: improving video super-resolution with enhanced propagation and alignment. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5972–5981 (2022)

    Google Scholar 

  26. Chen, S., Ye, T., Liu, Y., Chen, E., Shi, J., Zhou, J.: SnowFormer: Scale-aware transformer via context interaction for single image desnowing. arXiv preprint arXiv:2208.09703 (2022)

  27. Milan, A., Leal-Taixé, L., Reid, I., Roth, S., Schindler, K.: MOT16: A benchmark for multi-object tracking. arXiv:1603.00831 [cs] (2016). http://arxiv.org/abs/1603.00831, arXiv: 1603.00831

  28. Bernardin, K., Stiefelhagen, R.: Evaluating multiple object tracking performance: the clear mot metrics. EURASIP J. Image Video Process. 2008, 1–10 (2008)

    Article  Google Scholar 

  29. Luiten, J., Osep, A., Dendorfer, P., Torr, P., Geiger, A., Leal-Taixé, L., Leibe, B.: HOTA: a higher order metric for evaluating multi-object tracking. In: International Journal of Computer Vision, pp. 1–31 (2020)

    Google Scholar 

  30. Ristani, E., Solera, F., Zou, R., Cucchiara, R., Tomasi, C.: Performance measures and a data set for multi-target, multi-camera tracking. In: Hua, G., Jégou, H. (eds.) ECCV 2016. LNCS, vol. 9914, pp. 17–35. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-48881-3_2

    Chapter  Google Scholar 

  31. He, K., Zhang, X., Ren, S., Sun, J.: Identity mappings in deep residual networks. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9908, pp. 630–645. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-46493-0_38

    Chapter  Google Scholar 

Download references

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Gang Zhou .

Editor information

Editors and Affiliations

Rights and permissions

Reprints and permissions

Copyright information

© 2025 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

He, R., Zhou, G., Xue, T., Liu, Z., Jia, Z. (2025). Deformable Multi-Scale Network for Snow Removal in Video. In: Antonacopoulos, A., Chaudhuri, S., Chellappa, R., Liu, CL., Bhattacharya, S., Pal, U. (eds) Pattern Recognition. ICPR 2024. Lecture Notes in Computer Science, vol 15332. Springer, Cham. https://doi.org/10.1007/978-3-031-78125-4_20

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-78125-4_20

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-78124-7

  • Online ISBN: 978-3-031-78125-4

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics