Skip to main content
SpringerLink
Log in

Memory Selection Network for Video Propagation

  • Conference paper
  • First Online:
Computer Vision – ECCV 2020 (ECCV 2020)

Part of the book series: Lecture Notes in Computer Science ((LNIP,volume 12360))

Included in the following conference series:

Abstract

Video propagation is a fundamental problem in video processing where guidance frame predictions are propagated to guide predictions of the target frame. Previous research mainly treats the previous adjacent frame as guidance, which, however, could make the propagation vulnerable to occlusion, large motion and inaccurate information in the previous adjacent frame. To tackle this challenge, we propose a memory selection network, which learns to select suitable guidance from all previous frames for effective and robust propagation. Experimental results on video object segmentation and video colorization tasks show that our method consistently improves performance and can robustly handle challenging scenarios in video propagation.

R. Wu and H. Lin—Equal Contribution.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 84.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 109.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    YouTube-VOS online server returns a TEXT file containing the per frame IoU for each submission.

References

  1. Bao, L., Wu, B., Liu, W.: CNN in MRF: video object segmentation via inference in a CNN-based higher-order spatio-temporal MRF. In: CVPR (2018)

    Google Scholar 

  2. Caelles, S., Maninis, K.K., Pont-Tuset, J., Leal-Taixé, L., Cremers, D., Van Gool, L.: One-shot video object segmentation. In: CVPR (2017)

    Google Scholar 

  3. Chai, Y.: Patchwork: a patch-wise attention network for efficient object detection and segmentation in video streams. In: ICCV (2019)

    Google Scholar 

  4. Cheng, J., Tsai, Y.H., Hung, W.C., Wang, S., Yang, M.H.: Fast and accurate online video object segmentation via tracking parts. In: CVPR (2018)

    Google Scholar 

  5. Cheng, J., Tsai, Y.H., Wang, S., Yang, M.H.: Segflow: joint learning for video object segmentation and optical flow. In: ICCV (2017)

    Google Scholar 

  6. Ci, H., Wang, C., Wang, Y.: Video object segmentation by learning location-sensitive embeddings. In: ECCV (2018)

    Google Scholar 

  7. Duarte, K., Rawat, Y.S., Shah, M.: Capsulevos: semi-supervised video object segmentation using capsule routing. In: ICCV (2019)

    Google Scholar 

  8. Farnebäck, G.: Two-frame motion estimation based on polynomial expansion. In: Scandinavian conference on Image analysis (2003)

    Google Scholar 

  9. Griffin, B.A., Corso, J.J.: Bubblenets: learning to select the guidance frame in video object segmentation by deep sorting frames. In: CVPR (2019)

    Google Scholar 

  10. He, K., Sun, J., Tang, X.: Guided image filtering. TPAMI (2013)

    Google Scholar 

  11. Hu, Y.T., Huang, J.B., Schwing, A.: Maskrnn: instance level video object segmentation. In: NeurIPS (2017)

    Google Scholar 

  12. Hu, Y.T., Huang, J.B., Schwing, A.G.: Videomatch: matching based video object segmentation. In: ECCV (2018)

    Google Scholar 

  13. Ilg, E., Mayer, N., Saikia, T., Keuper, M., Dosovitskiy, A., Brox, T.: Flownet 2.0: evolution of optical flow estimation with deep networks. In: CVPR (2017)

    Google Scholar 

  14. Jampani, V., Gadde, R., Gehler, P.V.: Video propagation networks. In: CVPR (2017)

    Google Scholar 

  15. Johnander, J., Danelljan, M., Brissman, E., Khan, F.S., Felsberg, M.: A generative appearance model for end-to-end video object segmentation. In: CVPR (2019)

    Google Scholar 

  16. Khoreva, A., Benenson, R., Ilg, E., Brox, T., Schiele, B.: Lucid data dreaming for multiple object tracking. arXiv:1703.09554 (2017)

  17. Kingma, D.P., Ba, J.: Adam: a method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014)

  18. Levin, A., Lischinski, D., Weiss, Y.: Colorization using optimization. In: TOG (2004)

    Google Scholar 

  19. Levin, A., Lischinski, D., Weiss, Y.: A closed-form solution to natural image matting. TPAMI (2008)

    Google Scholar 

  20. Li, X., et al.: Video object segmentation with re-identification. In: The 2017 DAVIS Challenge on Video Object Segmentation-CVPR Workshops (2017)

    Google Scholar 

  21. Li, X., Change Loy, C.: Video object segmentation with joint re-identification and attention-aware mask propagation. In: ECCV (2018)

    Google Scholar 

  22. Li, Z., Chen, Q., Koltun, V.: Interactive image segmentation with latent diversity. In: CVPR (2018)

    Google Scholar 

  23. Liu, S., De Mello, S., Gu, J., Zhong, G., Yang, M.H., Kautz, J.: Learning affinity via spatial propagation networks. In: NeurIPS (2017)

    Google Scholar 

  24. Liu, S., Zhong, G., De Mello, S., Gu, J., Yang, M.H., Kautz, J.: Switchable temporal propagation network. arXiv:1804.08758 (2018)

  25. Luiten, J., Voigtlaender, P., Leibe, B.: Premvos: proposal-generation, refinement and merging for video object segmentation. In: ACCV (2018)

    Google Scholar 

  26. Maninis, K.K., et al.: Video object segmentation without temporal information. TPAMI 41, 1515–1530 (2018)

    Article  Google Scholar 

  27. Miksik, O., Munoz, D., Bagnell, J.A., Hebert, M.: Efficient temporal consistency for streaming video scene analysis. In: ICRA (2013)

    Google Scholar 

  28. Oh, S.W., Lee, J.Y., Xu, N., Kim, S.J.: Video object segmentation using space-time memory networks. In: ICCV (2019)

    Google Scholar 

  29. Perazzi, F., Khoreva, A., Benenson, R., Schiele, B., Sorkine-Hornung, A.: Learning video object segmentation from static images. In: CVPR (2017)

    Google Scholar 

  30. Perazzi, F., Pont-Tuset, J., McWilliams, B., Van Gool, L., Gross, M., Sorkine-Hornung, A.: A benchmark dataset and evaluation methodology for video object segmentation. In: CVPR (2016)

    Google Scholar 

  31. Pont-Tuset, J., Perazzi, F., Caelles, S., Arbeláez, P., Sorkine-Hornung, A., Van Gool, L.: The 2017 davis challenge on video object segmentation. arXiv:1704.00675 (2017)

  32. Rick Chang, J.H., Frank Wang, Y.C.: Propagated image filtering. In: CVPR (2015)

    Google Scholar 

  33. Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. arXiv:1409.1556 (2014)

  34. Tokmakov, P., Alahari, K., Schmid, C.: Learning video object segmentation with visual memory. In: ICCV (2017)

    Google Scholar 

  35. Ventura, C., Bellver, M., Girbau, A., Salvador, A., Marques, F., Giro-i Nieto, X.: Rvos: end-to-end recurrent network for video object segmentation. In: CVPR (2019)

    Google Scholar 

  36. Voigtlaender, P., Chai, Y., Schroff, F., Adam, H., Leibe, B., Chen, L.C.: Feelvos: fast end-to-end embedding learning for video object segmentation. In: CVPR (2019)

    Google Scholar 

  37. Voigtlaender, P., Leibe, B.: Online adaptation of convolutional neural networks for video object segmentation. arXiv:1706.09364 (2017)

  38. Vondrick, C., Shrivastava, A., Fathi, A., Guadarrama, S., Murphy, K.: Tracking emerges by colorizing videos. In: ECCV (2018)

    Google Scholar 

  39. Wug Oh, S., Lee, J.Y., Sunkavalli, K., Joo Kim, S.: Fast video object segmentation by reference-guided mask propagation. In: CVPR (2018)

    Google Scholar 

  40. Xu, K., Wen, L., Li, G., Bo, L., Huang, Q.: Spatiotemporal CNN for video object segmentation. In: CVPR (2019)

    Google Scholar 

  41. Xu, N., et al.: Youtube-VOS: Sequence-to-sequence video object segmentation. In: ECCV (2018)

    Google Scholar 

  42. Xu, N., et al.: Youtube-VOS: a large-scale video object segmentation benchmark. arXiv:1809.03327 (2018)

  43. Yang, L., Wang, Y., Xiong, X., Yang, J., Katsaggelos, A.K.: Efficient video object segmentation via network modulation. In: CVPR (2018)

    Google Scholar 

  44. Yatziv, L., Sapiro, G.: Fast image and video colorization using chrominance blending. TIP 15, 1120–1129 (2006)

    Google Scholar 

  45. Zeng, X., Liao, R., Gu, L., Xiong, Y., Fidler, S., Urtasun, R.: DMM-net: Differentiable mask-matching network for video object segmentation. In: ICCV (2019)

    Google Scholar 

  46. Zhang, R., Isola, P., Efros, A.A.: Colorful image colorization. In: ECCV (2016)

    Google Scholar 

  47. Zhang, R., et al.: Real-time user-guided image colorization with learned deep priors. TOG (2017)

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. The Chinese University of Hong Kong, Hong Kong, China

    Ruizheng Wu, Huaijia Lin & Jiaya Jia

  2. University of Hong Kong, Hong Kong, China

    Xiaojuan Qi

  3. SmartMore, Shenzhen, China

    Jiaya Jia

Authors
  1. Ruizheng Wu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Huaijia Lin
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Xiaojuan Qi
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Jiaya Jia
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Ruizheng Wu .

Editor information

Editors and Affiliations

  1. University of Oxford, Oxford, UK

    Andrea Vedaldi

  2. Graz University of Technology, Graz, Austria

    Horst Bischof

  3. University of Freiburg, Freiburg im Breisgau, Germany

    Thomas Brox

  4. University of North Carolina at Chapel Hill, Chapel Hill, NC, USA

    Jan-Michael Frahm

1 Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary material 1 (pdf 654 KB)

Rights and permissions

Reprints and permissions

Copyright information

© 2020 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Wu, R., Lin, H., Qi, X., Jia, J. (2020). Memory Selection Network for Video Propagation. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, JM. (eds) Computer Vision – ECCV 2020. ECCV 2020. Lecture Notes in Computer Science(), vol 12360. Springer, Cham. https://doi.org/10.1007/978-3-030-58555-6_11

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-58555-6_11

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-58554-9

  • Online ISBN: 978-3-030-58555-6

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation