Skip to main content
SpringerLink
Log in

Unsupervised Optical Flow Estimation Based on Improved Feature Pyramid

  • Published:
Neural Processing Letters Aims and scope Submit manuscript

Abstract

Deep learning methods for optical flow estimation usually increase the receptive field of convolution through reducing image resolution, which results in loss of spatial detail information during feature extraction. In this paper, we introduce dilated convolution into feature pyramid network, which can extract multi-scale features containing more motion details and can further improve the accuracy of optical flow estimation. The unsupervised loss function is based on forward–backward consistency check and robust census transform that has a good constraint performance in the case of illumination changes, which can train an unsupervised learning optical flow model with higher accuracy. Our network is trained on FlyingChairs and KITTI raw datasets with an unsupervised manner and tested on MPI-Sintel, KITTI 2012 and KITTI 2015 benchmarks. The experimental results show the advantages of our method in unsupervised learning approaches.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8

Similar content being viewed by others

References

  1. Horn B, Schunck B (1981) Determining optical flow. Artif Intell 17:185–203

    Article  Google Scholar 

  2. Dakhia A, Wang T, Lu H (2019) Multi-scale pyramid pooling network for salient object detection. Neurocomputing 333:211–220

    Article  Google Scholar 

  3. Xu W, Liu W, Huang X, Yang J, Qiu S (2018) Multi-modal self-paced learning for image classification. Neurocomputing 309:134–144

    Article  Google Scholar 

  4. Wang F, Ainouz S, Lian C, Bensrhair A (2017) Multimodality semantic segmentation based on polarization and color images. Neurocomputing 253:1–9

    Article  Google Scholar 

  5. Dosovitskiy A, Fischer P, Ilg E, Häusser P, Hazirbas C, Golkov V, Smagt P. v. d, Cremers D, Brox T (2015) FlowNet: learning optical flow with convolutional networks. In: 2015 IEEE international conference on computer vision (ICCV), pp 2758–2766

  6. Ilg E, Mayer N, Saikia T, Keuper M, Dosovitskiy A, Brox T (2017) Flownet 2.0: evolution of optical flow estimation with deep networks. In: 2017 IEEE conference on computer vision and pattern recognition (CVPR), pp 1647–1655

  7. Ranjan A, Black MJ (2017) Optical flow estimation using a spatial pyramid network. In: 2017 IEEE conference on computer vision and pattern recognition (CVPR), pp 2720–2729

  8. Sun D, Yang X, Liu M, Kautz J (2018) PWC-net: CNNs for optical flow using pyramid, warping, and cost volume. In: 2018 IEEE/CVF conference on computer vision and pattern recognition (CVPR), pp 8934–8943

  9. Ahmadi A, Patras I (2016) Unsupervised convolutional neural networks for motion estimation. In: 2016 IEEE international conference on image processing (ICIP), pp 1629–1633

  10. Yu J, Harley A, Derpanis K (2016) Back to basics: unsupervised learning of optical flow via brightness constancy and motion smoothness. In: European conference on computer visio, pp 3–10

  11. Zhu Y, Newsam S (2017) Densenet for dense flow. In: 2017 IEEE international conference on image processing (ICIP), pp 790–794

  12. Huang G, Liu Z, van der Maaten L (2017) Weinberger KQ, Densely connected convolutional networks. In: 2017 IEEE conference on computer vision and pattern recognition (CVPR), pp 2261–2269

  13. Ren Z, Yan J, Ni B, Liu B, Yang X, Zha H (2017) Unsupervised deep learning for optical flow estimation. In: AAAI conference on artificial intelligence, San Francisco, California

  14. Meister S, Hur J, Roth S (2018) UnFlow: unsupervised learning of optical flow with a bidirectional census loss. In: AAAI conference on artificial intelligence, New Orleans, Louisiana

  15. Wang Y, Yang Y, Yang Z, Zhao L, Wang P, Xu W (2018) Occlusion aware unsupervised learning of optical flow. In: 2018 IEEE/CVF conference on computer vision and pattern recognition, pp 4884–4893

  16. Liu P, King I, Lyu M R, Xu J (2019) DDFlow: learning optical flow with unlabeled data distillation. In: The thirty-third AAAI conference on artificial intelligence, pp 8770–8777

  17. Hafner D, Demetz O, Weickert J (2013) Why is the census transform good for robust optic flow computation? In: Scale space and variational methods in computer vision: 4th international conference, pp 210–221

  18. Tretiak O, Pastor L (1984) Velocity estimation from image sequences with second order differential operators. In: Proceedings of IEEE international conference on pattern recognition, pp 16–19

  19. Brox T, Bruhn A, Papenberg N, Weickert J (2004) High accuracy optical flow estimation based on a theory for warping. In: Conference paper in lecture notes in computer science, pp 25–36

  20. Bruhn A, Weickert J, Schnorr C (2005) Lucas/Kanade meets Horn/Schunck: combining local and global optic flow methods. Int J Comput Vis 61(3):211–231

    Article  Google Scholar 

  21. Papenberg N, Bruhn A, Brox T, Didas S, Weickert J (2006) Highly accurate optic flow computation with theoretically justified warping. Int J Comput Vis 67(2):141–158

    Article  Google Scholar 

  22. Brox T, Malik J (2011) Large displacement optical flow: descriptor matching in variational motion estimation. IEEE Trans Pattern Anal Mach Intell 33(3):500–513

    Article  Google Scholar 

  23. Weinzaepfel P, Revaud J, Harchaoui Z, Schmid C (2013) Deepflow: large displacement optical flow with deep matching. In: 2013 IEEE international conference on computer vision, pp 1385–1392

  24. Revaud J, Weinzaepfel P, Harchaoui Z, Schmid C (2015) EpicFlow: edge-preserving interpolation of correspondences for optical flow. In: 2015 IEEE conference on computer vision and pattern recognition (CVPR), pp 1164–1172

  25. Nagel H, Enkelmann W (1986) An investigation of smoothness constraints for the estimation of displacement vector fields from image sequences. In: IEEE transactions on pattern analysis and machine intelligence PAMI-8(5), pp 565–593

  26. Sun D, Roth S, Black MJ (2010) Secrets of optical flow estimation and their principles. In: 2010 IEEE computer society conference on computer vision and pattern recognition, pp 2432–2439

  27. Hua M, Bie X, Zhang M, Wang W (2014) Edge-aware gradient domain optimization framework for image filtering by local propagation. In: 2014 IEEE conference on computer vision and pattern recognition, pp 2838–2845

  28. Cheng J, Tsai Y, Wang S, Yang M (2017) Segflow: joint learning for video object segmentation and optical flow. In: 2017 IEEE international conference on computer vision (ICCV), pp 686–695

  29. Gadot D, Wolf L (2016) PatchBatch: a batch augmented loss for optical flow. In: 2016 IEEE conference on computer vision and pattern recognition (CVPR), pp 4236–4245

  30. Guney F, Geiger A, Lai SH, Lepetit V, Nishino K, Sato Y (2017) Deep discrete flow, computer vision—ACCV 2016. Springer, Berlin, pp 207–224

    Book  Google Scholar 

  31. Dai J, Huang S, Nguyen T (2018) Pyramid structured optical flow learning with motion cues. In: 2018 25th IEEE international conference on image processing (ICIP), pp 3338–3342

  32. Hu P, Wang G, Tan Y (2018) Recurrent spatial pyramid cnn for optical flow estimation. IEEE Trans Multimedia 20(10):2814–2823

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. The Electric Power Research Institute of State Grid Beijing Electric Power Company, Beijing, China

    Bo Yang, Huan Xie, Zhigang Ren, Kuan Ye & Rong Zhu

  2. State Grid Beijing Electric Power Company, Beijing, China

    Hongbin Li & Anchang Liu

  3. Beijing Jindian United Power Supply Consulting Co., Ltd., Beijing, China

    Nuohan Li

  4. School of Information and Communication Engineering, Harbin Engineering University, Harbin, 150001, China

    Xuezhi Xiang

Authors
  1. Bo Yang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Huan Xie
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Hongbin Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Nuohan Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Anchang Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Zhigang Ren
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Kuan Ye
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. Rong Zhu
    View author publications

    You can also search for this author in PubMed Google Scholar

  9. Xuezhi Xiang
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Xuezhi Xiang.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

This work was supported in part by the State Grid Corporation Science and Technology Foundation under Grant 52022318001N, in part by the National Natural Science Foundation of China under Grant 61401113, and in part by the Natural Science Foundation of Heilongjiang Province of China under Grant LC201426.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Yang, B., Xie, H., Li, H. et al. Unsupervised Optical Flow Estimation Based on Improved Feature Pyramid. Neural Process Lett 52, 1601–1612 (2020). https://doi.org/10.1007/s11063-020-10328-2

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11063-020-10328-2

Keywords

Search

Navigation