Skip to main content
SpringerLink
Log in

A CNNs-based method for optical flow estimation with prior constraints and stacked U-Nets

  • Original Article
  • Published:
Neural Computing and Applications Aims and scope Submit manuscript

Abstract

Traditional approaches for optical flow estimation always build an energy function which contains data term and smoothness term. However, optimizing the complex function is usually time-consuming. Nowadays, convolution neural networks have been applied in optical flow area. Most of them use large dataset for learning optical flow end-to-end, which can learn motion information from a large amount of prior information prepared in advance. However, these methods rely excessively on the learning ability of the network while ignoring some of well-proven assumptions in traditional approaches. In this paper, inspired by traditional methods, we present a network for learning optical flow, which combines traditional constraints with a supervised network. In the process of network optimization, the brightness constancy, gradient constancy and spatial smoothness assumptions are used to guide the training of network. Moreover, we stack several sub-networks integrated with prior constraints to form a large network for iterative refinement. Our method is tested on several public datasets, such as MPI-Sintel, KITTI2012, KITTI2015, Middlebury. The experimental results show that adding the prior constraints during training can obtain more refined and accurate flow. Compared with other recent methods, our method can achieve state-of-the-art performance on several public benchmarks.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10

Similar content being viewed by others

References

  1. Ahmadi A, Patras I (2016) Unsupervised convolutional neural networks for motion estimation. In: 2016 IEEE international conference on image processing (ICIP), pp 1629–1633

  2. Banerjee B, Murino V (2017) Efficient pooling of image based CNN features for action recognition in videos. In: 2017 IEEE international conference on acoustics, speech and signal processing (ICASSP), pp 2637–2641

  3. Bao L, Yang Q, Jin H (2014) Fast edge-preserving patchmatch for large displacement optical flow. In: 2014 IEEE conference on computer vision and pattern recognition, pp 3534–3541. https://doi.org/10.1109/CVPR.2014.452

  4. Bao L, Yang Q, Jin H (2014) Fast edge-preserving patchmatch for large displacement optical flow. In: 2014 IEEE conference on computer vision and pattern recognition, pp 3534–3541

  5. Brox T, Bruhn A, Papenberg N, Weickert J (2004) High accuracy optical flow estimation based on a theory for warping. In: European conference on computer vision (ECCV), Lecture Notes in Computer Science, vol. 3024. Springer, pp 25–36

  6. Brox T, Malik J (2011) Large displacement optical flow: descriptor matching in variational motion estimation. IEEE Trans Pattern Anal Mach Intell 33(3):500–513. https://doi.org/10.1109/TPAMI.2010.143

    Article  Google Scholar 

  7. Chen LC, Papandreou G, Kokkinos I, Murphy K, Yuille AL (2018) Deeplab: semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected CRFS. IEEE Trans Pattern Anal Mach Intell 40(4):834–848. https://doi.org/10.1109/TPAMI.2017.2699184

    Article  Google Scholar 

  8. Chen Z, Jin H, Lin Z, Cohen S, Wu Y (2013) Large displacement optical flow from nearest neighbor fields. In: 2013 IEEE conference on computer vision and pattern recognition, pp 2443–2450. https://doi.org/10.1109/CVPR.2013.316

  9. Cheng J, Tsai YH, Wang S, Yang MH (2017) Segflow: joint learning for video object segmentation and optical flow. In: 2017 IEEE international conference on computer vision (ICCV), pp 686–695

  10. Dosovitskiy A, Fischery P, Ilg E, Husser P, Hazirbas C, Golkov V, van der Smagt P, Cremers D, Brox T (2015) Flownet: learning optical flow with convolutional networks. In: 2015 IEEE international conference on computer vision (ICCV), pp 2758–2766

  11. Gao Z, Wang L, Zhou L, Zhang J (2017) Hep-2 cell image classification with deep convolutional neural networks. IEEE J Biomed Health Inform 21(2):416–428

    Article  Google Scholar 

  12. Horn BK, Schunck BG (1981) Determining optical flow. Artif Intell 17(1):185–203

    Article  Google Scholar 

  13. Huang G, Liu Z, van der Maaten L, Weinberger KQ (2017) Densely connected convolutional networks. In: 2017 IEEE conference on computer vision and pattern recognition (CVPR), pp 2261–2269. https://doi.org/10.1109/CVPR.2017.243

  14. Ilg E, Mayer N, Saikia T, Keuper M, Dosovitskiy A, Brox T (2017) Flownet 2.0: evolution of optical flow estimation with deep networks. In: 2017 IEEE conference on computer vision and pattern recognition (CVPR), pp 1647–1655. https://doi.org/10.1109/CVPR.2017.179

  15. Jaderberg M, Simonyan K, Zisserman A, Kavukcuoglu K (2015) Spatial transformer networks. In: Cortes C, Lawrence ND, Lee DD, Sugiyama M, Garnett R (eds) Advances in neural information processing systems, vol 28. Curran Associates, Inc, Red Hook, pp 2017–2025

    Google Scholar 

  16. Kingma D, Ba J (2015) Adam: a method for stochastic optimization. In: International conference on learning representations, pp 1–13

  17. Li Y, Huttenlocher DP (2008) Learning for optical flow using stochastic optimization. In: Forsyth D, Torr P, Zisserman A (eds) Computer Vision - ECCV 2008. Springer, Berlin, Heidelberg, pp 379–391

    Chapter  Google Scholar 

  18. Liu W, Anguelov D, Erhan D, Szegedy C, Reed S, Fu CY, Berg AC (2016) Ssd: single shot multibox detector. In: Leibe B, Matas J, Sebe N, Welling M (eds) Computer Vision - ECCV 2016. Springer International Publishing, Cham, pp 21–37

    Chapter  Google Scholar 

  19. Lopez AM, Moreno-Noguer F, Sanfeliu A (2017) Joint coarse-and-fine reasoning for deep optical flow. In: 2017 IEEE international conference on image processing (ICIP)

  20. Marban A, Srinivasan V, Samek W, Fernndez J, Casals A (2017) Estimating position velocity in 3d space from monocular video sequences using a deep neural network. In: 2017 IEEE international conference on computer vision workshops (ICCVW), pp 1460–1469. https://doi.org/10.1109/ICCVW.2017.173

  21. Mayer N, Ilg E, Husser P, Fischer P, Cremers D, Dosovitskiy A, Brox T (2016) A large dataset to train convolutional networks for disparity, optical flow, and scene flow estimation. In: 2016 IEEE conference on computer vision and pattern recognition (CVPR), pp. 4040–4048. https://doi.org/10.1109/CVPR.2016.438

  22. Ranjan A, Black MJ (2017) Optical flow estimation using a spatial pyramid network. In: 2017 IEEE conference on computer vision and pattern recognition (CVPR), pp 2720–2729

  23. Redmon J, Farhadi A (2017) Yolo9000: better, faster, stronger. In: 2017 IEEE conference on computer vision and pattern recognition (CVPR), pp 6517–6525. https://doi.org/10.1109/CVPR.2017.690

  24. Revaud J, Weinzaepfel P, Harchaoui Z, Schmid C (2015) Epicflow: edge-preserving interpolation of correspondences for optical flow. In: 2015 IEEE conference on computer vision and pattern recognition (CVPR), pp 1164–1172. https://doi.org/10.1109/CVPR.2015.7298720

  25. Rosenbaum D, Zoran D, Weiss Y (2013) Learning the local statistics of optical flow

  26. Sun D, Roth S, Black MJ (2010) Secrets of optical flow estimation and their principles. In: 2010 IEEE computer society conference on computer vision and pattern recognition, pp. 2432–2439. https://doi.org/10.1109/CVPR.2010.5539939

  27. Sun D, Roth S, Lewis JP, Black MJ (2008) Learning optical flow. In: Proceedings of the 10th European conference on computer vision: Part III, ECCV ’08. Springer, Berlin, Heidelberg, pp 83–97

  28. Teney D, Hebert M (2017) Learning to extract motion from videos in convolutional neural networks. In: Lai SH, Lepetit V, Nishino K, Sato Y (eds) Computer Vision - ACCV 2016. Springer International Publishing, Cham, pp 412–428

    Chapter  Google Scholar 

  29. Weinzaepfel P, Revaud J, Harchaoui Z, Schmid C (2013) Deepflow: large displacement optical flow with deep matching. In: 2013 IEEE international conference on computer vision, pp 1385–1392

  30. Wulff J, Black MJ (2015) Efficient sparse-to-dense optical flow estimation using a learned basis and layers. In: 2015 IEEE conference on computer vision and pattern recognition (CVPR), pp 120–130

  31. Yu JJ, Harley AW, Derpanis KG (2016) Back to basics: unsupervised learning of optical flow via brightness constancy and motion smoothness. Springer International Publishing, Cham, pp 3–10

    Google Scholar 

  32. Zhang H, Cao X, Ho JKL, Chow TWS (2017) Object-level video advertising: an optimization framework. IEEE Trans Ind Inform 13(2):520–531

    Article  Google Scholar 

  33. Zhang H, Ji Y, Huang W, Liu L (2018) Sitcom-star-based clothing retrieval for video advertising: a deep learning framework. Neural Comput Appl. https://doi.org/10.1007/s00521-018-3579-x

    Article  Google Scholar 

  34. Zhu Y, Newsam S (2017) Densenet for dense flow. In: 2017 IEEE international conference on image processing (ICIP)

Download references

Acknowledgements

This work was supported in part by the National Natural Science Foundation of China under Grant (61401113), in part by the Natural Science Foundation of Heilongjiang Province of China under Grant (LC201426), and in part by the Fundamental Research Funds for the Central Universities of China under Grant (HEUCF180801).

Author information

Authors and Affiliations

  1. School of Information and Communication Engineering, Harbin Engineering University, Harbin, 150001, China

    Xuezhi Xiang, Mingliang Zhai, Rongfang Zhang & Yulong Qiao

  2. School of Electrical Engineering and Computer Science, University of Ottawa, Ottawa, ON K1N 6N5, Canada

    Abdulmotaleb El Saddik

Authors
  1. Xuezhi Xiang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Mingliang Zhai
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Rongfang Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Yulong Qiao
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Abdulmotaleb El Saddik
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Xuezhi Xiang.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Xiang, X., Zhai, M., Zhang, R. et al. A CNNs-based method for optical flow estimation with prior constraints and stacked U-Nets. Neural Comput & Applic 32, 4675–4688 (2020). https://doi.org/10.1007/s00521-018-3816-3

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00521-018-3816-3

Keywords

Search

Navigation