Skip to main content
SpringerLink
Log in

Learning to estimate optical flow using dual-frequency paradigm

  • Regular research paper
  • Published:
Memetic Computing Aims and scope Submit manuscript

Abstract

Deep learning-based optical flow estimation achieved impressive success with faster inference time and outperformed performance. Optical flow estimation networks are usually treated as a black box relying on large amounts of synthetic data for training, therefore the generalization and robustness of the network applying in realities remains a challenge. To overcome these problems, a dual-frequency paradigm is proposed for optical flow estimation. The proposed dual-frequency encoder captures discriminative features with both high-frequency and low-frequency biases. It is experimentally demonstrated that our method achieves better generalization while only pre-trained on FlyingChiars. Furthermore, our method improves the prediction of optical flow in occluded regions by enhancing the perception of high-frequency features that further improve the robustness of the network. Compared to the start-of-the-art RAFT, our approach obtains an improvement of the average end-point error by 10.6% on the Sintel Clean datasets and 11.7% on the challenging Sintel Final dataset.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11

Similar content being viewed by others

References

  1. Dosovitskiy A, Fischer P, Ilg E, Hausser P, Hazirbas C, Golkov V, Van Der Smagt P, Cremers D, Brox T (2015) Flownet: learning optical flow with convolutional networks. In: Proceedings of the IEEE international conference on computer vision, pp 2758–2766

  2. Tu Z, Xie W, Zhang D, Poppe R, Veltkamp RC, Li B, Yuan J (2019) A survey of variational and CNN-based optical flow techniques. Image Commun 72(C):9–24. https://doi.org/10.1016/j.image.2018.12.002

    Article  Google Scholar 

  3. Menze M, Geiger A (2015) Object scene flow for autonomous vehicles. In: 2015 IEEE conference on computer vision and pattern recognition (CVPR), pp 3061–3070. https://doi.org/10.1109/CVPR.2015.7298925

  4. Rahaman N, Baratin A, Arpit D, Draxler F, Lin M, Hamprecht F, Bengio Y, Courville A (2019) On the spectral bias of neural networks. In: International conference on machine learning, pp 5301–5310. PMLR

  5. Xu Z-QJ, Zhang Y, Luo T, Xiao Y, Zheng M (2020) Frequency principle: Fourier analysis sheds light on deep neural networks. Commun. Comput. Phys. 28(5):1746–1767. https://doi.org/10.4208/cicp.OA-2020-0085

    Article  MathSciNet  MATH  Google Scholar 

  6. Basri R, Galun M, Geifman A, Jacobs D, Kasten Y, Kritchman S (2020) Frequency bias in neural networks for input of non-uniform density. In: International conference on machine learning, pp 685–694. PMLR

  7. Wang H, Wu X, Huang Z, Xing EP (2020) High-frequency component helps explain the generalization of convolutional neural networks. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 8684–8694

  8. Sweldens W (1998) The lifting scheme: a construction of second generation wavelets. SIAM J Math Anal 29(2):511–546

    Article  MathSciNet  MATH  Google Scholar 

  9. Chui CK (1992) Wavelets: a tutorial in theory and applications. Academic Press, Cambridge

    MATH  Google Scholar 

  10. Ilg E, Mayer N, Saikia T, Keuper M, Dosovitskiy A, Brox T (2017) Flownet 2.0: evolution of optical flow estimation with deep networks. In: 2017 IEEE conference on computer vision and pattern recognition (CVPR), pp 1647–1655. https://doi.org/10.1109/CVPR.2017.179

  11. Ranjan A, Black MJ (2017) Optical flow estimation using a spatial pyramid network. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 4161–4170

  12. Hui T-W, Tang X, Loy CC (2018) Liteflownet: a lightweight convolutional neural network for optical flow estimation. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 8981–8989

  13. Sun D, Yang X, Liu M-Y, Kautz J (2018) PWC-net: CNNs for optical flow using pyramid, warping, and cost volume. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 8934–8943

  14. Yang G, Ramanan D (2019) Volumetric correspondence networks for optical flow. In: Advances in neural information processing systems 32

  15. Hur J, Roth S (2019) Iterative residual refinement for joint optical flow and occlusion estimation. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 5754–5763

  16. Zheng Y, Zhang M, Lu F (2020) Optical flow in the dark. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 6749–6757

  17. Yan W, Sharma A, Tan RT (2020) Optical flow in dense foggy scenes using semi-supervised learning. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 13259–13268

  18. Zhang Y, Jin X, Wang Z (2017) A new modified panoramic UAV image stitching model based on the GA-sift and adaptive threshold method. Memet Comput 9(3):231–244

    Article  Google Scholar 

  19. WangPing Z, Min J, JunFeng Y, KunHong L, QingQiang W (2022) The design of evolutionary feature selection operator for the micro-expression recognition. Memet Comput 14(1):61–76

    Article  Google Scholar 

  20. Teed Z, Deng J (2020) Raft: eecurrent all-pairs field transforms for optical flow. In: European conference on computer vision. Springer, Berlin, pp 402–419

  21. Jiang S, Campbell D, Lu Y, Li H, Hartley R (2021) Learning to estimate hidden motions with global motion aggregation. In: Proceedings of the IEEE/CVF international conference on computer vision, pp 9772–9781

  22. Bai S, Geng Z, Savani Y, Kolter JZ (2022) Deep equilibrium optical flow estimation. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 620–630

  23. Luo A, Yang F, Li X, Liu S (2022) Learning optical flow with kernel patch attention. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 8906–8915

  24. Zhang F, Woodford OJ, Prisacariu VA, Torr PH (2021) Separable flow: Learning motion cost volumes for optical flow estimation. In: Proceedings of the IEEE/CVF international conference on computer vision, pp 10807–10817

  25. Zhao S, Zhao L, Zhang Z, Zhou E, Metaxas D (2022) Global matching with overlapping attention for optical flow estimation. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 17592–17601

  26. Peebles W, Zhu J-Y, Zhang R, Torralba A, Efros A, Shechtman E (2022) Gan-supervised dense visual alignment. In: CVPR

  27. Li Y, Barnes C, Huang K, Zhang F-L (2022) Deep \(360^{\circ }\) optical flow estimation based on multi-projection fusion. In: Proceedings of the European conference on computer vision (ECCV) 2022, pp 336–352. https://doi.org/10.1007/978-3-031-19833-5_20

  28. Huang J, Guan D, Xiao A, Lu S (2021) Rda: Robust domain adaptation via Fourier adversarial attacking. In: Proceedings of the IEEE/CVF international conference on computer vision, pp 8988–8999

  29. Hong D, Wu X, Ghamisi P, Chanussot J, Yokoya N, Zhu XX (2020) Invariant attribute profiles: a spatial-frequency joint feature extractor for hyperspectral image classification. IEEE Trans Geosci Remote Sens 58(6):3791–3808

    Article  Google Scholar 

  30. Liu Y, Li Q, Sun Z (2019) Attribute-aware face aging with wavelet-based generative adversarial networks. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 11877–11886

  31. Chen Y, Fan H, Xu B, Yan Z, Kalantidis Y, Rohrbach M, Shuicheng Y, Feng J (2019) Drop an octave: reducing spatial redundancy in convolutional neural networks with octave convolution. In: 2019 IEEE/CVF international conference on computer vision (ICCV), pp 3434–3443. https://doi.org/10.1109/ICCV.2019.00353

  32. Williams T, Li R (2018) Wavelet pooling for convolutional neural networks. In: International conference on learning representations

  33. Ferra A, Aguilar E, Radeva P (2018) Multiple wavelet pooling for CNNs. In: Proceedings of the European conference on computer vision (ECCV) workshops

  34. Li Q, Shen L, Guo S, Lai Z (2020) Wavelet integrated CNNs for noise-robust image classification. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (CVPR)

  35. Gomez AN, Ren M, Urtasun R, Grosse RB (2017) The reversible residual network: Backpropagation without storing activations. In: Advances in neural information processing systems, 30

  36. Zheng Y, Shi Z, He C, Zhang Q (2020) Lifting based object detection networks of remote sensing imagery for FPGA accelerator. IEEE Access 8:200430–200439. https://doi.org/10.1109/ACCESS.2020.3035839

    Article  Google Scholar 

  37. Vaswani A, Shazeer N, Parmar N, Uszkoreit J, Jones L, Gomez AN, Kaiser L, Polosukhin I (2017) Attention is all you need. In: Proceedings of the 31st international conference on neural information processing systems. NIPS’17. Curran Associates Inc., Red Hook, NY, USA, pp 6000–6010

  38. Claypoole RL, Davis GM, Sweldens W, Baraniuk RG (2003) Nonlinear wavelet transforms for image coding via lifting. IEEE Trans Image Process 12(12):1449–1459

    Article  MathSciNet  Google Scholar 

  39. Zheng Y, Wang R, Li J (2010) Nonlinear wavelets and BP neural networks adaptive lifting scheme. In: The 2010 international conference on apperceiving computing and intelligence analysis proceeding. IEEE, pp 316–319

  40. Mayer N, Ilg E, Häusser P, Fischer P, Cremers D, Dosovitskiy A, Brox T (2016) A large dataset to train convolutional networks for disparity, optical flow, and scene flow estimation. In: 2016 IEEE conference on computer vision and pattern recognition (CVPR), pp 4040–4048. https://doi.org/10.1109/CVPR.2016.438

  41. Butler DJ, Wulff J, Stanley GB, Black MJ (2012) A naturalistic open source movie for optical flow evaluation. In: European conference on computer vision. Springer, Berlin, pp 611–625

  42. Kondermann D, Nair R, Honauer K, Krispin K, Andrulis J, Brock A, Güssefeld B, Rahimimoghaddam M, Hofmann S, Brenner C, Jähne B (2016) The HCI benchmark suite: Stereo and flow ground truth with uncertainties for urban autonomous driving. In: 2016 IEEE conference on computer vision and pattern recognition workshops (CVPRW), pp 19–28. https://doi.org/10.1109/CVPRW.2016.10

Download references

Acknowledgements

This work is funded by the National Key Research and Development Program of China (No. 2016YFC0803000) and the National Natural Science Foundation of China (No. 41371342).

Author information

Author notes

  1. Yujin Zheng, Chu He, Yan Huang, Shenghua Fan and Min Jiang have contributed equally to this work.

Authors and Affiliations

  1. School of Computer Science, Wuhan University, Wuhan, 430000, Hubei, China

    Yujin Zheng, Shenghua Fan & Dingwen Wang

  2. School of Electronic Information, Wuhan University, Wuhan, 430000, Hubei, China

    Chu He

  3. Zmvision Technology, Wuhan, 430000, Hubei, China

    Yan Huang

  4. School of Informatics, Xiamen University, Xiamen, 361005, Fujian, China

    Min Jiang

  5. Aerospace Information Research Institute Chinese Academy of Science, Beijing, 100094, China

    Yang Yi

Authors
  1. Yujin Zheng
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Chu He
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Yan Huang
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Shenghua Fan
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Min Jiang
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Dingwen Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Yang Yi
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding authors

Correspondence to Dingwen Wang or Yang Yi.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Zheng, Y., He, C., Huang, Y. et al. Learning to estimate optical flow using dual-frequency paradigm. Memetic Comp. 15, 341–354 (2023). https://doi.org/10.1007/s12293-023-00395-y

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s12293-023-00395-y

Keywords

Search

Navigation