Skip to main content
SpringerLink
Log in

Object-Scale Adaptive Optical Flow Estimation Network

  • Conference paper
  • First Online:
PRICAI 2022: Trends in Artificial Intelligence (PRICAI 2022)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 13631))

Included in the following conference series:

  • 1157 Accesses

Abstract

Current deep learning methods for optical flow estimation often use spatial feature pyramids to extract image features. To get the correlation between images, they directly compute the cost volume of the obtained image features. In this process, fine object details tend to be ignored. To solve this fundamental problem, an object-scale adaptive optical flow estimation network is proposed, in which multi-scale features are selectively extracted and exploited using our developed feature selectable block (FSB). As a result, we can obtain the multi-scale receptive fields of objects at different scales in the image. To consolidate all image features generated from all scales, a new cost volume generation scheme called multi-scale cost volume generation block (MCVGB) is further proposed to aggregate information from different scales. Extensive experiments conducted on the Sintel and KITTI2015 datasets show that our proposed method can capture fine details of different scale objects with high accuracy and thus deliver superior performance over a number of state-of-the-art methods.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 84.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 109.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Brox, T., Bruhn, A., Papenberg, N., Weickert, J.: High accuracy optical flow estimation based on a theory for warping. In: Pajdla, T., Matas, J. (eds.) ECCV 2004. LNCS, vol. 3024, pp. 25–36. Springer, Heidelberg (2004). https://doi.org/10.1007/978-3-540-24673-2_3

    Chapter  Google Scholar 

  2. Brox, T., Malik, J.: Large displacement optical flow: descriptor matching in variational motion estimation. IEEE Trans. Pattern Anal. Mach. Intell. (TPAMI) 33, 500-13 (2011)

    Google Scholar 

  3. C, Y., X, B., Y, F.: LightPIVNet: An effective convolutional neural network for particle image velocimetry. IEEE Trans. Instrum. Meas. 70, 1–1 (2021)

    Google Scholar 

  4. Dosovitskiy, A., et al.: FlowNet: Learning optical flow with convolutional networks. In: International Conference on Computer Vision (ICCV), pp. 2758–2766 (2015)

    Google Scholar 

  5. Geiger, A., Lenz, P., Urtasun, R.: Are we ready for autonomous driving? the KITTI vision benchmark suite. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3354–3361 (2012)

    Google Scholar 

  6. Ghaywate, P., Vyas, F., Telang, S., Mangale, S.: A deep learning approach for motion segmentation using an optical flow technique. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1–6 (2019)

    Google Scholar 

  7. He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 770–778 (2016)

    Google Scholar 

  8. Hosni, A., Rhemann, C.: Fast cost-volume filtering for visual correspondence and beyond. IEEE Trans. Pattern Anal. Mach. Intell. (TPAMI) 35, 504–511 (2013)

    Google Scholar 

  9. Hui, T., Tang, X., Loy, C.C.: LiteFlowNet: A lightweight convolutional neural network for optical flow estimation. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 8981–8989 (2018)

    Google Scholar 

  10. Hur, J., Roth, S.: Iterative residual refinement for joint optical flow and occlusion estimation. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 5754–5763 (2019)

    Google Scholar 

  11. Ilg, E., Mayer, N., Saikia, T., Keuper, M., Dosovitskiy, A., Brox, T.: FlowNet 2.0: Evolution of optical flow estimation with deep networks. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1647–1655 (2017)

    Google Scholar 

  12. Jaderberg, M., Simonyan, K., Zisserman, A.: Spatial transformer networks. In: Advances in Neural Information Processing Systems. pp. 2017–2025 (2015)

    Google Scholar 

  13. Jayasundara, V., Roy, D., Fernando, B.: Flowcaps: Optical flow estimation with capsule networks for action recognition. In: IEEE Winter Conference on Applications of Computer Visio (WACV), pp. 3408–3417 (2021)

    Google Scholar 

  14. Jiang, S., Lu, Y., Li, H., Hartley, R.: Learning optical flow from a few matches. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 16592–16600 (2021)

    Google Scholar 

  15. Li, X., Wang, W., Hu, X., Yang, J.: Selective kernel networks. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 510–519 (2019)

    Google Scholar 

  16. Mayer, N., Ilg, E.: A large dataset to train convolutional networks for disparity, optical flow, and scene flow estimation. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 4040–4048 (2016)

    Google Scholar 

  17. Papenberg, N., Bruhn, A., Brox, T., Didas, S., Weickert, J.: Highly accurate optic flow computation with theoretically justified warping. IEEE Int. J. Comput. Vision (IJCV) 67, 141–158 (2006)

    Google Scholar 

  18. Ranjan, A., Black, M.J.: Optical flow estimation using a spatial pyramid network. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2720–2729 (2017)

    Google Scholar 

  19. Revaud, J., Weinzaepfel, P., Harchaoui, Z., Schmid, C.: Epicflow: Edge-preserving interpolation of correspondences for optical flow. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1164–1172 (2015)

    Google Scholar 

  20. Scharstein, D., Szeliski, R.: A taxonomy and evaluation of dense two-frame stereo correspondence algorithms. IEEE International J. Comput. Vis. (IJCV) 47, 7–42 (2002)

    Google Scholar 

  21. Scharstein, D., Szeliski, R.: High-resolution optical flow from 1d attention and correlation. In: International Conference on Computer Vision (ICCV), pp. 10478–10487 (2021)

    Google Scholar 

  22. Sun, D., Roth, S., Black, M.J.: A quantitative analysis of current practices in optical flow estimation and the principles behind them. IEEE Int. J. Comput. Vis. (IJCV) 106, 115–137 (2014)

    Google Scholar 

  23. Sun, D., Yang, X., Liu, M., Kautz, J.: PWC-Net: CNNs for optical flow using pyramid, warping, and cost volume. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 8934–8943 (2018)

    Google Scholar 

  24. Szegedy, C., Liu, W., Jia, Y.: Going deeper with convolutions. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1–9 (2015)

    Google Scholar 

  25. Teed, Z., Deng, J.: RAFT: Recurrent all-pairs field transforms for optical flow. In: European Conference on Computer Vision (ECCV), pp. 402–419 (2020)

    Google Scholar 

  26. Wang, H., Cai, P., Fan, R., Sun, Y., Liu, M.: End-to-end interactive prediction and planning with optical flow distillation for autonomous driving. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, pp. 2229–2238 (2021)

    Google Scholar 

  27. Wang, L., Koniusz, P., Huynh, D.: Hallucinating IDT descriptors and I3D optical flow features for action recognition with CNNs. In: International Conference on Computer Vision (ICCV), pp. 8697–8707 (2019)

    Google Scholar 

  28. Wulff, J., Butler, D.J., Stanley, G.B., Black, M.J.: Lessons and insights from creating a synthetic optical flow benchmark. In: Fusiello, A., Murino, V., Cucchiara, R. (eds.) ECCV 2012. LNCS, vol. 7584, pp. 168–177. Springer, Heidelberg (2012). https://doi.org/10.1007/978-3-642-33868-7_17

    Chapter  Google Scholar 

  29. Yang, G., Ramanan, D.: Volumetric correspondence networks for optical flow. In: Advances in Neural Information Processing Systems, pp. 793–803 (2019)

    Google Scholar 

  30. Yin, Z., Darrell, T., Yu, F.: Hierarchical discrete distribution decomposition for match density estimation. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 6044–6053 (2019)

    Google Scholar 

  31. Zhao, C., Feng, C., Li, D., Li, S.: Optical flow-auxiliary multi-task regression network for direct quantitative measurement, segmentation and motion estimation. In: Association for the Advancement of Artificial Intelligence (AAAI), pp. 1218–1225 (2020)

    Google Scholar 

  32. Zhao, S., Sheng, Y., Dong, Y., Chang, E.I., Xu, Y.: MaskFlowNet: Asymmetric feature matching with learnable occlusion mask. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 6277–6286 (2020)

    Google Scholar 

  33. Zuehlke, D., Posada, D.: Autonomous satellite detection and tracking using optical flow. CoRR abs/2204.07025 (2022)

    Google Scholar 

Download references

Acknowledgements

This work was supported in part by the Natural Science Foundation of the Jiangsu Higher Education Institutions of China under Grant 21KJA520007, in part by the National Natural Science Foundation of China under Grant 61572341, in part by the Priority Academic Program Development of Jiangsu Higher Education Institutions, in part by Collaborative Innovation Center of Novel Software Technology and Industrialization, and in part by the NTU-WASP Joint Project under Grants M4082184.

Author information

Authors and Affiliations

  1. School of Computer Science and Technology, Soochow University, Suzhou, China

    Mu Li & Bao jiang Zhong

  2. School of Electrical and Electronic Engineering, Nanyang Technological University, Singapore, Singapore

    Kai-Kuang Ma

Authors
  1. Mu Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Bao jiang Zhong
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Kai-Kuang Ma
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Bao jiang Zhong .

Editor information

Editors and Affiliations

  1. CSIRO Australian e-Health Research Centre, Brisbane, QLD, Australia

    Sankalp Khanna

  2. Shanghai Jiao Tong University, Shanghai, China

    Jian Cao

  3. University of Tasmania, Hobart, TAS, Australia

    Quan Bai

  4. University of Technology Sydney, Sydney, NSW, Australia

    Guandong Xu

Rights and permissions

Reprints and permissions

Copyright information

© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Li, M., Zhong, B.j., Ma, KK. (2022). Object-Scale Adaptive Optical Flow Estimation Network. In: Khanna, S., Cao, J., Bai, Q., Xu, G. (eds) PRICAI 2022: Trends in Artificial Intelligence. PRICAI 2022. Lecture Notes in Computer Science, vol 13631. Springer, Cham. https://doi.org/10.1007/978-3-031-20868-3_44

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-20868-3_44

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-20867-6

  • Online ISBN: 978-3-031-20868-3

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation