Skip to main content
Springer Nature Link
Log in

Robust dense correspondence using deep convolutional features

  • Original Article
  • Published:
The Visual Computer Aims and scope Submit manuscript

Abstract

Image matching is a challenging problem as different views often undergo significant appearance changes caused by illumination changes, scale variations, large displacement, and deformation. Most state-of-the-art algorithms, however, still cannot perform well enough in handling challenging real-world cases, especially in different objects and scenes. In this paper, we explore deep features extracted from pretrained convolutional neural networks to help the estimation of image matching so that dense pixel correspondence can be built. As the deep features are able to describe the image structures and details hierarchically, the matching method based on these features is able to match different scenes and object appearances effectively. We analyze the deep features and compare them with other robust features, e.g., SIFT. Extensive experiments on benchmark datasets demonstrate that the proposed algorithm performs favorably against the state-of-the-art methods in terms of visually matching quality and accuracy.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15

Similar content being viewed by others

Explore related subjects

Discover the latest articles, news and stories from top researchers in related subjects.

References

  1. Baker, S., Scharstein, D., Lewis, J.P., Roth, S., Black, M.J., Szeliski, R.: A database and evaluation methodology for optical flow. In: NIPS, pp. 1–31 (2011)

  2. Barnes, C., Shechtman, E., Finkelstein, A., Goldman, D.B.: Patchmatch: a randomized correspondence algorithm for structural image editing. In: ACM TOG, vol. 28, no. 3 (2009)

  3. Black, M.J., Anandan, P.: The robust estimation of multiple motions—parametric and piecewise-smooth flow fields. Comput. Visi. Image Underst. 63(1), 75–104 (1996)

    Article  Google Scholar 

  4. Brox, T., Bregler, C., Malik, J.: Large displacement optical flow. In: CVPR, pp. 41–48 (2009)

  5. Brox, T., Bruhn, A., Papenberg, N., Weickert, J.: High accuracy optical flow estimation based on a theory for warping. In: ECCV, pp. 25–36 (2004)

  6. Brox, T., Malik, J.: Large displacement optical flow: descriptor matching in variational motion estimation. IEEE Trans Pattern Anal. Mach. Intell. 33(3), 500–513 (2011)

    Article  Google Scholar 

  7. Choy, C.B., Gwak, J., Savarese, S., Chandraker, M.K.: Universal correspondence network. In: NIPS (2016)

  8. Datta, R., Joshi, D., Li, J., Wang, J.Z.: Image retrieval—Ideas, influences, and trends of the new age. ACM Comput. Surv. 40(2), 1–60 (2008)

    Article  Google Scholar 

  9. Deng, J., Dong, W., Socher, R., Li, L.J., Li, K., Li, F.F.: ImageNet—a large-scale hierarchical image database. In: CVPR, pp. 248–255 (2009)

  10. Dosovitskiy, A., Fischer, P., Ilg, E., Häusser, P., Hazirbas, C., Golkov, V., van der Smagt, P., Cremers, D., Brox, T.: FlowNet—learning optical flow with convolutional networks. In: ICCV (2015)

  11. van den Elsen, P.A., Pol, E.J.D., Viergever, M.A.: Medical image matching—a review with classification. IEEE Eng. Med. Biol. Mag. 12(1), 26–39 (1993)

    Article  Google Scholar 

  12. Fei-Fei, L., Fergus, R., Perona, P.: Learning generative visual models from few training examples: an incremental Bayesian approach tested on 101 object categories. Comput. Vis. Image Underst. 106(1), 59–70 (2007)

    Article  Google Scholar 

  13. Franke, U., Rabe, C., Badino, H., Gehrig, S.K.: 6D-Vision—fusion of stereo and motion for robust environment perception. In: DAGM-Symposium vol. 3663, pp. 216–223 (2005) (Chapter 27)

    Google Scholar 

  14. Han, K., Rezende, R.S., Ham, B., Wong, K.Y.K., Cho, M., Schmid, C., Ponce, J.: SCNet—learning semantic correspondence. In: ICCV, pp. 1849–1858 (2017)

  15. He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: CVPR, pp. 770–778 (2016)

  16. Horn, B.K., Schunck, B.G.: Determining optical flow. Artif. Intell. 17, 185–203 (1981)

    Article  Google Scholar 

  17. Hur, J., Lim, H., Park, C., Ahn, S.C.: Generalized deformable spatial pyramid—geometry-preserving dense correspondence estimation. In: CVPR, pp. 1392–1400 (2015)

  18. Ilg, E., Mayer, N., Saikia, T., Keuper, M., Dosovitskiy, A., Brox, T.: FlowNet 2.0: evolution of optical flow estimation with deep networks. In: CVPR, pp. 1647–1655 (2017)

  19. Kim, J., Liu, C., Sha, F., Grauman, K.: Deformable spatial pyramid matching for fast dense correspondences. In: CVPR, pp. 2307–2314 (2013)

  20. Kim, S., Min, D., Ham, B., Jeon, S., Lin, S., Sohn, K.: FCSS—fully convolutional self-similarity for dense semantic correspondence. In: CVPR, pp. 616–625 (2017)

  21. Lester, H., Arridge, S.R.: A survey of hierarchical non-linear medical image registration. Pattern Recognit. 32(1), 129–149 (1999)

    Article  Google Scholar 

  22. Li, C., Wand, M.: Combining Markov random fields and convolutional neural networks for image synthesis. In: CVPR, pp. 2479–2486 (2016)

  23. Li, W., Cosker, D., Brown, M., Tang, R.: Optical flow estimation using Laplacian mesh energy. In: CVPR, pp. 2435–2442 (2013)

  24. Liu, C., Yuen, J., Torralba, A.: SIFT flow—dense correspondence across scenes and its applications. IEEE Trans. Pattern Anal. Mach. Intell. 33(5), 978–994 (2011)

    Article  Google Scholar 

  25. Liu, Y., Pan, J., Su, Z.: Deep feature matching for dense correspondence. In: ICIP (2017)

  26. Long, J., Shelhamer, E., Darrell, T.: Fully convolutional networks for semantic segmentation. In: CVPR, pp. 3431–3440 (2015)

  27. Long, J., Zhang, N., Darrell, T.: Do convnets learn correspondence? In: NIPS (2014)

  28. Lowe, D.G.: Distinctive image features from scale-invariant keypoints. Int. J. Comput. Vis. 60(2), 91–110 (2004)

    Article  Google Scholar 

  29. Ma, C., Huang, J.B., Yang, X., Yang, M.H.: Hierarchical convolutional features for visual tracking. In: ICCV, pp. 3074–3082 (2015)

  30. Nag, S.: Image registration techniques: a survey (2017). arXiv:1712.07540

  31. Revaud, J., Weinzaepfel, P., Harchaoui, Z., Schmid, C.: EpicFlow: edge-preserving interpolation of correspondences for optical flow. In: CVPR, pp. 1164–1172 (2015)

  32. Russell, B.C., Torralba, A., Murphy, K.P., Freeman, W.T.: LabelMe: a database and web-based tool for image annotation. In: NIPS, pp. 157–173 (2007)

    Article  Google Scholar 

  33. Shekhovtsov, A., Kovtun, I., Hlavac, V.: Efficient MRF deformation model for non-rigid image matching. In: CVPR, pp. 1–6 (2007)

  34. Shen, X., Tao, X., Zhou, C., Gao, H., Jia, J.: Regional foremost matching for internet scene images. In: ACM TOG , vol. 35, no. 6, pp. 1–12 (2016)

    Article  Google Scholar 

  35. Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. In: ICLR (2015)

  36. Sun, D., Roth, S., Black, M.J.: Secrets of optical flow estimation and their principles. In: CVPR, pp. 2432–2439 (2010)

  37. Tau, M., Hassner, T.: Dense correspondences across scenes and scales. IEEE Trans. Pattern Anal. Mach. Intell. 38(5), 875–888 (2016)

    Article  Google Scholar 

  38. Weinzaepfel, P., Revaud, J., Harchaoui, Z., Schmid, C.: DeepFlow—large displacement optical flow with deep matching. In: ICCV, pp. 1385–1392 (2013)

  39. Xu, L., Jia, J., Matsushita, Y.: Motion detail preserving optical flow estimation. IEEE Trans. Pattern Anal. Mach. Intell. 34(9), 1744–1757 (2012)

    Article  Google Scholar 

  40. Yan, Z., Xiang, X.: Scene flow estimation: a survey (2016). arXiv:1612.02590

  41. Yang, H., Lin, W.Y., Lu, J.: DAISY filter flow: a generalized discrete approach to dense correspondences. In: CVPR, pp. 3406–3413 (2014)

  42. Zeiler, M.D., Fergus, R.: Visualizing and understanding convolutional networks. In: ECCV (2014)

  43. Zhou, T., Krähenbühl, P., Aubry, M., Huang, Q., Efros, A.A.: learning dense correspondence via 3D-guided cycle consistency. In: CVPR, pp. 117–126 (2016)

Download references

Funding

This work is partially supported by the National Natural Science Foundation of China (Nos. 61872421, 61572099, and 61702243), the National Science and Technology Major Project (Nos. ZX20140419, 2014ZX04001011), and the Natural Science Foundation of Jiangsu Province (No. BK20180471).

Author information

Authors and Affiliations

  1. School of Mathematical Sciences, Dalian University of Technology, Dalian, China

    Yang Liu & Zhixun Su

  2. School of Computer Science and Engineering, Nanjing University of Science and Technology, Nanjing, China

    Jinshan Pan

  3. School of Mathematics, Liaoning Normal University, Dalian, China

    Kewei Tang

Authors
  1. Yang Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Jinshan Pan
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Zhixun Su
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Kewei Tang
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Jinshan Pan.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Liu, Y., Pan, J., Su, Z. et al. Robust dense correspondence using deep convolutional features. Vis Comput 36, 827–841 (2020). https://doi.org/10.1007/s00371-019-01656-z

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00371-019-01656-z

Keywords

Search

Navigation