Skip to main content
SpringerLink
Log in

Unsupervised Deep Plane-Aware Multi-homography Learning for Image Alignment

  • Conference paper
  • First Online:
Artificial Intelligence (CICAI 2021)

Part of the book series: Lecture Notes in Computer Science ((LNAI,volume 13069))

Included in the following conference series:

  • 2158 Accesses

Abstract

Due to its feature representation capabilities, deep learning has been applied to homography estimation in the field of image alignment. Most deep homography learning methods focus on estimating a single global homography, and cannot deal with the problem of parallax when the scene contains multiple different planes, and the translation of the camera’s optical center is not negligible. In this paper, we propose an unsupervised multi-homography learning method with a plane-perception trait to mitigate this parallax problem. In our model, the problem of multi-homography learning and plane perception are jointly considered, which can benefit from each other. To make the learning process stable under unsupervised setting, we design a special attention mechanism to bootstrap the collaboration between multi-homography learning and plane perception. We construct a new dataset that is captured in real scenes, having many challenges such as multiple planes, large parallax, etc. Quantitative and qualitative results show that our proposed method can better align images with large parallax and multiple planes.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 89.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 119.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Arefin, M.R., et al.: Multi-image super-resolution for remote sensing using deep recurrent networks. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pp. 206–207 (2020)

    Google Scholar 

  2. Baker, S., Datta, A., Kanade, T.: Parameterizing homographies. In: Technical Report CMU-RI-TR-06-11 (2006)

    Google Scholar 

  3. Bonetti, F., Warnaby, G., Quinn, L.: Augmented reality and virtual reality in physical and online retailing: a review, synthesis and research agenda. In: Jung, T., tom Dieck, M.C. (eds.) Augmented Reality and Virtual Reality. PI, pp. 119–132. Springer, Cham (2018). https://doi.org/10.1007/978-3-319-64027-3_9

    Chapter  Google Scholar 

  4. Brown, M., Lowe, D.G.: Automatic panoramic image stitching using invariant features. Int. J. Comput. Vis. 74(1), 59–73 (2007)

    Article  Google Scholar 

  5. Burdea, G.C., Coiffet, P.: Virtual Reality Technology. Wiley, Hoboken (2003)

    Book  Google Scholar 

  6. Chang, C.H., Chou, C.N., Chang, E.Y.: CLKN: cascaded Lucas-Kanade networks for image alignment. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2213–2221 (2017)

    Google Scholar 

  7. Chen, S.Y., Chuang, Y.Y.: Deep exposure fusion with deghosting via homography estimation and attention learning. In: ICASSP 2020–2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 1464–1468. IEEE (2020)

    Google Scholar 

  8. Corbetta, M., Shulman, G.L.: Control of goal-directed and stimulus-driven attention in the brain. Nat. Rev. Neurosci. 3(3), 201–215 (2002)

    Article  Google Scholar 

  9. Dempster, A.P., Laird, N.M., Rubin, D.B.: Maximum likelihood from incomplete data via the EM algorithm. J. Roy. Stat. Soc.: Ser. B (Methodol.) 39(1), 1–22 (1977)

    MathSciNet  MATH  Google Scholar 

  10. DeTone, D., Malisiewicz, T., Rabinovich, A.: Deep image homography estimation. arXiv preprint arXiv:1606.03798 (2016)

  11. Erlik Nowruzi, F., Laganiere, R., Japkowicz, N.: Homography estimation from image pairs with hierarchical convolutional networks. In: Proceedings of the IEEE International Conference on Computer Vision Workshops, pp. 913–920 (2017)

    Google Scholar 

  12. He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 770–778 (2016)

    Google Scholar 

  13. Itti, L., Koch, C., Niebur, E.: A model of saliency-based visual attention for rapid scene analysis. IEEE Trans. Pattern Anal. Mach. Intell. 20(11), 1254–1259 (1998)

    Article  Google Scholar 

  14. Jaderberg, M., Simonyan, K., Zisserman, A., Kavukcuoglu, K.: Spatial transformer networks. In: Proceedings of the 28th International Conference on Neural Information Processing Systems, vol. 2, pp. 2017–2025 (2015)

    Google Scholar 

  15. Le, H., Liu, F., Zhang, S., Agarwala, A.: Deep homography estimation for dynamic scenes. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7652–7661 (2020)

    Google Scholar 

  16. Li, Y., Pei, W., He, Z.: SRHEN: stepwise-refining homography estimation network via parsing geometric correspondences in deep latent space. In: Proceedings of the 28th ACM International Conference on Multimedia, pp. 3063–3071 (2020)

    Google Scholar 

  17. Lin, T.-Y., et al.: Microsoft COCO: common objects in context. In: Fleet, D., Pajdla, T., Schiele, B., Tuytelaars, T. (eds.) ECCV 2014. LNCS, vol. 8693, pp. 740–755. Springer, Cham (2014). https://doi.org/10.1007/978-3-319-10602-1_48

    Chapter  Google Scholar 

  18. Lucas, B.D., Kanade, T.: An iterative image registration technique with an application to stereo vision. In: Proceedings of the 7th International Joint Conference on Artificial Intelligence, IJCAI 1981, vol. 2, pp. 674–679. Morgan Kaufmann Publishers Inc., San Francisco (1981)

    Google Scholar 

  19. Molina-Cabello, M.A., Elizondo, D.A., Luque Baena, R.M., López-Rubio, E., et al.: Homography estimation with deep convolutional neural networks by random color transformations. In: British Machine Vision Conference, pp. 1–11 (2019)

    Google Scholar 

  20. Nguyen, T., Chen, S.W., Shivakumar, S.S., Taylor, C.J., Kumar, V.: Unsupervised deep homography: a fast and robust homography estimation model. IEEE Robot. Autom. Lett. 3(3), 2346–2353 (2018)

    Article  Google Scholar 

  21. Niblick, D., Kak, A.: Homography estimation with convolutional neural networks under conditions of variance. arXiv preprint arXiv:2010.01041 (2020)

  22. Nie, L., Lin, C., Liao, K., Zhao, Y.: Learning edge-preserved image stitching from large-baseline deep homography. arXiv preprint arXiv:2012.06194 (2020)

  23. Reinhard, E., Heidrich, W., Debevec, P., Pattanaik, S., Ward, G., Myszkowski, K.: High Dynamic Range Imaging: Acquisition, Display, and Image-based Lighting. Morgan Kaufmann, Burlington (2010)

    Google Scholar 

  24. Rocco, I., Arandjelovic, R., Sivic, J.: Convolutional neural network architecture for geometric matching. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 6148–6157 (2017)

    Google Scholar 

  25. Shen, D., Wu, G., Suk, H.I.: Deep learning in medical image analysis. Ann. Rev. Biomed. Eng. 19, 221–248 (2017)

    Article  Google Scholar 

  26. Shen, X., Darmon, F., Efros, A.A., Aubry, M.: RANSAC-flow: generic two-stage image alignment. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, J.-M. (eds.) ECCV 2020. LNCS, vol. 12349, pp. 618–637. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-58548-8_36

    Chapter  Google Scholar 

  27. Szeliski, R.: Image alignment and stitching: a tutorial. Found. Trends® Comput. Graph. Vis. 2(1), 1–104 (2006)

    MathSciNet  MATH  Google Scholar 

  28. Wang, C., Wang, X., Bai, X., Liu, Y., Zhou, J.: Self-supervised deep homography estimation with invertibility constraints. Pattern Recogn. Lett. 128, 355–360 (2019)

    Article  Google Scholar 

  29. Wang, Z., Bovik, A.C., Sheikh, H.R., Simoncelli, E.P.: Image quality assessment: from error visibility to structural similarity. IEEE Trans. Image Process. 13(4), 600–612 (2004)

    Article  Google Scholar 

  30. Wronski, B., et al.: Handheld multi-frame super-resolution. ACM Trans. Graph. (TOG) 38(4), 1–18 (2019)

    Article  Google Scholar 

  31. Ye, N., Wang, C., Fan, H., Liu, S.: Motion basis learning for unsupervised deep homography estimation with subspace projection. arXiv preprint arXiv:2103.15346 (2021)

  32. Ye, N., Wang, C., Liu, S., Jia, L., Wang, J., Cui, Y.: DeepMeshFlow: content adaptive mesh deformation for robust image registration. arXiv preprint arXiv:1912.05131 (2019)

  33. Zaragoza, J., Chin, T.J., Brown, M.S., Suter, D.: As-projective-as-possible image stitching with moving DLT. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2339–2346 (2013)

    Google Scholar 

  34. Zeng, R., Denman, S., Sridharan, S., Fookes, C.: Rethinking planar homography estimation using perspective fields. In: Jawahar, C.V., Li, H., Mori, G., Schindler, K. (eds.) ACCV 2018. LNCS, vol. 11366, pp. 571–586. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-20876-9_36

    Chapter  Google Scholar 

  35. Zhang, J., et al.: Content-aware unsupervised deep homography estimation. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, J.-M. (eds.) ECCV 2020. LNCS, vol. 12346, pp. 653–669. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-58452-8_38

    Chapter  Google Scholar 

Download references

Acknowledgment

This work was supported in part by the Natural Science Foundation of China (NSFC) under Grants No. 61773062.

Author information

Authors and Affiliations

  1. School of Computer Science, Beijing Institute of Technology, Beijing, China

    Tao Cai, Yunde Jia, Huijun Di & Yuwei Wu

Authors
  1. Tao Cai
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Yunde Jia
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Huijun Di
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Yuwei Wu
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Huijun Di .

Editor information

Editors and Affiliations

  1. Tsinghua University, Beijing, China

    Lu Fang

  2. Duke University, Durham, NC, USA

    Yiran Chen

  3. Shanghai Jiao Tong University, Shanghai, China

    Guangtao Zhai

  4. University of British Columbia, Vancouver, BC, Canada

    Jane Wang

  5. Institute of Computing Technology, Chinese Academy of Sciences, Beijing, China

    Ruiping Wang

  6. Xidian University, Xi’an, China

    Weisheng Dong

Rights and permissions

Reprints and permissions

Copyright information

© 2021 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Cai, T., Jia, Y., Di, H., Wu, Y. (2021). Unsupervised Deep Plane-Aware Multi-homography Learning for Image Alignment. In: Fang, L., Chen, Y., Zhai, G., Wang, J., Wang, R., Dong, W. (eds) Artificial Intelligence. CICAI 2021. Lecture Notes in Computer Science(), vol 13069. Springer, Cham. https://doi.org/10.1007/978-3-030-93046-2_45

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-93046-2_45

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-93045-5

  • Online ISBN: 978-3-030-93046-2

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation