Abstract
A digital twin is a next-generation technology that connects virtual and physical environments into a single world. Although the virtual environment of a digital twin models the real world, the technology used to match the real world with the virtual environment has yet to be studied. The existing deep-learning-based image registration methods aim to extract feature points and descriptors and show a good registration performance in real images. However, these methods are difficult to apply in an actual digital twin environment because 3D and real 2D images have a significant difference in terms of the external and physical characteristics of the image itself. In this paper, we propose a deep learning model that self-learns the difference between virtual and real environments using a generative-adversarial network and self-supervised learning. Image registration between virtual environments with real-world images is a new method that has not been previously achieved, and we have demonstrated experimentally that the proposed method is applicable to various virtual environments and real-world image matching.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Hicks, B.: Industry 4.0 and Digital Twins: Key Lessons From NASA. https://www.thefuturefactory.com/blog/24. Accessed 09 June 2021
Lange, A., Heldmann, S.: Multilevel 2D-3D intensity-based image registration. In: Špiclin, Ž, McClelland, J., Kybic, J., Goksel, O. (eds.) WBIR 2020. LNCS, vol. 12120, pp. 57–66. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-50120-4_6oop
Zhu, J.Y., Park, T., Isola, P., Efros, A.A.: Unpaired image-to-image translation using cycle-consistent adversarial networks. In: ICCV, pp. 2223–2232. IEEE, Venice (2017)
Rosten, E., Drummond, T.: Machine learning for high-speed corner detection. In: Leonardis, A., Bischof, H., Pinz, A. (eds.) ECCV 2006. LNCS, vol. 3951, pp. 430–443. Springer, Heidelberg (2006). https://doi.org/10.1007/11744023_34
Lowe, D.G.: Object recognition from local scale-invariant features. In: ICCV, pp. 1150–1157. IEEE, Kerkyra (1999)
Bay, H., Tuytelaars, T., Van Gool, L.: SURF: speeded up robust features. In: Leonardis, A., Bischof, H., Pinz, A. (eds.) ECCV 2006. LNCS, vol. 3951, pp. 404–417. Springer, Heidelberg (2006). https://doi.org/10.1007/11744023_32
Rublee, E., Rabaud, V., Konolige, K., Bradski, G.: ORB: an efficient alternative to SIFT or SURF. In: ICCV, pp. 2564–2571. IEEE, Barcelona (2011)
Alcantrilla, P.F., Solutions, T.: Fast explicit diffusion for accelerated features in nonlinear scale spaces. IEEE TPAMI 34(7), 1281–1298 (2011)
DeTone, D., Malisiewicz, T., Rabinovich, A.: Superpoint: self-supervised interest point detection and description. In: CVPR Workshops, pp. 224–236. IEEE, Long Beach (2018)
Revaud, J., et al.: R2D2: repeatable and reliable detector and descriptor. arXiv preprint arXiv:1906.06195 (2019)
Kim, S., Jeong, M., Ko, B.C.: Self-supervised keypoint detection based on multi-layer random forest regressor. IEEE ACCESS 9, 40850–40859 (2021)
Acknowledgement
This research was supported by Basic Science Research Program through the National Research Foundation of Korea (NRF) funded by the Ministry of Education (2021R1A6A3A13039438) and partially supported by Electronics and Telecommunications Research Institute (ETRI) grant funded by the Korean government. [21ZD1120, Development of ICT Convergence Technology for Daegu-Gyeongbuk Regional Industry].
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2022 Springer Nature Switzerland AG
About this paper
Cite this paper
Kim, S., Jang, IS., Ko, B.C. (2022). Image Registration Between Real Image and Virtual Image Based on Self-supervised Keypoint Learning. In: Wallraven, C., Liu, Q., Nagahara, H. (eds) Pattern Recognition. ACPR 2021. Lecture Notes in Computer Science, vol 13189. Springer, Cham. https://doi.org/10.1007/978-3-031-02444-3_30
Download citation
DOI: https://doi.org/10.1007/978-3-031-02444-3_30
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-02443-6
Online ISBN: 978-3-031-02444-3
eBook Packages: Computer ScienceComputer Science (R0)