Abstract
Infrared-visible image patches matching has many applications, such as target recognition, vision-based navigation, and others. At present, deep learning has achieved excellent performance in visible image patches matching. Due to imaging differences in infrared-visible images and the small number of supervised samples, the existing networks cannot solve them well. We propose a Dual-Y network based on semi-supervised transfer learning to address the challenges. Since the infrared and visible image patches have similarities in the same scene, adversarial domain adaptation uses visible images as the source domain and infrared images as the target domain to solve different imaging principles. Through a large number of unannotated samples training, the two convolutional autoencoders in our network can reconstruct the infrared-visible image patches respectively to improve feature representation ability. In the high-level convolution layers, the cross-domain features are extracted by sharing weights in two domains. The adversarial domain adaptation achieves the domain confusion in the high layers. In the low-level convolutional layers, independent weights in two branches can ensure that the domain-related imaging principles can be preserved. Finally, the annotated data are used for supervised training in infrared-visible image patches matching. With the improvements above, the matching accuracy increase by 10.78%, compared with the fine-tuning based on the pre-training module.
![](http://media.springernature.com/m312/springer-static/image/art%3A10.1007%2Fs10489-020-01996-7/MediaObjects/10489_2020_1996_Fig1_HTML.png)
![](http://media.springernature.com/m312/springer-static/image/art%3A10.1007%2Fs10489-020-01996-7/MediaObjects/10489_2020_1996_Fig2_HTML.png)
![](http://media.springernature.com/m312/springer-static/image/art%3A10.1007%2Fs10489-020-01996-7/MediaObjects/10489_2020_1996_Fig3_HTML.png)
![](http://media.springernature.com/m312/springer-static/image/art%3A10.1007%2Fs10489-020-01996-7/MediaObjects/10489_2020_1996_Fig4_HTML.png)
![](http://media.springernature.com/m312/springer-static/image/art%3A10.1007%2Fs10489-020-01996-7/MediaObjects/10489_2020_1996_Fig5_HTML.png)
![](http://media.springernature.com/m312/springer-static/image/art%3A10.1007%2Fs10489-020-01996-7/MediaObjects/10489_2020_1996_Fig6_HTML.png)
![](http://media.springernature.com/m312/springer-static/image/art%3A10.1007%2Fs10489-020-01996-7/MediaObjects/10489_2020_1996_Fig7_HTML.png)
![](http://media.springernature.com/m312/springer-static/image/art%3A10.1007%2Fs10489-020-01996-7/MediaObjects/10489_2020_1996_Fig8_HTML.png)
Similar content being viewed by others
Explore related subjects
Discover the latest articles, news and stories from top researchers in related subjects.References
Acharya UR, Fujita H, Oh SL, Hagiwara Y., Tan JH, Adam M, Tan RS (2019) Deep convolutional neural network for the automated diagnosis of congestive heart failure using ecg signals. Appl Intell 49 (1):16–27
Badrinarayanan V, Kendall A, Cipolla R (2017) Segnet: a deep convolutional encoder-decoder architecture for image segmentation. IEEE Trans Pat Anal Mach Intell 39(12):2481–2495
Bay H, Tuytelaars T, Gool LJV (2006) Surf:, speeded up robust features. European Conf Comput Vis 3951:404–417
Cheng KS, Lin HY (2015) Automatic target recognition by infrared and visible image matching. In: Iapr international conference on machine vision applications, pp 312–315
Ganin Y, Ustinova E, Ajakan H, Germain P, Larochelle H, Laviolette F, Marchand M, Lempitsky V (2016) Domain-adversarial training of neural networks. J Mach Learn Res 17(1):189–209
Ghifary M, Kleijn WB, Zhang M., Balduzzi D, Li W (2016) Deep reconstruction-classification networks for unsupervised domain adaptation. In: European conference on computer vision. Springer, pp 597–613
Goodfellow IJ, Pouget-Abadie J, Mirza M, Xu B, Warde-Farley D, Ozair S, Courville AC, Bengio Y (2014) Generative adversarial nets. In: Advances in neural information processing systems, vol 27, pp 2672–2680
Han X, Leung T, Jia Y, Sukthankar R, Berg AC (2015) Matchnet: unifying feature and metric learning for patch-based matching. In: 2015 IEEE conference on computer vision and pattern recognition (CVPR), pp 3279–3286
Hanif MS (2019) Patch match networks: improved two-channel and siamese networks for image patch matching. Pattern Recogn Lett 120:54–61
Jiang B, Chen C, Jin X (2018) Unsupervised domain adaptation with target reconstruction and label confusion in the common subspace. Neural Comput Applic: 1–14
Li DM, Zhang JL (2013) A improved infrared and visible images matching based on surf. Appl Mech Mater: 1637–1640
Li H, Ding W, Cao X, Liu C (2017) Image registration and fusion of visible and infrared integrated camera for medium-altitude unmanned aerial vehicle remote sensing. Remote Sensing 9(5):441
Li X, Aouf N (2013) Sift and surf feature analysis in visible and infrared imaging for uavs. In: 2012 IEEE 11th international conference on cybernetic intelligent systems (CIS), pp 46–51
Li X, Aouf N (2014) Sift and surf feature analysis in visible and infrared imaging for uavs. In: IEEE international conference on cybernetic intelligent systems
Li-E JI, Yang FB, Wang ZS, Chen L (2012) Visible and infrared image matching algorithm based on edge image and surf features. Infrared Technol 34(11):629–635
Liu H, Cocea M (2017) Granular computing based approach for classification towards reduction of bias in ensemble learning. Granular Comput 2(3):131–139
Liu H, Cocea M (2019) Granular computing-based approach of rule learning for binary classification. Granular Comput 4(2): 275–283
Liu H, Cocea M (2019) Nature-inspired framework of ensemble learning for collaborative classification in granular computing context. Granular Comput 4(4):715–724
Liu H, Cocea M, Ding W (2018) Multi-task learning for intelligent data processing in granular computing context. Granular Comput 3(3):257–273
Long M, Cao Y, Cao Z, Wang J, Jordan MI (2019) Transferable representation learning with deep adaptation networks. IEEE Trans Pattern Anal Mach Intell 41(12):3071–3085
Long M, Zhu H, Wang J, Jordan MI (2016) Unsupervised domain adaptation with residual transfer networks. In: NIPS’16 proceedings of the 30th international conference on neural information processing systems, pp 136–144
Long M, Zhu H, Wang J, Jordan MI (2017) Deep transfer learning with joint adaptation networks. In: ICML’17 proceedings of the 34th international conference on machine learning, vol 70, pp 2208–2217
Lowe DG (2004) Distinctive image features from scale-invariant keypoints. Int J Comput Vis 60 (2):91–110
van der Maaten L, Hinton GE (2008) Visualizing data using t-sne. J Mach Learn Res 9:2579–2605
Nie W, Liu A, Su Y (2016) Cross-domain semantic transfer from large-scale social media. Multimedia Sys 22(1):75–85
Rublee E, Rabaud V, Konolige K, Bradski GR (2011) Orb: an efficient alternative to sift or surf. In: 2011 international conference on computer vision, pp 2564–2571
Sima AA, Buckley SJ (2013) Optimizing sift for matching of short wave infrared and visible wavelength images. Remote Sens 5(5):2037–2056
Simonyan K, Zisserman A (2014) Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556
Tzeng E, Hoffman J, Saenko K, Darrell T (2017) Adversarial discriminative domain adaptation. In: 2017 IEEE conference on computer vision and pattern recognition (CVPR), pp 2962–2971
Wang M, Deng W (2018) Deep visual domain adaptation: a survey. Neurocomputing 312:135–153
Wang S, Zhao Z, Yu P, Guang Z (2011) Infrared/visible image matching algorithm based on nsct and daisy. In: 2011 4th international congress on image and signal processing, vol 4, pp 2072–2075
Wu X, Du Z, Guo Y, Fujita H (2019) Hierarchical attention based long short-term memory for chinese lyric generation. Appl Intell 49(1):44–52
Yosinski J, Clune J, Bengio Y, Lipson H (2014) How transferable are features in deep neural networks?. In: Advances in neural information processing systems, pp 3320–3328
Zagoruyko S, Komodakis N (2015) Learning to compare image patches via convolutional neural networks. In: 2015 IEEE conference on computer vision and pattern recognition (CVPR), pp 4353–4361
Zhao L, Zhou Y, Lu H, Fujita H (2019) Parallel computing method of deep belief networks and its application to traffic flow prediction. Knowledge Based Systems 163:972–987
Author information
Authors and Affiliations
Corresponding author
Additional information
Publisher’s note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
About this article
Cite this article
Mao, Y., He, Z. Dual-Y network: infrared-visible image patches matching via semi-supervised transfer learning. Appl Intell 51, 2188–2197 (2021). https://doi.org/10.1007/s10489-020-01996-7
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10489-020-01996-7