Abstract
Infrared (IR) and Near-Infrared (NIR) images, which are more robust to illumination variances and more suitable for all-whether applications than visible (VIS) images, have been widely applied in the computer vision community. However, it’s cost-intensive and labor-demanding to collect IR/NIR images for downstream tasks. To solve this issue, a promising solution is generating IR/NIR images from visible ones via style transfer. Unfortunately, existing style transfer methods impose excessive constraints on preserving content or style clues while attaching little importance to both of them, which can not well capture the characteristic of IR/NIR image generation. In this paper, we propose an effective style transfer framework, termed Content Consistency and Style Adversarial Learning (\(C^2SAL\)), for IR and NIR image generation. Firstly, we propose the content consistency learning which is imposed on the refined content features from a content feature refining module, leading to the improvement of content information preservation. Besides, a style adversarial learning is proposed to achieve the style consistency between generated images and the images of the target style, which promotes the overall style transfer by utilizing both pixel-level and image-level style loss. Extensive experiments on challenging benchmarks, including detailed ablation study and comparisons with state-of-the-art methods, demonstrate the effectiveness of our method.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Yoo, J., Uh, Y., Chun, S., et al.: Photorealistic style transfer via wavelet transforms. In: ICCV(2019)
Deng, Y., Tang, F., Dong, W., et al.: Arbitrary style transfer via multi-adaptation network. In: 28th ACM MM, pp. 2719–2727 (2020)
Luo, F., Li, Y., Zeng, G., et al.: Thermal infrared image colorization for nighttime driving scenes with top-down guided attention. IEEE TITS, 1–16 (2022). IEEE
Chiu, T.Y., Gurari, D.: PhotoWCT2: compact autoencoder for photorealistic style transfer resulting from blockwise training and skip connections of high-frequency residuals. In: WACV, pp. 2868–2877 (2022)
Qiao, Y., Cui, J., Huang, F., et al.: Efficient style-corpus constrained learning for photorealistic style transfer. IEEE TIP 30, 3154–3166 (2021). IEEE
Cheng, M.M., Liu, X.C., Wang, J., et al.: Structure-preserving neural style transfer. IEEE TIP 29, 909–920 (2019). IEEE
Hong, K., Jeon, S., Yang, H., et al.: Domain-aware universal style transfer. In: ICCV, pp. 14609–14617 (2021)
Huo, J., Jin, S., Li, W., et al.: Manifold alignment for semantically aligned style transfer. In: ICCV, pp. 14861–14869 (2021)
Zhu, J.Y., Park, T., Isola, P., et al.: Unpaired image-to-image translation using cycle-consistent adversarial networks. In: ICCV, pp. 2223–2232 (2017)
Lee, H.-Y., Tseng, H.-Y., Huang, J.-B., Singh, M., Yang, M.-H.: Diverse image-to-image translation via disentangled representations. In: Ferrari, V., Hebert, M., Sminchisescu, C., Weiss, Y. (eds.) ECCV 2018. LNCS, vol. 11205, pp. 36–52. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-01246-5_3
Yang, Z., Chen, Z.: Learning from paired and unpaired data: alternately trained CycleGAN for near infrared image colorization. In: VCIP, pp. 467–470. IEEE (2020)
Yan, L., Wang, X., Zhao, M., et al.: A multi-model fusion framework for NIR-to-RGB translation. In: VCIP, pp. 459–462. IEEE (2020)
Wang, T., Zhang, T., Lovell, B.C.: EBIT: weakly-supervised image translation with edge and boundary enhancement. PR Lett. 138, 534–539 (2020). Elsevier
Babu, K.K., Dubey, S.R.: PCSGAN: perceptual cyclic-synthesized generative adversarial networks for thermal and NIR to visible image transformation. Neurocomputing 413, 41–50 (2020). Elsevier
Mehri, A., Sappa, A.D.: Colorizing near infrared images through a cyclic adversarial approach of unpaired samples. In: CVPR (Workshops) (2019)
Gatys, L.A., Ecker, A.S., Bethge, M.: Image style transfer using convolutional neural networks. In: CVPR, pp. 2414–2423 (2016)
Johnson, J., Alahi, A., Fei-Fei, L.: Perceptual losses for real-time style transfer and super-resolution. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9906, pp. 694–711. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-46475-6_43
Li, Y., Fang, C., Yang, J., et al.: Universal style transfer via feature transforms. In: NeurIPS (2017)
Huang, X., Belongie, S.: Arbitrary style transfer in real-time with adaptive instance normalization. In: ICCV, pp. 1501–1510 (2017)
Luan, F., Paris, S., Shechtman, E., et al.: Deep photo style transfer. In: CVPR, pp. 4990–4998 (2017)
Li, Y., Liu, M.-Y., Li, X., Yang, M.-H., Kautz, J.: A closed-form solution to photorealistic image stylization. In: Ferrari, V., Hebert, M., Sminchisescu, C., Weiss, Y. (eds.) ECCV 2018. LNCS, vol. 11207, pp. 468–483. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-01219-9_28
An, J., Xiong, H., Huan, J., et al.: Ultrafast photorealistic style transfer via neural architecture search. In: AAAI, vol. 34, pp. 10443–10450 (2020)
Jia, X., Zhu, C., Li, M., et al.: LLVIP: a visible-infrared paired dataset for low-light vision. In: ICCV, pp. 3496–3504 (2021)
Choe, G., Kim, S.H., Im, S., et al.: RANUS: RGB and NIR urban scene dataset for deep scene parsing. IEEE RA-L 3, 1808–1815 (2018). IEEE
Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556 (2014)
Isola, P., Zhu, J.Y., Zhou, T., et al.: Image-to-image translation with conditional adversarial networks. In: CVPR, pp. 1125–1134 (2017)
Deng, J., Dong, W., Socher, R., et al.: ImageNet: a large-scale hierarchical image database. In: CVPR, pp. 248–255. IEEE (2009)
Kingma, D.P., Ba, J.: Adam: a method for stochastic optimization. In: ICLR (2015)
He, K., Zhang, X., Ren, S., et al.: Deep residual learning for image recognition. In: CVPR, pp. 770–778 (2016)
Ronneberger, O., Fischer, P., Brox, T.: U-net: convolutional networks for biomedical image segmentation. In: Navab, N., Hornegger, J., Wells, W.M., Frangi, A.F. (eds.) MICCAI 2015. LNCS, vol. 9351, pp. 234–241. Springer, Cham (2015). https://doi.org/10.1007/978-3-319-24574-4_28
Goodfellow, I., Pouget-Abadie, J., Mirza, M., et al.: Generative adversarial nets. In: NeurIPS, pp. 2672–2680 (2014)
Mirza, M., Osindero, S.: Conditional generative adversarial nets. arXiv preprint arXiv:1411.1784 (2014)
Acknowledgement
This work is partially supported by National Natural Science Foundation of China (Grants no. 62176271 and 61772568), Guangdong Basic and Applied Basic Research Foundation (Grant no. 2019A1515012029), and Science and Technology Program of Guangzhou (Grant no. 202201011681).
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Mao, K., Yang, M., Wang, H. (2022). Infrared and Near-Infrared Image Generation via Content Consistency and Style Adversarial Learning. In: Yu, S., et al. Pattern Recognition and Computer Vision. PRCV 2022. Lecture Notes in Computer Science, vol 13534. Springer, Cham. https://doi.org/10.1007/978-3-031-18907-4_48
Download citation
DOI: https://doi.org/10.1007/978-3-031-18907-4_48
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-18906-7
Online ISBN: 978-3-031-18907-4
eBook Packages: Computer ScienceComputer Science (R0)