Skip to main content
SpringerLink
Log in

Exploiting Style Transfer and Semantic Segmentation to Facilitate Infrared and Visible Image Fusion

  • Conference paper
  • First Online:
Technologies and Applications of Artificial Intelligence (TAAI 2023)

Part of the book series: Communications in Computer and Information Science ((CCIS,volume 2074))

  • 45 Accesses

Abstract

Image fusion integrates different imaging sources to generate one with improved scene representation or visual perception, supporting advanced vision tasks such as object detection and semantic analysis. Fusing infrared and visible images is a widely studied subject, and the current trend is to adopt deep learning models. It is well known that training a deep fusion model often requires many labeled data. Nevertheless, existing datasets only provide images without precise annotations, affecting the fusion presentation and limiting further development. This research creates a dataset for infrared and visible image fusion with semantic segmentation information. We utilize existing image datasets specific to semantic segmentation and generate corresponding infrared images by style transferring. A labeled dataset for image fusion is formed, in which each pair of infrared and visible images is accompanied by their semantic segmentation labels. The performance of image fusion in target datasets can thus be improved.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 79.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 99.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Wang, D., Liu, J., Fan, X., Liu, R.: Unsupervised misaligned infrared and visible image fusion via cross-modality image generation and registration. arXiv preprint arXiv:2205.11876 (2022)

  2. Tang, L., Deng, Y., Ma, Y., Huang, J., Ma, J.: SuperFusion: a versatile image registration and fusion network with semantic awareness. IEEE/CAA J. Autom. Sinica 9(12), 2121–2137 (2022). https://doi.org/10.1109/JAS.2022.106082

    Article  Google Scholar 

  3. Liu, J., et al.: Target-aware dual adversarial learning and a multi-scenario multi-modality benchmark to fuse infrared and visible for object detection. In: 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 18–24 June 2022, pp. 5792–5801 (2022). https://doi.org/10.1109/CVPR52688.2022.00571

  4. Tang, L., Yuan, J., Ma, J.: Image fusion in the loop of high-level vision tasks: a semantic-aware real-time infrared and visible image fusion network. Inform. Fusion 82, 28–42 (2022). https://doi.org/10.1016/j.inffus.2021.12.004

    Article  Google Scholar 

  5. Zhang, H., Xu, H., Tian, X., Jiang, J., Ma, J.: Image fusion meets deep learning: a survey and perspective. Inform. Fusion 76, 323–336 (2021). https://doi.org/10.1016/j.inffus.2021.06.008

    Article  Google Scholar 

  6. Li, H., Wu, X.J.: DenseFuse: a fusion approach to Infrared and Visible Images. IEEE Trans. Image Process. 28(5), 2614–2623 (2019). https://doi.org/10.1109/TIP.2018.2887342

    Article  MathSciNet  Google Scholar 

  7. Li, H., Wu, X.J., Durrani, T.: NestFuse: an infrared and visible image fusion architecture based on nest connection and spatial/channel attention models. IEEE Trans. Instrum. Meas. 69(12), 9645–9656 (2020). https://doi.org/10.1109/TIM.2020.3005230

    Article  Google Scholar 

  8. Tang, L., Yuan, J., Zhang, H., Jiang, X., Ma, J.: PIAFusion: a progressive infrared and visible image fusion network based on illumination aware. Inform. Fusion 83–84, 79–92 (2022). https://doi.org/10.1016/j.inffus.2022.03.007

    Article  Google Scholar 

  9. Ma, J., Yu, W., Liang, P., Li, C., Jiang, J.: FusionGAN: a generative adversarial network for infrared and visible image fusion. Inform. Fusion 48, 11–26 (2019). https://doi.org/10.1016/j.inffus.2018.09.004

    Article  Google Scholar 

  10. Ma, J., Xu, H., Jiang, J., Mei, X., Zhang, X.P.: DDcGAN: a dual-discriminator conditional generative adversarial network for multi-resolution image fusion. IEEE Trans. Image Process. 29, 4980–4995 (2020). https://doi.org/10.1109/TIP.2020.2977573

    Article  Google Scholar 

  11. Toet, A.: The TNO multiband image data collection. Data in Brief 15, 249–251 (2017). https://doi.org/10.1016/j.dib.2017.09.038

    Article  Google Scholar 

  12. Xu, H., Ma, J., Jiang, J., Guo, X., Ling, H.: U2Fusion: a unified unsupervised image fusion network. IEEE Trans. Pattern Anal. Mach. Intell. 44(1), 502–518 (2022). https://doi.org/10.1109/TPAMI.2020.3012548

    Article  Google Scholar 

  13. Jia, X., Zhu, C., Li, M., Tang, W., Liu, S., Zhou, W.: LLVIP: A Visible-infrared Paired Dataset for Low-light Vision. arXiv e-prints, p. arXiv:2108.10831 (2021). https://ui.adsabs.harvard.edu/abs/2021arXiv210810831J

  14. Cordts, M., et al.: The cityscapes dataset for semantic urban scene understanding. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 27–30 June 2016, pp. 3213–3223 (2016).s https://doi.org/10.1109/CVPR.2016.350

  15. Xie, E., Wang, W., Yu, Z., Anandkumar, A., Alvarez, J.M., Luo, P.: SegFormer: simple and efficient design for semantic segmentation with transformers. Adv. Neural. Inf. Process. Syst. 34, 12077–12090 (2021)

    Google Scholar 

  16. Peng, C., Tian, T., Chen, C., Guo, X., Ma, J.: Bilateral attention decoder: a lightweight decoder for real-time semantic segmentation. Neural Netw. 137, 188–199 (2021). https://doi.org/10.1016/j.neunet.2021.01.021

    Article  Google Scholar 

  17. Liu, J., Fan, X., Jiang, J., Liu, R., Luo, Z.: Learning a deep multi-scale feature ensemble and an edge-attention guidance for image fusion. IEEE Trans. Circuits Syst. Video Technol. 32(1), 105–119 (2022). https://doi.org/10.1109/TCSVT.2021.3056725

    Article  Google Scholar 

  18. Qu, G., Zhang, D., Yan, P.: Information measure for performance of image fusion. Electron. Lett. 38(7), 313 (2002). https://doi.org/10.1049/el:20020212

    Article  Google Scholar 

  19. Sheikh, H.R., Bovik, A.C.: Image information and visual quality. IEEE Trans. Image Process. 15(2), 430–444 (2006). https://doi.org/10.1109/TIP.2005.859378

    Article  Google Scholar 

  20. Aslantas, V., Bendes, E.: A new image quality metric for image fusion: the sum of the correlations of differences. AEU – Int. J. Electron. Commun. 69(12), 1890–1896 (2015). https://doi.org/10.1016/j.aeue.2015.09.004

    Article  Google Scholar 

  21. Xydeas, C., Petrovic, V.: Objective image fusion performance measure. Electron. Lett. 36, 308–309 (2000). https://doi.org/10.1049/el:20000267

    Article  Google Scholar 

Download references

Acknowledgment

This research is supported by the National Science and Technology Council, Taiwan, under Grants NSTC 111-2221-E-008-098 and 112-2221-E-008-077.

Author information

Authors and Affiliations

  1. Department of Computer Science and Information Engineering, National Central University, Taoyuan, Taiwan

    Hsing-Wei Chang, Po-Chyi Su & Si-Ting Lin

Authors
  1. Hsing-Wei Chang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Po-Chyi Su
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Si-Ting Lin
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Po-Chyi Su .

Editor information

Editors and Affiliations

  1. National Yunlin University of Science and Technology, Douliou, Taiwan

    Chao-Yang Lee

  2. National Yunlin University of Science and Technology, Douliou, Taiwan

    Chun-Li Lin

  3. National Yunlin University of Science and Technology, Douliou, Taiwan

    Hsuan-Ting Chang

Rights and permissions

Reprints and permissions

Copyright information

© 2024 The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd.

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Chang, HW., Su, PC., Lin, ST. (2024). Exploiting Style Transfer and Semantic Segmentation to Facilitate Infrared and Visible Image Fusion. In: Lee, CY., Lin, CL., Chang, HT. (eds) Technologies and Applications of Artificial Intelligence. TAAI 2023. Communications in Computer and Information Science, vol 2074. Springer, Singapore. https://doi.org/10.1007/978-981-97-1711-8_21

Download citation

  • DOI: https://doi.org/10.1007/978-981-97-1711-8_21

  • Published:

  • Publisher Name: Springer, Singapore

  • Print ISBN: 978-981-97-1710-1

  • Online ISBN: 978-981-97-1711-8

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation