Skip to main content
SpringerLink
Log in

Infrared image super-resolution reconstruction by using generative adversarial network with an attention mechanism

  • Published:
Applied Intelligence Aims and scope Submit manuscript

Abstract

Due to the limitations of infrared imaging principles and imaging systems, many problems are typically encountered with collected infrared images, such as low resolution, insufficient detail information, and blurred edges. In response to these problems, a method of infrared image super-resolution reconstruction that uses recursive attention and is based on a generative adversarial network is proposed. First, according to the characteristics of low-resolution infrared images such as uniform pixel distributions, low contrast, and poor perceived quality, a deep generator structure with a recursive-attention network is designed in this article. The recursive-attention module is used to extract high-frequency information from the feature maps, suppress useless information, and enhance the expressiveness of the features, which facilitates the reconstruction of texture details of infrared images. Then, to better distinguish the reconstructed images from the original high-resolution images, we designed a discriminator that was composed of a deep convolutional neural network. In addition, targeted improvements were made to the content loss function of GAN. We used the pre-trained VGG-19 network features before activation to calculate the perceptual loss, which helps recover the texture details of the infrared images. The experimental results on infrared image datasets demonstrated that the reconstruction performance of the proposed method is higher than those of several typical methods, and it realizes higher image visual quality.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9

Similar content being viewed by others

References

  1. Chen Q (2013) The status and development trend of infrared image processing technology. Infrared Tech 35(6):311–318

    Google Scholar 

  2. Yu X, Ye X, Gao Q (2019) Pipeline image segmentation algorithm and heat loss calculation based on gene-regulated apoptosis mechanism. Int J Press Vessel Pip , 329-336

  3. Baker S, Kanade T (2002) Limits on super-resolution and how to break them. IEEE Trans Pattern Anal Mach Intell 24(9):1167–1183

    Article  Google Scholar 

  4. Zhang L, Wu X (2006) An edge-guided image interpolation algorithm via directional filtering and data fusion. IEEE Trans Image Process 15(8):2226–2238

    Article  Google Scholar 

  5. Zhang K, Gao X, Tao D (2012) Single image super-resolution with non-local means and steering kernel regression. IEEE Trans Image Process 21(11):4544–4556

    Article  MathSciNet  Google Scholar 

  6. Timofte R, De Smet V, Van G (2014) A+: adjusted anchored neighborhood regression for fast super-resolution. In Asian Conf Comput Vis (ACCV), 111-126

  7. Huang J, Singh A, Ahuja N (2015) Single image super-resolution from transformed self-exemplars. In proceedings of the Proc IEEE Conf Comput Vis Pattern Recognit, 5197-5206

  8. Tong T, Li G, Liu X(2017) Image super-resolution using dense skip connections. In Proc IEEE Int conf comput vis (ICCV), 4809–4817

  9. Liu X, Jia R, Liu Q, Zhao C, Sun H (2019) Coastline extraction method based on convolutional neural networks-a case study of Jiaozhou Bay in Qingdao, China. IEEE Access 7:180281–180291

    Article  Google Scholar 

  10. Zhang K, Zuo W, Zhang L (2018) Learning a single convolutional super-resolution network for multiple degradations. In Proc IEEE Conf Comput Vis Pattern Recognit (CVPR), 3262-3271

  11. Mao Q, Sun H, Liu Y, Jia R (2019) Fast and efficient non-contact ball detector for picking robots. IEEE Access 7:175487–175498

    Article  Google Scholar 

  12. He K, Cao X, Shi Y (2019) Pelvic organ segmentation using distinctive curve guided fully convolutional networks. IEEE Trans Med Imaging 38(2):585–595

    Article  Google Scholar 

  13. Cui Y, Schuon S, Chan D (2010) 3D shape scanning with a time-of-flight camera.In Proc IEEE Conf Comput Vis Pattern Recognit (CVPR), 1173-1180

  14. Tian Y, Jia R, Xu S, Hua R, Deng M (2019) Super-resolution reconstruction of remote sensing images based on convolutional neural network. J Appl Remote Sens 13(4):046502

    Article  Google Scholar 

  15. Liu Q, Jia R, Zhao C, Liu X, Sun H (2020) Face super-resolution reconstruction based on self-attention residual network. IEEE Access 8:4110–4121

    Article  Google Scholar 

  16. Dong C, Loy C, He K (2014) Learning a deep convolutional network for image super-resolution. Eur Conf Comput Vis Springer:184–199

  17. Kim J, Lee J, Lee K (2016) Deeply-recursive convolutional network for image super—resolution. In Proc IEEE Conf Comput Vis Pattern Recognit (CVPR), 1637-1645

  18. Goodfellow I, Pougetabadie J, Mirza M (2014) Generative adversarial nets. In Proc Annu Conf Neural Inf Process Syst (NIPS), 2672–2680

  19. Ledig C, Theis L, Huszár F (2017) Photo-realistic single image super-resolution using a generative adversarial network. In Proc IEEE Conf Comput Vis Pattern Recognit (CVPR), 105–114

  20. Qian R, Tan R, Yang W (2018) Attentive generative adversarial network for raindrop removal from a single image. In Proc IEEE Conf Comput Vis Pattern Recognit (CVPR), 2482–2491

  21. Zhang H, Goodfellow I, Metaxas D (2017) Self-attention generative adversarial networks. Mach Learn arXiv preprint arXiv:1805.08318 :1805.08318

  22. Zhang Y, Li K, Li K (2018) Image super-resolution using very deep residual channel attention networks. In Proc Eur Conf Comput Vis (ECCV), 294-310

  23. Adler J, Lunz S (2018) Banach Wasserstein GAN. Neural Inf Process Syst (NIPS), 6755–6764

  24. Gulrajani I, Ahmed F, Arjovsky M, Dumoulin V, Courville A (2017) Improved training of Wasserstein GANs In Adv Neural Inf Proces Syst (NIPS), 5769-5779

  25. Berthelot D, Schumm T, Metz L (2017) BEGAN: boundary equilibrium generative adversarial networks. arXiv preprint arXiv:1703.10717

  26. Yeh R, Chen C, Lim T (2017) Semantic image inpainting with deep generative models. In Proc IEEE Conf Comput Vis Pattern Recognit (CVPR), 6882-6890

  27. Xu C, Liu T, Tao D (2016) Local Rademacher complexity for multi-label learning. IEEE Trans Image Process 25(3):1495–1507

    Article  MathSciNet  Google Scholar 

  28. Hu J, Shen L, Sun G (2018) Squeeze-and-excitation networks. In Proc IEEE Conf Comput Vis Pattern Recognit (CVPR), 7132-7141

  29. Shi W, Caballero J, Huszár F (2016) Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network. In Proc IEEE Conf Comput Vis Pattern Recognit (CVPR), 1874-1883

  30. Wang X, Yu K, Wu S (2018) ESRGAN: enhanced super-resolution generative adversarial networks. In Proc Euro Conf Comp Vis (ECCV), 63-79

  31. Arjovsky M, Chintala S, Bottou L (2017) Wasserstein generative adversarial networks. Int Conf Mach Learn (ICML), 214-223

  32. He K, Zhang X, Ren S (2015) Delving deep into rectifiers: surpassing human-level performance on ImageNet classification. In IEEE Int Conf Compu Vis (ICCV), 1026–1034

  33. Kingma D, Ba J (2014) Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980

  34. Chen A, Chen B, Chai X, Bian R, Li H (2017) A novel stochastic stratified average gradient method: convergence rate and its complexity. arXiv preprint arXiv:1710.07783

  35. Mittal A, Soundararajan R, Bovik A (2013) Making a “completely blind” image quality analyzer. IEEE Signal Process Lett 20(3):209–212

    Article  Google Scholar 

Download references

Acknowledgements

The authors are grateful for collaborative funding support from the Natural Science Foundation of Shandong Province, China (ZR2018 MEE008) and the Key Research and Development Project of Shandong Province, China (2019JZZY020326, 2019GGX101066).

Author information

Authors and Affiliations

  1. College of Computer Science and Engineering, Shandong University of Science and Technology, Qingdao, 266590, China

    Qing-Ming Liu, Rui-Sheng Jia, Yan-Bo Liu, Hai-Bin Sun, Jian-Zhi Yu & Hong-Mei Sun

  2. Shandong Province Key Laboratory of Wisdom Mine Information Technology, Shandong University of Science and Technology, Qingdao, 266590, China

    Rui-Sheng Jia, Hai-Bin Sun, Jian-Zhi Yu & Hong-Mei Sun

Authors
  1. Qing-Ming Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Rui-Sheng Jia
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Yan-Bo Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Hai-Bin Sun
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Jian-Zhi Yu
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Hong-Mei Sun
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding authors

Correspondence to Rui-Sheng Jia or Hong-Mei Sun.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Liu, QM., Jia, RS., Liu, YB. et al. Infrared image super-resolution reconstruction by using generative adversarial network with an attention mechanism. Appl Intell 51, 2018–2030 (2021). https://doi.org/10.1007/s10489-020-01987-8

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10489-020-01987-8

Keywords

Search

Navigation