skip to main content
10.1145/3641584.3641676acmotherconferencesArticle/Chapter ViewAbstractPublication PagesaiprConference Proceedingsconference-collections
research-article

Multi-loss Super-Resolution Generative Adversarial Network

Published: 14 June 2024 Publication History

Abstract

Single-image super-resolution (SISR) based on deep neural networks has achieved excellent performance in recent years. However, how to recover texture details is still a challenging problem in the field of super-resolution. In this paper, in order to obtain reconstructed images with natural and realistic textures, we propose a network structure and a super-resolution reconstruction algorithm combining variational autoencoder (VAE) with generative adversarial network (GAN). This paper proposes optimized variational autoencoder (OVAE) as a discriminator to predict relative realness. The discriminator could project the input images into a latent space. And the latent space in OVAE could learn the latent distribution of high-resolution images. By learning the latent distribution of the input images, the discriminator can obtain high-frequency features and contextual information. In addition, this paper uses Kullback-Leibler (KL) divergence to optimize the discriminator and constrain the data distribution. Finally, to achieve stable training, we improve GAN by using Wasserstein divergence for GANs (WGAN-div). The network structure acquires a sophisticated progressive growing training scheme and consistently achieves better visual quality. Experimental results have proven that the method in this paper outperforms several state-of-the-art methods in both objective and subjective evaluation, and the proposed algorithm can efficiently reconstruct high-resolution images with natural and realistic textures.

References

[1]
J. Kim, J. K. Lee, and K. M. Lee. 2016. Deeply-recursive convolutional network for image super-resolution. In Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition. Las Vegas, NV, USA, 1637–1645. https://doi.org/10.1109/ CVPR.2016.181
[2]
Wenzhe Shi, Jose Caballero, Ferenc Huszár, and 2016. Real-Time Single Image and Video Super-Resolution Using an Efficient Sub-Pixel Convolutional Neural Network. In Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition. Las Vegas, NV, USA, 1874–1883. https://doi.org/10.1109/CVPR.2016. 207
[3]
Chao Dong, Chen Change Loy, Kaiming He, and Xiaoou Tang. 2014. Learning a deep convolutional network for image super-resolution. [C]// European Conference on Computer Vision. Springer, Cham, 184–199. https://doi.org/10.1007/978-3-319-10593-2_13
[4]
Jiwon Kim, Jung Kwon Lee, and Kyoung Mu Lee. 2016. Accurate image super-resolution using very deep convolutional networks. In Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition. Las Vegas, NV, USA, 1646–1654. https://doi.org/10.1109/CVPR.2016.182
[5]
Wenzhe Shi, Jose Caballero, Ferenc Huszár, and 2016. Real-Time Single Image and Video Super-Resolution Using an Efficient Sub-Pixel Convolutional Neural Network. In Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition. Las Vegas, NV, USA, 1874–1883. https://doi.org/10.1109/CVPR.2016.207
[6]
B. Lim, S. Son, H. Kim, S. Nah, and K. Mu Lee. 2017. Enhanced deep residual networks for single image super-resolution. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops. Honolulu, HI, USA, 136–144. https://doi.org/10.1109/CVPRW.2017.151
[7]
K. He, X. Zhang, S. Ren, and J. Sun. 2016. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition. Las Vegas, NV, USA, 770–778. https://doi.org/10.1109/CVPR.2016.90
[8]
Yulun Zhang, Kunpeng Li, Kai Li, and 2018. Image super-resolution using very deep residual channel attention networks. In Proceedings of European Conference on Computer Vision. Munich, Germany, 294–310. https://doi.org/ 10.1007/978-3-030-01234-2_18
[9]
Y. Zhang, K. Li, K. Li, and 2018. Image super-resolution using very deep residual channel attention networks. In Proceedings of the European Conference on Computer Vision (September 2018), 286–301. https://doi.org/10.48550/arXiv.1807.02758
[10]
Christian Ledig, Lucas Theis, Ferenc Huszár, and 2017. Photo-realistic single image super-resolution using a generate adversarial network. In Proceedings of the 30th IEEE Conference on Computer Vision and Pattern Recognition. Honolulu,HI, USA, 105–114. https://doi.org/ 10.1109/CVPR.2017.19
[11]
I. Goodfellow, J. Pouget-Abadie, M. Mirza, and 2014. Generative adversarial nets. In Advances in neural information processing systems. vol. 2, 2672–2680. https://doi.org/10.48550/arXiv.1406.2661
[12]
X. Wang, K. Yu, S. Wu, and 2018. ESRGAN: Enhanced super-resolution generative adversarial networks. In European Conference on Computer Vision. ECCV Workshops, 63–79. https://doi.org/10.48550/arXiv.1809.00219
[13]
A. Jolicoeur-Martineau. 2019. The relativistic discriminator: a key element missing from standard GAN. In International Conference on Learning Representations. 1–26. https://doi.org/10.48550/arXiv.1807.00734
[14]
Zhi-Song Liu, Wan-Chi Siu, and Yui-Lam Chan. 2020. Photo-realistic image super-resolution via variational autoencoders. IEEE Transactions on Circuits and Systems for Video Technology. vol. 31, no. 4, 1351–1365. https://doi.org/10.1109/TCSVT.2020.3003832
[15]
Zhi-Song Liu, Wan-Chi Siu, Wang Li-Wen, and 2020. Unsupervised real image super-resolution via generative variational autoencoder. In IEEE International Conference on Computer Vision and Pattern Recognition Workshop (CVPRW). Seattle, WA, USA, 1788–1797. https://doi.org/10.1109/CVPRW50498.2020.00229
[16]
Jesse Engel, Matthew Hoffman, and Adam Roberts. 2017. Latent constraints: Learning to generate conditionally from unconditional generative models. CoRR, 2017.2. https://doi.org/10.48550/arXiv.1711.05772
[17]
Z. -S. Liu, W. -C. Siu and L. -W. Wang. 2021. Variational AutoEncoder for Reference based Image Super-Resolution. In IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops. Nashville, TN, USA, 516-525. https://doi.org/10.1109/ CVPRW53098.2021.00063
[18]
Wu, Jiqing, Zhiwu Huang, Janine Thoma, and 2018. Wasserstein Divergence for GANs. European Conference on Computer Vision. LNIP, volume 11209. https://doi.org/10.48550/arXiv.1712.01026
[19]
A. Paszke, S. Gross, S. Chintala, and 2017. Automatic differentiation in pytorch. In Neural Information Processing Systems Workshop.1–4
[20]
Radu Timofte, Eirikur Agustsson, Luc Van Gool, and 2017. Ntire 2017 challenge on single image super-resolution: Methods and results. In Proceedings of 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW). Honolulu, HI, USA, 1110-1121. https://doi.org/10.1109/CVPRW.2017.149
[21]
M Bevilacqua, A Roumy, C Guillemot, and A Morel. 2012. Low-complexity single-image super-resolution based on nonnegative neighbor embedding. In Proceedings British Machine Vision Conference 2012. 135.1–135.10. https://doi.org/10.5244/C.26.135
[22]
R Zeyde, M Elad, and M Protter. 2019. On single image scale-up using sparse-representations. In International Conference on Curves and Surfaces. Springer, Berlin, Heidelberg, 711–730. https://doi.org/10.1007/978-3-642-27413-8_47
[23]
D. Martin, C. Fowlkes, D. Tal, and J. Malik. 2001. A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics. In Proceedings Eighth IEEE International Conference on Computer Vision. Vancouver, BC, Canada, 416–423. https://doi.org/10.1109/ICCV.2001.937655
[24]
Jia Bin Huang, Abhishek Singh, and Narendra Ahuja. 2015. Single image super-resolution from transformed self-exemplars. In Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition. IEEE, Boston, MA, USA, 5197–5206. https://doi.org/10.1109/CVPR.2015.7299156
[25]
Z. Wang, A. C. Bovik, H. R. Sheikh, E. P. Simoncelli 2004. Image quality assessment: from error visibility to structural similarity. IEEE transactions on image processing, vol. 13, no. 4, pp. 600–612, 2004. https://doi.org/10.1109/TIP.2003.819861

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Other conferences
AIPR '23: Proceedings of the 2023 6th International Conference on Artificial Intelligence and Pattern Recognition
September 2023
1540 pages
ISBN:9798400707674
DOI:10.1145/3641584
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 14 June 2024

Permissions

Request permissions for this article.

Check for updates

Author Tags

  1. Generative adversarial network
  2. KL divergence
  3. Optimized variational autoencoder
  4. Single-image super-resolution
  5. Wasserstein divergence for GANs

Qualifiers

  • Research-article
  • Research
  • Refereed limited

Conference

AIPR 2023

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • 0
    Total Citations
  • 7
    Total Downloads
  • Downloads (Last 12 months)7
  • Downloads (Last 6 weeks)2
Reflects downloads up to 20 Jan 2025

Other Metrics

Citations

View Options

Login options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

HTML Format

View this article in HTML Format.

HTML Format

Media

Figures

Other

Tables

Share

Share

Share this Publication link

Share on social media