Skip to main content
SpringerLink
Log in

Unsupervised Real-World Super-resolution Using Variational Auto-encoder and Generative Adversarial Network

  • Conference paper
  • First Online:
Pattern Recognition. ICPR International Workshops and Challenges (ICPR 2021)

Part of the book series: Lecture Notes in Computer Science ((LNIP,volume 12662))

Included in the following conference series:

Abstract

Convolutional Neural Networks (CNNs) have shown promising results on Single Image Super-Resolution (SISR) task. A pair of Low-Resolution (LR) and High-Resolution (HR) images are typically used in the CNN models to train them to super-resolve LR images in a fully supervised manner. Owing to non-availability of true LR-HR pairs, the LR images are generally synthesized from HR data by applying synthetic degradation such as bicubic downsampling. Such networks under-perform when used on real-world data where degradation is different from the synthetically generated LR image. As obtaining true LR-HR pair is a tedious and resource (time and effort) consuming task, we propose a new approach and architecture to super-resolve the real-world LR images in an unsupervised manner by using a Generative Adversarial Network (GAN) framework with Variational Auto-Encoder (VAE). Along with a new network architecture, we also introduce a novel loss metric based on no-reference quality scores of SR images to improve the perceptual fidelity of the SR images. Through the experiments on NTIRE-2020 Real-World SR Challenge dataset, we demonstrate the superiority of the proposed approach over the other competing state-of-the-art methods.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 84.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 109.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Agustsson, E., Timofte, R.: NTIRE 2017 challenge on single image super-resolution: dataset and study. In: 2017 IEEE Conference on CVPRW, pp. 1122–1131 (2017)

    Google Scholar 

  2. Cai, J., Zeng, H., Yong, H., Cao, Z., Zhang, L.: Toward real-world single image super-resolution: a new benchmark and a new model. In: ICCV, pp. 3086–3095, October 2019

    Google Scholar 

  3. Cheng, G., Matsune, A., Li, Q., Zhu, L., Zang, H., Zhan, S.: Encoder-decoder residual network for real super-resolution. In: CVPR Workshops, June 2019

    Google Scholar 

  4. Dong, C., Loy, C.C., He, K., Tang, X.: Image super-resolution using deep convolutional networks. IEEE Trans. PAMI 38(2), 295–307 (2016)

    Article  Google Scholar 

  5. Dong, C., Loy, C.C., Tang, X.: Accelerating the super-resolution convolutional neural network. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9906, pp. 391–407. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-46475-6_25

    Chapter  Google Scholar 

  6. Du, C., Zewei, H., Anshun, S., et al.: Orientation-aware deep neural network for real image super-resolution. In: The IEEE Conference on CVPRW, June 2019

    Google Scholar 

  7. Goodfellow, I., Pouget-Abadie, J., Mirza, M., et al.: Generative adversarial nets. In: Ghahramani, Z., Welling, M., Cortes, C., Lawrence, N.D., Weinberger, K.Q. (eds.) Advances in Neural Information Processing Systems, vol. 27, pp. 2672–2680. Curran Associates, Inc. (2014)

    Google Scholar 

  8. Jolicoeur-Martineau, A.: The relativistic discriminator: a key element missing from standard GAN. arXiv preprint arXiv:1807.00734 (2018)

  9. Kim, J., Lee, J.K., Lee, K.M.: Accurate image super-resolution using very deep convolutional networks. In: 2016 IEEE CVPR, pp. 1646–1654 (2016)

    Google Scholar 

  10. Kingma, D.P., Welling, M.: Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114 (2013)

  11. Ledig, C., Theis, L., Huszár, F., et al.: Photo-realistic single image super-resolution using a generative adversarial network. In: Proceedings IEEE Conference on CVPR, pp. 4681–4690 (2017)

    Google Scholar 

  12. Li, Y., Agustsson, E., Gu, S., Timofte, R., Van Gool, L.: CARN: convolutional anchored regression network for fast and accurate single image super-resolution. In: Leal-Taixé, L., Roth, S. (eds.) ECCV 2018. LNCS, vol. 11133, pp. 166–181. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-11021-5_11

    Chapter  Google Scholar 

  13. Lim, B., Son, S., Kim, H., Nah, S., Lee, K.M.: Enhanced deep residual networks for single image super-resolution. In: 2017 IEEE Conference on CVPRW, pp. 1132–1140 (2017)

    Google Scholar 

  14. Lin, H., Hosu, V., Saupe, D.: KADID-10K: a large-scale artificially distorted IQA database. In: 2019 10th Conference QoMEX, pp. 1–3. IEEE (2019)

    Google Scholar 

  15. Lugmayr, A., Danelljan, M., Timofte, R.: Unsupervised learning for real-world super-resolution. In: ICCV Workshops (2019)

    Google Scholar 

  16. Lugmayr, A., Danelljan, M., Timofte, R., et al.: Aim 2019 challenge on real-world image super-resolution: methods and results. In: ICCV Workshops (2019)

    Google Scholar 

  17. Lugmayr, A., Danelljan, M., Timofte, R., et al.: NTIRE 2020 challenge on real-world image super-resolution: methods and results. In: CVPRW (2020)

    Google Scholar 

  18. Park, S.-J., Son, H., Cho, S., Hong, K.-S., Lee, S.: SRFeat: single image super-resolution with feature discrimination. In: Ferrari, V., Hebert, M., Sminchisescu, C., Weiss, Y. (eds.) ECCV 2018. LNCS, vol. 11220, pp. 455–471. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-01270-0_27

    Chapter  Google Scholar 

  19. Prajapati, K., Chudasama, V., Patel, H., Upla, K., Ramachandra, R., Raja, K., Busch, C.: Unsupervised single image super-resolution network (USISResNet) for real-world data using generative adversarial network. In: 2020 IEEE CVPR-W, pp. 1904–1913 (2020)

    Google Scholar 

  20. Radford, A., Metz, L., Chintala, S.: Unsupervised representation learning with deep convolutional generative adversarial networks. arXiv preprint arXiv:1511.06434 (2015)

  21. Shi, Y., Zhong, H., Yang, Z., Yang, X., Lin, L.: DDet: dual-path dynamic enhancement network for real-world image super-resolution. IEEE Signal Process. Lett. 27, 481–485 (2020)

    Article  Google Scholar 

  22. Shocher, A., Cohen, N., Irani, M.: Zero-shot super-resolution using deep internal learning. In: 2018 IEEE/CVF Conference on CVPR, pp. 3118–3126 (2018)

    Google Scholar 

  23. Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556 (2014)

  24. Spearman, C.: The proof and measurement of association between two things. In: Jenkins, J.J., Paterson, D.G. (eds.) Studies in individual differences: The search for intelligence, Appleton-Century-Crofts, pp. 45–58 (1961)

    Google Scholar 

  25. Timofte, R., Agustsson, E., Van Gool, L., Yang, M.H., Zhang, L.: NTIRE 2017 challenge on single image super-resolution: methods and results. In: Proceedings of IEEE Conference on CVPRW, pp. 114–125 (2017)

    Google Scholar 

  26. Tong, T., Li, G., Liu, X., Gao, Q.: Image super-resolution using dense skip connections. In: Proceedings of IEEE ICCV, pp. 4799–4807 (2017)

    Google Scholar 

  27. Wang, X., et al.: ESRGAN: enhanced super-resolution generative adversarial networks. In: Leal-Taixé, L., Roth, S. (eds.) ECCV 2018. LNCS, vol. 11133, pp. 63–79. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-11021-5_5

    Chapter  Google Scholar 

  28. Xu, X., Li, X.: Scan: spatial color attention networks for real single image super-resolution. In: The IEEE Conference on CVPRW, June 2019

    Google Scholar 

  29. Yuan, Y., Liu, S., Zhang, J., Zhang, Y., Dong, C., Lin, L.: Unsupervised image super-resolution using cycle-in-cycle generative adversarial networks. In: 2018 IEEE/CVF Conference on CVPRW, pp. 814–81409, June 2018

    Google Scholar 

  30. Zhang, K., Zuo, W., Zhang, L.: Learning a single convolutional super-resolution network for multiple degradations. In: Proceedings IEEE Conference on CVPR, pp. 3262–3271 (2018)

    Google Scholar 

  31. Zhang, R., Isola, P., Efros, A.A., Shechtman, E., Wang, O.: The unreasonable effectiveness of deep features as a perceptual metric. In: Proceedings of IEEE Conference on CVPR, pp. 586–595 (2018)

    Google Scholar 

  32. Zhang, Y., Li, K., Li, K., Wang, L., Zhong, B., Fu, Y.: Image super-resolution using very deep residual channel attention networks. In: Ferrari, V., Hebert, M., Sminchisescu, C., Weiss, Y. (eds.) ECCV 2018. LNCS, vol. 11211, pp. 294–310. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-01234-2_18

    Chapter  Google Scholar 

  33. Zhang, Y., Tian, Y., Kong, Y., Zhong, B., Fu, Y.: Residual dense network for image super-resolution. In: Proceedings of IEEE Conference on CVPR, pp. 2472–2481 (2018)

    Google Scholar 

Download references

Acknowledgment

This work is supported by ERCIM, who kindly enabled the internship of Kishor Upla at NTNU, Gjøvik. Authors are also thankful to Science and Engineering Research Board (SERB), a statutory body of Department of Science and Technology (DST), Government of India for providing support for this research work (ECR/2017/003268).

Author information

Authors and Affiliations

  1. Sardar Vallabhbhai National Institute of Technology (SVNIT), Surat, India

    Kalpesh Prajapati, Vishal Chudasama, Heena Patel & Kishor Upla

  2. Norwegian University of Science and Technology (NTNU), Gjøvik, Norway

    Kishor Upla, Kiran Raja, Raghavendra Ramachandra & Christoph Busch

Authors
  1. Kalpesh Prajapati
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Vishal Chudasama
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Heena Patel
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Kishor Upla
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Kiran Raja
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Raghavendra Ramachandra
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Christoph Busch
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Kishor Upla .

Editor information

Editors and Affiliations

  1. Dipartimento di Ingegneria dell’Informazione, University of Firenze, Firenze, Italy

    Alberto Del Bimbo

  2. Dipartimento di Ingegneria “Enzo Ferrari”, Università di Modena e Reggio Emilia, Modena, Italy

    Rita Cucchiara

  3. Department of Computer Science, Boston University, Boston, MA, USA

    Stan Sclaroff

  4. Dipartimento di Matematica e Informatica, University of Catania, Catania, Italy

    Giovanni Maria Farinella

  5. Cloud & AI, JD.COM, Beijing, China

    Tao Mei

  6. Dipartimento di Ingegneria dell’Informazione, Universita di Firenze, Firenze, Italy

    Marco Bertini

  7. Computational Sciences Department, National Institute of Astrophysics, Optics and Electronics (INAOE), Tonantzintla, Puebla, Mexico

    Hugo Jair Escalante

  8. Dipartimento di Ingegneria “Enzo Ferrari”, Università di Modena e Reggio Emilia, Modena, Italy

    Roberto Vezzani

Rights and permissions

Reprints and permissions

Copyright information

© 2021 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Prajapati, K. et al. (2021). Unsupervised Real-World Super-resolution Using Variational Auto-encoder and Generative Adversarial Network. In: Del Bimbo, A., et al. Pattern Recognition. ICPR International Workshops and Challenges. ICPR 2021. Lecture Notes in Computer Science(), vol 12662. Springer, Cham. https://doi.org/10.1007/978-3-030-68790-8_54

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-68790-8_54

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-68789-2

  • Online ISBN: 978-3-030-68790-8

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Societies and partnerships

Search

Navigation