Image Super-Resolution with Perceptual Quality Assessment Guidance
Pages 111 - 118
Abstract
Generative Adversarial Networks (GAN) have demonstrated the potential to recover realistic details for single image super-resolution (SISR). However, without considering perceptual metrics in optimization, existing perceptual SR methods could not show stable performance. To further improve the visual quality of super-resolution results, we propose a SISR method driven by image perceptual quality assessment. Through evaluating the generation results of GAN-based SISR and utilizing the evaluation results as the loss function, the generator can be optimized to obtain results with better perceptual quality. Specifically, we use a distorted dataset consisting of images generated by traditional and GAN-based image enhancement algorithms, and present an image perceptual quality assessment method which is highly consistent with human subjective evaluation. Then, we evaluate the generation results of GAN-based SISR and the evaluation result is utilized as the quality constraint to guide SISR model to obtain better subjective perception generation results. Although the proposed method is simple, it is effective and can be easily inserted into the existing SISR method. Extensive experiments show that the proposed method can produce visually pleasing results and achieve the state-of-the-art (SOTA) performance in perceptual metrics.
References
[1]
Chao Dong, Chen Change Loy, Kaiming He, and Xiaoou Tang. 2014. Learning a deep convolutional network for image super-resolution. In European Conference on Computer Vision (ECCV), pp. 184–199. https://doi.org/10.1007/978-3-319-10593-2_13
[2]
Jiwon Kim, Jung Kwon Lee, and Kyoung Mu Lee. 2016. Accurate image super-resolution using very deep convolutional networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1646–1654.
[3]
Jiwon Kim, Jung Kwon Lee, and Kyoung Mu Lee. 2016. Deeply-recursive convolutional network for image super-resolution. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1637–1645.
[4]
Yulun Zhang, Yapeng Tian, Yu Kong, Bineng Zhong, and Yun Fu. 2018. Residual dense network for image super-resolution. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2472–2481.
[5]
Yulun Zhang, Kunpeng Li, Kai Li, Lichen Wang, Bineng Zhong, and Yun Fu. 2018. Image super-resolution using very deep residual channel attention networks. In European Conference on Computer Vision (ECCV), pp. 286–301. https://doi.org/10.1007/978-3-030-01234-2_18
[6]
Zhen Li, Jinglei Yang, Zheng Liu, Xiaomin Yang, Gwanggil Jeon, and Wei Wu. 2019. Feedback network for image super-resolution. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3867–3876.
[7]
Nan Nan, Shijie Yue, Ruipeng Gang, LI, and Ruixia Song. 2022. Image super-resolution algorithm based on v-transform combined with neural network. In Proceedings of the 5th International Conference on Image and Graphics Processing (ICIGP), pp. 339–345. https://doi.org/10.1145/3512388.3512437
[8]
Yabo Duan, Huaizhan Li, Kefei Zhang, Shubi Zhang, and Suqin Wu. 2022. Channel-spatial attention network for lunar image super-resolution. In Proceedings of the 5th International Conference on Image and Graphics Processing (ICIGP), pp.333–338. https://doi.org/10.1145/3512388.3512436
[9]
Christian Ledig, Lucas Theis, Ferenc Huszár, Jose Caballero, Andrew Cunningham, Alejandro Acosta, Andrew Aitken, Alykhan Tejani, Johannes Totz, Zehan Wang. 2017. Photo-realistic single image super-resolution using a generative adversarial network. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 4681–4690.
[10]
Xintao Wang, Ke Yu, Shixiang Wu, Jinjin Gu, Yihao Liu, Chao Dong, Yu Qiao, and Chen Change Loy. 2018. Esrgan: Enhanced super-resolution generative adversarial networks. In Proceedings of the European conference on computer vision (ECCV) workshops, pp. 63–79. https://doi.org/10.1007/978-3-030-11021-5_5
[11]
Wenlong Zhang, Yihao Liu, Chao Dong, and Yu Qiao. 2019. Ranksrgan: Generative adversarial networks with ranker for image super-resolution. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 3096–3105.
[12]
Cheng Ma, Yongming Rao, Yean Cheng, Ce Chen, Jiwen Lu, and Jie Zhou. 2020. Structure-preserving super resolution with gradient guidance. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 7769–7778.
[13]
Wenbo Li, Kun Zhou, Lu Qi, Liying Lu, and Jiangbo Lu. 2022. Best-buddy gans for highly detailed image super-resolution. In Proceedings of the AAAI Conference on Artificial Intelligence (AAAI), Vol. 36. pp. 1412–1420.
[14]
Alexia Jolicoeur-Martineau. 2018. The relativistic discriminator: a key element missing from standard GAN.
[15]
Justin Johnson, Alexandre Alahi, and Li Fei-Fei. 2016. Perceptual losses for real-time style transfer and super-resolution. In Proceedings of the European Conference on Computer Vision (ECCV). pp. 694–711. https://doi.org/10.1007/978-3-319-46475-6_43
[16]
Bo Yan, Bahetiyaer Bare, Chenxi Ma, Kemeng Li, and Weimin Tan. 2019. Deep objective quality assessment driven single image super-resolution. IEEE Transactions on Multimedia. Vol. 21, No. 11, pp. 2957–2971. https://doi.org/10.1109/TMM.2019.2914883
[17]
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. 2021. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF International Conference on Computer Vision (CVPR), pp. 10012–10022.
[18]
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. 2014. Generative adversarial nets. Communications of the ACM, Vol. 63, No. 11, pp. 139–144. https://doi.org/10.1145/3422622
[19]
Ishaan Gulrajani, Faruk Ahmed, Martin Arjovsky, Vincent Dumoulin, and Aaron C Courville. 2017. Improved training of wasserstein gans. In Proceedings of the Neural Information Processing Systems (NeurIPS), pp. 5767–5777.
[20]
Alec Radford, Luke Metz, and Soumith Chintala. 2015. Unsupervised representation learning with deep convolutional generative adversarial networks. arXiv preprint arXiv:1511.06434 (2015).
[21]
Gu Jinjin, Cai Haoming, Chen Haoyu, Ye Xiaoxing, Jimmy S Ren, and Dong Chao. 2020. Pipal: a large-scale image quality assessment dataset for perceptual image restoration. In European Conference on Computer Vision (ECCV), pp. 633–651. https://doi.org/10.1007/978-3-030-58621-8_37
[22]
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. 2009. Imagenet: A large-scale hierarchical image database. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 248–255.
[23]
Diederik P. Kingma and Jimmy Ba. 2014. Adam: A Method for Stochastic Optimization. arXiv preprint arXiv:1412.6980 (2014).
[24]
Eirikur Agustsson and Radu Timofte. 2017. Ntire 2017 challenge on single image super-resolution: Dataset and study. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 126–135.
[25]
David Martin, Charless Fowlkes, Doron Tal, and Jitendra Malik. 2001. A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), Vol. 2, pp.416–423.
[26]
Chao Dong, Chen Change Loy, and Xiaoou Tang. 2016. Accelerating the super-resolution convolutional neural network. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 391–407. https://doi.org/10.1007/978-3-319-46475-6_25
[27]
Jia-Bin Huang, Abhishek Singh, and Narendra Ahuja. 2015. Single image super-resolution from transformed self-exemplars. In Proceedings of the IEEE/CVF International Conference on Computer Vision (CVPR), pp. 5197–5206.
[28]
Xintao Wang, K. Yu, Chao Dong, and Chen Change Loy. 2018. Recovering realistic texture in image super-resolution by deep spatial feature transform. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 606–615.
[29]
Yochai Blau, Roey Mechrez, Radu Timofte, Tomer Michaeli, and Lihi Zelnik-Manor. 2018. The 2018 PIRM Challenge on Perceptual Image Super-Resolution. In Proceedings of the European Conference on Computer Vision (ECCV), pp.334–355 https://doi.org/334–355.10.1007/978-3-030-11021-5_21
[30]
Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. 2018. The unreasonable effectiveness of deep features as a perceptual metric. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 586–595.
[31]
Keyan Ding, Kede Ma, Shiqi Wang, and Eero P. Simoncelli. 2022. Image quality assessment: Unifying structure and texture similarity. IEEE Transactions on Pattern Analysis and Machine Intelligence. Vol. 44, No. 5, pp. 2567–2581. https://doi.org/10.1109/TPAMI.2020.3045810
[32]
Anish Mittal, Rajiv Soundararajan, and Alan Conrad Bovik. 2013. Making a “completely blind” image quality analyzer. IEEE Signal Processing Letters. Vol. 20, No. 3, pp. 209–212. https://doi.org/10.1109/LSP.2012.2227726
[33]
Chao Ma, Chih-Yuan Yang, Xiaokang Yang, and Ming-Hsuan Yang. 2017. Learning a no-reference quality metric for single-image super-resolution. Computer Vision and Image Understanding. Vol. 158, pp. 1–16. https://doi.org/10.1016/j.cviu.2016.12.009
[34]
Yochai Blau and Tomer Michaeli. 2018. The perception-distortion tradeoff. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 6228–6237.
[35]
Zheng Hui, Xinbo Gao, Yunchu Yang, and Xiumei Wang. 2019. Lightweight image super-resolution with information multi-distillation network. In Proceedings of the 27th ACM International Conference on Multimedia, pp. 2024–2032.
[36]
Jingyun Liang, Jiezhang Cao, Guolei Sun, Kai Zhang, Luc Van Gool, and Radu Timofte. 2021. Swinir: Image restoration using swin transformer. In Proceedings of the IEEE/CVF International Conference on Computer Vision (CVPR), pp. 1833–1844.
[37]
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 770–778.
[38]
Karen Simonyan and Andrew Zisserman. 2014. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556 (2014).
[39]
K. Zarei Zefreh, W. van Aarle, K. J. Batenburg, and J. Sijbers. 2013. Super-resolution of license plate images using algebraic reconstruction technique. Journal of Image and Graphics, Vol. 1, No. 2, pp. 94-98. https://doi.org/10.12720/joig.1.2.94-98
[40]
Suhail Hamdan, Yohei Fukumizu, Tomonori Izumi, and Hironori Yamauchi. 2018. Face image super-resolution with adaptive patch size to scaling factor. Journal of Image and Graphics, Vol. 6, No. 2, pp. 167-173. https://doi.org/10.18178/joig.6.2.167-173
[41]
Keigo Kano, Tomio Goto, and Satoshi Hirano. 2019. Performance improvement of face image super-resolution processing by high-precision skin color detection. Journal of Image and Graphics, Vol. 7, No. 3, pp. 82-89. https://doi.org/10.18178/joig.7.3.82-89
[42]
Hwei Jen Lin, Yoshimasa Tokuyama, and Zi Jun Lin. 2019. Residual learning based convolutional neural network for super resolution," Journal of Image and Graphics, Vol. 7, No. 4, pp. 126-129. https://doi.org/10.18178/joig.7.4.126-129
[43]
Ying Liu, Yangge Qiao, Yu Hao, Fuping Wang, and Sheikh Faisal Rashid. 2021. Single image super resolution techniques based on deep learning: Status, applications and future directions. Journal of Image and Graphics, Vol. 9, No. 3, pp. 74-86, September 2021. https://doi.org/10.18178/joig.9.3.74-86
[44]
Yoshitaka Izumi, Dan Suto, Sota Kawakami, and Hiroyuki Kudo. 2022. Blind image restoration and super-resolution for multispectral images using sparse optimization. Journal of Image and Graphics, Vol. 10, No. 1, pp. 10-16. https://doi.org/ 10.18178/joig.10.1.10-1
Index Terms
- Image Super-Resolution with Perceptual Quality Assessment Guidance
Recommendations
Super-resolution Quality Criterion (SRQC): a super-resolution image quality assessment metric
AbstractRecently, image super-resolution has reinforced image resolution enhancement approaches in real-time and ensuring visual quality of super resolved images has evolved as a key research problem. Most quantitative benchmarks rely on full reference ...
Single Image Super-Resolution Reconstruction Technique based on A Single Hybrid Dictionary
A new sparse domain approach is proposed in this paper to realize the single image super-resolution (SR) reconstruction based upon one single hybrid dictionary, which is deduced from the mixture of both the high resolution (HR) image patch samples and ...
Wavelet detail perception network for single image super-resolution
Highlights- A novel WDPNet is proposed to effectively solve the smooth problem of SR image details with novel L2HID and DPE mechanisms.
- Low- and high-frequency branches recover low-frequency structure and high-frequency details, respectively.
- ...
AbstractSingle image super-resolution (SR) is an important topic in computer vision because of its ability to generate high-resolution (HR) images. Traditional SR methods do not pay attention to high-frequency detail perception in the reconstruction ...
Comments
Information & Contributors
Information
Published In
January 2023
246 pages
ISBN:9781450398572
DOI:10.1145/3582649
Copyright © 2023 ACM.
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 07 April 2023
Check for updates
Author Tags
Qualifiers
- Research-article
- Research
- Refereed limited
Funding Sources
- National Natural Science Foundation of China
- Key R&D and Transformation Program of Qinghai Province
Conference
ICIGP 2023
ICIGP 2023: 2023 The 6th International Conference on Image and Graphics Processing
January 6 - 8, 2023
Chongqing, China
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 53Total Downloads
- Downloads (Last 12 months)18
- Downloads (Last 6 weeks)7
Reflects downloads up to 27 Feb 2025
Other Metrics
Citations
View Options
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign inFull Access
View options
View or Download as a PDF file.
PDFeReader
View online with eReader.
eReaderHTML Format
View this article in HTML Format.
HTML Format