skip to main content
10.1145/3474085.3479234acmconferencesArticle/Chapter ViewAbstractPublication PagesmmConference Proceedingsconference-collections
research-article

Recycling Discriminator: Towards Opinion-Unaware Image Quality Assessment Using Wasserstein GAN

Published: 17 October 2021 Publication History

Abstract

Generative adversarial networks (GANs) have been extensively used for training networks that perform image generation. After training, the discriminator in GAN was not used anymore. We propose to recycle the trained discriminator for another use: no-reference image quality assessment (NR-IQA). We are motivated by twofold facts. First, in Wasserstein GAN (WGAN), the discriminator is designed to calculate the distance between the distribution of generated images and that of real images; thus, the trained discriminator may encode the distribution of real-world images. Second, NR-IQA often needs to leverage the distribution of real-world images for assessing image quality. We then conjecture that using the trained discriminator for NR-IQA may help get rid of any human-labeled quality opinion scores and lead to a new opinion-unaware (OU) method. To validate our conjecture, we start from a restricted NR-IQA problem, that is IQA for artificially super-resolved images. We train super-resolution (SR) WGAN with two kinds of discriminators: one is to directly evaluate the entire image, and the other is to work on small patches. For the latter kind, we obtain patch-wise quality scores, and then have the flexibility to fuse the scores, e.g., by weighted average. Moreover, we directly extend the trained discriminators for authentically distorted images that have different kinds of distortions. Our experimental results demonstrate that the proposed method is comparable to the state-of-the-art OU NR-IQA methods on SR images and is even better than them on authentically distorted images. Our method provides a better interpretable approach to NR-IQA. Our code and models are available at https://github.com/YunanZhu/RecycleD.

Supplementary Material

ZIP File (bni3173aux.zip)
In this supplementary document, we first give more details about our generator and discriminator architecture. Then we briefly discuss the relation between the prediction performance and the training epochs. Lastly we present more cases to supplement the case study of the main text.

References

[1]
Eirikur Agustsson and Radu Timofte. 2017. NTIRE 2017 challenge on single image super-resolution: Dataset and study. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops. 126--135.
[2]
Martin Arjovsky, Soumith Chintala, and Léon Bottou. 2017. Wasserstein generative adversarial networks. In International Conference on Machine Learning. PMLR, 214--223.
[3]
Yochai Blau and Tomer Michaeli. 2018. The perception-distortion tradeoff. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 6228--6237.
[4]
Deepti Ghadiyaram and Alan C. Bovik. 2015. Massive online crowdsourced study of subjective and objective picture quality. IEEE Transactions on Image Processing, Vol. 25, 1 (2015), 372--387.
[5]
Ian J. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron C. Courville, and Yoshua Bengio. 2014. Generative Adversarial Nets. In NIPS.
[6]
Jinjin Gu, Haoming Cai, Haoyu Chen, Xiaoxing Ye, Jimmy Ren, and Chao Dong. 2020. Image Quality Assessment for Perceptual Image Restoration: A New Dataset, Benchmark and Metric. arxiv: 2011.15002 [eess.IV]
[7]
Jinjin Gu, Haoming Cai, Chao Dong, Jimmy S. Ren, Yu Qiao, Shuhang Gu, Radu Timofte, Manri Cheon, Sungjun Yoon, Byungyeon Kangg Kang, et almbox. 2021. NTIRE 2021 challenge on perceptual image quality assessment. arxiv: 2105.03072 [eess.IV]
[8]
Ke Gu, Shiqi Wang, Huan Yang, Weisi Lin, Guangtao Zhai, Xiaokang Yang, and Wenjun Zhang. 2016. Saliency-guided quality assessment of screen content images. IEEE Transactions on Multimedia, Vol. 18, 6 (2016), 1098--1110.
[9]
Ke Gu, Guangtao Zhai, Weisi Lin, Xiaokang Yang, and Wenjun Zhang. 2015. No-reference image sharpness assessment in autoregressive parameter space. IEEE Transactions on Image Processing, Vol. 24, 10 (2015), 3218--3231.
[10]
Ke Gu, Guangtao Zhai, Xiaokang Yang, and Wenjun Zhang. 2014. Hybrid no-reference quality metric for singly and multiply distorted images. IEEE Transactions on Broadcasting, Vol. 60, 3 (2014), 555--567.
[11]
Ke Gu, Jun Zhou, Jun-Fei Qiao, Guangtao Zhai, Weisi Lin, and Alan Conrad Bovik. 2017. No-reference quality assessment of screen content pictures. IEEE Transactions on Image Processing, Vol. 26, 8 (2017), 4005--4018.
[12]
Ishaan Gulrajani, Faruk Ahmed, Martin Arjovsky, Vincent Dumoulin, and Aaron Courville. 2017. Improved training of wasserstein GANs. In Proceedings of the 31st International Conference on Neural Information Processing Systems. 5769--5779.
[13]
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition. 770--778.
[14]
Vlad Hosu, Hanhe Lin, Tamas Sziranyi, and Dietmar Saupe. 2020. KonIQ-10k: An ecologically valid database for deep learning of blind image quality assessment. IEEE Transactions on Image Processing, Vol. 29 (2020), 4041--4056.
[15]
Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A. Efros. 2017. Image-to-image translation with conditional adversarial networks. In Proceedings of the IEEE conference on computer vision and pattern recognition. 1125--1134.
[16]
Gu Jinjin, Cai Haoming, Chen Haoyu, Ye Xiaoxing, Jimmy S. Ren, and Dong Chao. 2020. PIPAL: a large-scale image quality assessment dataset for perceptual image restoration. In European Conference on Computer Vision. Springer, 633--651.
[17]
Maurice G. Kendall. 1938. A new measure of rank correlation. Biometrika, Vol. 30, 1/2 (1938), 81--93.
[18]
Diederik P. Kingma and Jimmy Ba. 2017. Adam: A Method for Stochastic Optimization. arxiv: 1412.6980 [cs.LG]
[19]
Orest Kupyn, Volodymyr Budzan, Mykola Mykhailych, Dmytro Mishkin, and Jiři Matas. 2018. Deblurgan: Blind motion deblurring using conditional adversarial networks. In Proceedings of the IEEE conference on computer vision and pattern recognition. 8183--8192.
[20]
Christian Ledig, Lucas Theis, Ferenc Huszár, Jose Caballero, Andrew Cunningham, Alejandro Acosta, Andrew Aitken, Alykhan Tejani, Johannes Totz, Zehan Wang, et al. 2017. Photo-realistic single image super-resolution using a generative adversarial network. In Proceedings of the IEEE conference on computer vision and pattern recognition. 4681--4690.
[21]
Ang Li, Jichun Li, Qing Lin, Chenxi Ma, and Bo Yan. 2020. Deep Image Quality Assessment Driven Single Image Deblurring. In 2020 IEEE International Conference on Multimedia and Expo (ICME). IEEE, 1--6.
[22]
Bee Lim, Sanghyun Son, Heewon Kim, Seungjun Nah, and Kyoung Mu Lee. 2017. Enhanced deep residual networks for single image super-resolution. In Proceedings of the IEEE conference on computer vision and pattern recognition workshops. 136--144.
[23]
Kwan-Yee Lin and Guanxiang Wang. 2018. Hallucinated-IQA: No-reference image quality assessment via adversarial learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 732--741.
[24]
Xialei Liu, Joost Van De Weijer, and Andrew D. Bagdanov. 2017. RankIQA: Learning from rankings for no-reference image quality assessment. In Proceedings of the IEEE International Conference on Computer Vision. 1040--1049.
[25]
Yutao Liu, Ke Gu, Yongbing Zhang, Xiu Li, Guangtao Zhai, Debin Zhao, and Wen Gao. 2019. Unsupervised blind image quality evaluation via statistical measurements of structure, naturalness, and perception. IEEE Transactions on Circuits and Systems for Video Technology, Vol. 30, 4 (2019), 929--943.
[26]
Chao Ma, Chih-Yuan Yang, Xiaokang Yang, and Ming-Hsuan Yang. 2017b. Learning a no-reference quality metric for single-image super-resolution. Computer Vision and Image Understanding, Vol. 158 (2017), 1--16.
[27]
Jupo Ma, Jinjian Wu, Leida Li, Weisheng Dong, and Xuemei Xie. 2020. Active Inference of GAN for No-Reference Image Quality Assessment. In 2020 IEEE International Conference on Multimedia and Expo (ICME). IEEE, 1--6.
[28]
Kede Ma, Wentao Liu, Tongliang Liu, Zhou Wang, and Dacheng Tao. 2017a. dipIQ: Blind image quality assessment by learning-to-rank discriminable image pairs. IEEE Transactions on Image Processing, Vol. 26, 8 (2017), 3951--3964.
[29]
Kede Ma, Xuelin Liu, Yuming Fang, and Eero P. Simoncelli. 2019. Blind image quality assessment by learning from multiple annotators. In 2019 IEEE International Conference on Image Processing (ICIP). IEEE, 2344--2348.
[30]
David Martin, Charless Fowlkes, Doron Tal, and Jitendra Malik. 2001. A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics. In Proceedings Eighth IEEE International Conference on Computer Vision. ICCV 2001, Vol. 2. IEEE, 416--423.
[31]
Xiongkuo Min, Ke Gu, Guangtao Zhai, Jing Liu, Xiaokang Yang, and Chang Wen Chen. 2017. Blind quality assessment based on pseudo-reference image. IEEE Transactions on Multimedia, Vol. 20, 8 (2017), 2049--2062.
[32]
Anish Mittal, Rajiv Soundararajan, and Alan C. Bovik. 2012. Making a "completely blind" image quality analyzer. IEEE Signal Processing Letters, Vol. 20, 3 (2012), 209--212.
[33]
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et almbox. 2019. PyTorch: An Imperative Style, High-Performance Deep Learning Library. Advances in Neural Information Processing Systems, Vol. 32 (2019), 8026--8037.
[34]
Xuebin Qin, Zichen Zhang, Chenyang Huang, Masood Dehghan, Osmar R Zaiane, and Martin Jagersand. 2020. U2-Net: Going deeper with nested U-structure for salient object detection. Pattern Recognition, Vol. 106 (2020), 107404.
[35]
Xuebin Qin, Zichen Zhang, Chenyang Huang, Chao Gao, Masood Dehghan, and Martin Jagersand. 2019. Basnet: Boundary-aware salient object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 7479--7489.
[36]
Muhammad Ali Qureshi, Mohamed Deriche, Azeddine Beghdadi, and Asjad Amin. 2017. A critical survey of state-of-the-art image inpainting quality assessment metrics. Journal of Visual Communication and Image Representation, Vol. 49 (2017), 177--191.
[37]
Hongyu Ren, Diqi Chen, and Yizhou Wang. 2018. RAN4IQA: restorative adversarial nets for no-reference image quality assessment. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 32.
[38]
Mehdi SM Sajjadi, Bernhard Scholkopf, and Michael Hirsch. 2017. Enhancenet: Single image super-resolution through automated texture synthesis. In Proceedings of the IEEE International Conference on Computer Vision. 4491--4500.
[39]
Xintao Wang, Ke Yu, Shixiang Wu, Jinjin Gu, Yihao Liu, Chao Dong, Yu Qiao, and Chen Change Loy. 2018. ESRGAN: Enhanced Super-Resolution Generative Adversarial Networks. In Proceedings of the European Conference on Computer Vision (ECCV) Workshops.
[40]
Zhou Wang. 2011. Applications of objective image quality assessment methods [applications corner]. IEEE Signal Processing Magazine, Vol. 28, 6 (2011), 137--142.
[41]
Zhou Wang, H. R. Sheikh, and A. C. Bovik. 2002. No-reference perceptual quality assessment of JPEG compressed images. In Proceedings. International Conference on Image Processing, Vol. 1. I--I. https://doi.org/10.1109/ICIP.2002.1038064
[42]
Qingbo Wu, Zhou Wang, and Hongliang Li. 2015. A highly efficient method for blind image quality assessment. In 2015 IEEE International Conference on Image Processing (ICIP). IEEE, 339--343.
[43]
Wufeng Xue, Lei Zhang, and Xuanqin Mou. 2013. Learning without human scores for blind image quality assessment. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 995--1002.
[44]
Sheng Yang, Qiuping Jiang, Weisi Lin, and Yongtao Wang. 2019. Sgdnet: An end-to-end saliency-guided deep neural network for no-reference image quality assessment. In Proceedings of the 27th ACM International Conference on Multimedia. 1383--1391.
[45]
Xiaohan Yang, Fan Li, and Hantao Liu. 2020. TTL-IQA: Transitive Transfer Learning based No-reference Image Quality Assessment. IEEE Transactions on Multimedia (2020).
[46]
Peng Ye, Jayant Kumar, and David Doermann. 2014. Beyond human opinion scores: Blind image quality assessment based on synthetic scores. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 4241--4248.
[47]
Zili Yi, Qiang Tang, Shekoofeh Azizi, Daesik Jang, and Zhan Xu. 2020. Contextual residual aggregation for ultra high-resolution image inpainting. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 7508--7517.
[48]
Jiahui Yu, Zhe Lin, Jimei Yang, Xiaohui Shen, Xin Lu, and Thomas S. Huang. 2018. Generative image inpainting with contextual attention. In Proceedings of the IEEE conference on computer vision and pattern recognition. 5505--5514.
[49]
Lin Zhang, Lei Zhang, and Alan C. Bovik. 2015. A feature-enriched completely blind image quality evaluator. IEEE Transactions on Image Processing, Vol. 24, 8 (2015), 2579--2591.
[50]
Richard Zhang, Phillip Isola, Alexei A. Efros, Eli Shechtman, and Oliver Wang. 2018a. The unreasonable effectiveness of deep features as a perceptual metric. In Proceedings of the IEEE conference on computer vision and pattern recognition. 586--595.
[51]
Yi Zhang and Damon M. Chandler. 2018. Opinion-unaware blind quality assessment of multiply and singly distorted images via distortion parameter estimation. IEEE Transactions on Image Processing, Vol. 27, 11 (2018), 5433--5448.
[52]
Yulun Zhang, Kunpeng Li, Kai Li, Lichen Wang, Bineng Zhong, and Yun Fu. 2018b. Image super-resolution using very deep residual channel attention networks. In Proceedings of the European conference on computer vision (ECCV). 286--301.
[53]
Jun-Yan Zhu, Taesung Park, Phillip Isola, and Alexei A. Efros. 2017. Unpaired image-to-image translation using cycle-consistent adversarial networks. In Proceedings of the IEEE international conference on computer vision. 2223--2232.
[54]
Daniel Zoran and Yair Weiss. 2009. Scale invariance and noise in natural images. In 2009 IEEE 12th International Conference on Computer Vision. IEEE, 2209--2216.

Cited By

View all
  • (2024)Opinion Unaware Image Quality Assessment via Adversarial Convolutional Variational Autoencoder2024 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV)10.1109/WACV57701.2024.00215(2142-2152)Online publication date: 3-Jan-2024
  • (2024)Survey on Visual Signal Coding and Processing With Generative Models: Technologies, Standards, and OptimizationIEEE Journal on Emerging and Selected Topics in Circuits and Systems10.1109/JETCAS.2024.340352414:2(149-171)Online publication date: Jun-2024
  • (2024)No-reference quality assessment for underwater imagesComputers and Electrical Engineering10.1016/j.compeleceng.2024.109293118(109293)Online publication date: Aug-2024
  • Show More Cited By

Index Terms

  1. Recycling Discriminator: Towards Opinion-Unaware Image Quality Assessment Using Wasserstein GAN

      Recommendations

      Comments

      Information & Contributors

      Information

      Published In

      cover image ACM Conferences
      MM '21: Proceedings of the 29th ACM International Conference on Multimedia
      October 2021
      5796 pages
      ISBN:9781450386517
      DOI:10.1145/3474085
      Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

      Sponsors

      Publisher

      Association for Computing Machinery

      New York, NY, United States

      Publication History

      Published: 17 October 2021

      Permissions

      Request permissions for this article.

      Check for updates

      Author Tags

      1. image quality assessment
      2. no-reference
      3. opinion-unaware
      4. wasserstein gan (wgan).

      Qualifiers

      • Research-article

      Funding Sources

      • Natural Science Foundation of China

      Conference

      MM '21
      Sponsor:
      MM '21: ACM Multimedia Conference
      October 20 - 24, 2021
      Virtual Event, China

      Acceptance Rates

      Overall Acceptance Rate 2,145 of 8,556 submissions, 25%

      Contributors

      Other Metrics

      Bibliometrics & Citations

      Bibliometrics

      Article Metrics

      • Downloads (Last 12 months)38
      • Downloads (Last 6 weeks)3
      Reflects downloads up to 20 Feb 2025

      Other Metrics

      Citations

      Cited By

      View all
      • (2024)Opinion Unaware Image Quality Assessment via Adversarial Convolutional Variational Autoencoder2024 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV)10.1109/WACV57701.2024.00215(2142-2152)Online publication date: 3-Jan-2024
      • (2024)Survey on Visual Signal Coding and Processing With Generative Models: Technologies, Standards, and OptimizationIEEE Journal on Emerging and Selected Topics in Circuits and Systems10.1109/JETCAS.2024.340352414:2(149-171)Online publication date: Jun-2024
      • (2024)No-reference quality assessment for underwater imagesComputers and Electrical Engineering10.1016/j.compeleceng.2024.109293118(109293)Online publication date: Aug-2024
      • (2023)Neural Image Popularity Assessment with Retrieval-augmented TransformerProceedings of the 31st ACM International Conference on Multimedia10.1145/3581783.3611918(2427-2436)Online publication date: 26-Oct-2023
      • (2023)Rectified Wasserstein Generative Adversarial Networks for Perceptual Image RestorationIEEE Transactions on Pattern Analysis and Machine Intelligence10.1109/TPAMI.2022.318531645:3(3648-3663)Online publication date: 1-Mar-2023
      • (2023)On the Effectiveness of Spectral Discriminators for Perceptual Quality Improvement2023 IEEE/CVF International Conference on Computer Vision (ICCV)10.1109/ICCV51070.2023.01218(13197-13206)Online publication date: 1-Oct-2023

      View Options

      Login options

      View options

      PDF

      View or Download as a PDF file.

      PDF

      eReader

      View online with eReader.

      eReader

      Figures

      Tables

      Media

      Share

      Share

      Share this Publication link

      Share on social media