skip to main content
10.1145/3689093.3689182acmconferencesArticle/Chapter ViewAbstractPublication PagesmmConference Proceedingsconference-collections
research-article
Open access

A Metric for Evaluating Image Quality Difference Perception Ability in Blind Image Quality Assessment Models

Published: 28 October 2024 Publication History

Abstract

Blind Image Quality Assessment (BIQA) is a challenging research area essential for preprocessing and optimizing visual tasks like semantic recognition and image restoration. Due to the high cost of subjective experiments and the complexity of IQA data, BIQA remains a small-sample learning problem. Transfer learning methods have been introduced to address this. With advancements from VGG to Swin Transformer, new semantic backbones are continuously proposed. This paper is interested in whether the backbones with improving performance in semantic recognition also enhance predictive accuracy in BIQA tasks. However, comparative experiments showed that different semantic backbones, using appropriate pipelines, exhibit minimal differences in PLCC and SRCC, making it hard to identify the superior model. To resolve this, we propose a novel model comparison method, IQDP, based on the model falsification method. IQDP experiments revealed that models with similar accuracy can differ significantly in perceiving image quality differences, which traditional PLCC and SRCC struggle to capture. Based on this, we further implemented a new metric, Image Quality Difference Perception Ability, to supplement the traditional PLCC and SRCC, providing an effective means of identifying superior models.

References

[1]
Sebastian Bosse, Dominique Maniry, Klaus-Robert Müller, Thomas Wiegand, and Wojciech Samek. 2017. Deep neural networks for no-reference and full-reference image quality assessment. IEEE Trans. Image Process., Vol. 27, 1 (2017), 206--219.
[2]
Kenneth P Burnham and David R Anderson. 2002. Model selection and multimodel inference: a practical information-theoretic approach. Springer.
[3]
Tai-Yin Chiu, Yinan Zhao, and Danna Gurari. 2020. Assessing image quality issues for real-world problems. In Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. 3646--3656.
[4]
Subhayan De, Patrick T Brewick, Erik A Johnson, and Steven F Wojtkiewicz. 2018. Investigation of model falsification using error and likelihood bounds with application to a structural system. J. Eng. Mech., Vol. 144, 9 (2018), 04018078.
[5]
Xinbo Gao, Fei Gao, Dacheng Tao, and Xuelong Li. 2013. Universal blind image quality assessment metrics via natural scene statistics and multiple kernel learning. IEEE Trans. Neural Netw. Learn. Syst., Vol. 24, 12 (2013).
[6]
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recognition. In Proc. IEEE Conf. Comput. Vis. Pattern Recognit. 770--778.
[7]
Vlad Hosu, Hanhe Lin, Tamas Sziranyi, and Dietmar Saupe. 2020. KonIQ-10k: An ecologically valid database for deep learning of blind image quality assessment. IEEE Trans. Image Process., Vol. 29 (2020), 4041--4056.
[8]
International Telecommunication Union. 2023. Methodology for the subjective assessment of the quality of television images. Technical Report BT.500--15. ITU-R.
[9]
Jianzhao Liu, Xin Li, Yanding Peng, Tao Yu, and Zhibo Chen. 2022. Swiniqa: Learned swin distance for compressed image quality assessment. In Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. 1795--1799.
[10]
Kede Ma, Zhengfang Duanmu, Zhou Wang, Qingbo Wu, Wentao Liu, Hongwei Yong, Hongliang Li, and Lei Zhang. 2018. Group maximum differentiation competition: Model comparison with few samples. IEEE Trans. Pattern Anal. Mach. Intell., Vol. 42, 4 (2018), 851--864.
[11]
Kede Ma, Zhengfang Duanmu, Qingbo Wu, Zhou Wang, Hongwei Yong, Hongliang Li, and Lei Zhang. 2016. Waterloo exploration database: New challenges for image quality assessment models. IEEE Trans. Image Process., Vol. 26, 2 (2016), 1004--1016.
[12]
Pina Marziliano, Frederic Dufaux, Stefan Winkler, and Touradj Ebrahimi. 2002. A no-reference perceptual blur metric. In Proc. IEEE Int. Conf. Image Process., Vol. 3. IEEE, III--III.
[13]
Anish Mittal, Anush Krishna Moorthy, and Alan Conrad Bovik. 2012. No-reference image quality assessment in the spatial domain. IEEE Trans. Image Process., Vol. 21, 12 (2012), 4695--4708.
[14]
Alexander McFarlane Mood. 1950. Introduction to the Theory of Statistics. (1950).
[15]
Anush Krishna Moorthy and Alan Conrad Bovik. 2010. A two-step framework for constructing blind image quality indices. IEEE Signal Process. Lett., Vol. 17, 5 (2010), 513--516.
[16]
Hatef Otroshi-Shahreza, Arash Amini, and Hamid Behroozi. 2018. No-reference image quality assessment using transfer learning. In Int. Symp. Telecommun. (IST). IEEE, 637--640.
[17]
Kalpana Seshadrinathan, Rajiv Soundararajan, Alan Conrad Bovik, and Lawrence K Cormack. 2010. Study of subjective and objective quality assessment of video. IEEE Trans. Image Process., Vol. 19, 6 (2010), 1427--1441.
[18]
Hamid R Sheikh, Alan C Bovik, and Lawrence Cormack. 2005. No-reference quality assessment using natural scene statistics: JPEG2000. IEEE Trans. Image Process., Vol. 14, 11 (2005), 1918--1927.
[19]
Karen Simonyan and Andrew Zisserman. 2014. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556 (2014).
[20]
Robert C Streijl, Stefan Winkler, and David S Hands. 2016. Mean opinion score (MOS) revisited: methods and applications, limitations and alternatives. Multimedia Syst., Vol. 22, 2 (2016), 213--227.
[21]
Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. 2015. Going deeper with convolutions. In Proc. IEEE Conf. Comput. Vis. Pattern Recognit. 1--9.
[22]
Domonkos Varga. 2020. Multi-pooled inception features for no-reference image quality assessment. Appl. Sci., Vol. 10, 6 (2020), 2186.
[23]
Domonkos Varga, Dietmar Saupe, and Tamás Szirányi. 2018. DeepRN: A content preserving deep architecture for blind image quality assessment. In Proc. IEEE Int. Conf. Multimedia Expo (ICME). IEEE, 1--6.
[24]
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Adv. Neural Inf. Process. Syst., Vol. 30 (2017).
[25]
Jing Wang, Haotian Fan, Xiaoxia Hou, Yitian Xu, Tao Li, Xuechao Lu, and Lean Fu. 2022. Mstriq: No reference image quality assessment based on swin transformer with multi-stage fusion. In Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. 1269--1278.
[26]
Zhou Wang and Alan Conrad Bovik. 2006. Modern image quality assessment. Ph.,D. Dissertation. Springer.
[27]
Zhou Wang and Alan C Bovik. 2009. Mean squared error: Love it or leave it? A new look at signal fidelity measures. IEEE Signal Process. Mag., Vol. 26, 1 (2009), 98--117.
[28]
Zhou Wang, Alan C Bovik, and Ligang Lu. 2002 a. Why is image quality assessment so difficult?. In Proc. IEEE Int. Conf. Acoust., Speech, Signal Process., Vol. 4. IEEE, IV--3313.
[29]
Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Simoncelli. 2004. Image quality assessment: from error visibility to structural similarity. IEEE Trans. Image Process., Vol. 13, 4 (2004), 600--612.
[30]
Zhou Wang, Hamid R Sheikh, and Alan C Bovik. 2002 b. No-reference perceptual quality assessment of JPEG compressed images. In Proc. IEEE Int. Conf. Image Process., Vol. 1. IEEE, I--I.
[31]
Zhou Wang and Eero P Simoncelli. 2008. Maximum differentiation (MAD) competition: A methodology for comparing computational models of perceptual quantities. J. Vis., Vol. 8, 12 (2008), 8--8.
[32]
Jingtao Xu, Peng Ye, Qiaohong Li, Haiqing Du, Yong Liu, and David Doermann. 2016. Blind image quality assessment based on high order statistics aggregation. IEEE Trans. Image Process., Vol. 25, 9 (2016), 4444--4457.
[33]
Peng Ye and David Doermann. 2012. No-reference image quality assessment using visual codebooks. IEEE Trans. Image Process., Vol. 21, 7 (2012), 3129--3138.
[34]
Junyong You and Jari Korhonen. 2021. Transformer for image quality assessment. In Proc. IEEE Int. Conf. Image Process. (ICIP). IEEE, 1389--1393.
[35]
Guangtao Zhai and Xiongkuo Min. 2020. Perceptual image quality assessment: a survey. Sci. China Inf. Sci., Vol. 63 (2020), 1--52.
[36]
Lin Zhang, Lei Zhang, and Alan C Bovik. 2015. A feature-enriched completely blind image quality evaluator. IEEE Trans. Image Process., Vol. 24, 8 (2015), 2579--2591.
[37]
Weixia Zhang, Kede Ma, Jia Yan, Dexiang Deng, and Zhou Wang. 2018. Blind image quality assessment using a deep bilinear convolutional neural network. IEEE Trans. Circuits Syst. Video Technol., Vol. 30, 1 (2018), 36--47.
[38]
Yu Zhang, Xinbo Gao, Lihuo He, Wen Lu, and Ran He. 2019. Objective video quality assessment combining transfer learning with CNN. IEEE Trans. Neural Netw. Learn. Syst., Vol. 31, 8 (2019), 2716--2730.

Index Terms

  1. A Metric for Evaluating Image Quality Difference Perception Ability in Blind Image Quality Assessment Models

      Recommendations

      Comments

      Information & Contributors

      Information

      Published In

      cover image ACM Conferences
      QoEVMA'24: Proceedings of the 3rd Workshop on Quality of Experience in Visual Multimedia Applications
      October 2024
      63 pages
      ISBN:9798400712043
      DOI:10.1145/3689093
      Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

      Sponsors

      Publisher

      Association for Computing Machinery

      New York, NY, United States

      Publication History

      Published: 28 October 2024

      Permissions

      Request permissions for this article.

      Check for updates

      Author Tags

      1. blind image quality assessment
      2. image quality difference perception ability
      3. model comparison
      4. model falsification methodology
      5. performance metric

      Qualifiers

      • Research-article

      Funding Sources

      • National Radio and Television Administration
      • Department of Science and Technology of Zhejiang Province

      Conference

      MM '24
      Sponsor:
      MM '24: The 32nd ACM International Conference on Multimedia
      October 28 - November 1, 2024
      Melbourne VIC, Australia

      Acceptance Rates

      QoEVMA'24 Paper Acceptance Rate 6 of 6 submissions, 100%;
      Overall Acceptance Rate 14 of 20 submissions, 70%

      Contributors

      Other Metrics

      Bibliometrics & Citations

      Bibliometrics

      Article Metrics

      • 0
        Total Citations
      • 71
        Total Downloads
      • Downloads (Last 12 months)71
      • Downloads (Last 6 weeks)20
      Reflects downloads up to 20 Feb 2025

      Other Metrics

      Citations

      View Options

      View options

      PDF

      View or Download as a PDF file.

      PDF

      eReader

      View online with eReader.

      eReader

      Login options

      Figures

      Tables

      Media

      Share

      Share

      Share this Publication link

      Share on social media