Skip to main content
SpringerLink
Log in

Misalignment Insensitive Perceptual Metric for Full Reference Image Quality Assessment

  • Conference paper
  • First Online:
Pattern Recognition and Computer Vision (PRCV 2023)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 14435))

Included in the following conference series:

  • 341 Accesses

Abstract

Full-reference (FR) image quality assessment (IQA) is crucial in the evaluation of restored images by comparing them with pristine-quality reference images, offering invaluable insights into the effectiveness of image restoration algorithms. Recently, with the advancement of generative adversarial networks (GANs), the GAN-based restoration algorithms demonstrate excellent restoration capability. Nevertheless, these algorithms introduce local spatial misalignment between the restored and the original reference images, posing a challenge for FR-IQA. To tackle this issue, we present a Misalignment Insensitive Perceptual Metric (MIPM) that strengthens the three components of FR-IQA, namely feature extraction, difference representation and quality regression. Specifically, a Vision Transformer-based network for global feature extraction is employed. Furthermore, MIPM utilizes Local Overlapping Wasserstein Difference (LOWD) and Channel Attention Block (CAB) to provide more accurate difference representation between the features of reference and distorted images in the spatial and channel dimensions, respectively. Lastly, a hybrid loss aids in regressing scores that align better with human subjective perception. Coupled with three key improvements, our MIPM exhibits superior performance over state-of-the-art approaches on five IQA datasets, LIVE, CSIQ, TID2013, KADID-10k, and PIPAL.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 109.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 139.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Blau, Y., Mechrez, R., Timofte, R., Michaeli, T., Zelnik-Manor, L.: The 2018 PIRM challenge on perceptual image super-resolution. In: Leal-Taixé, L., Roth, S. (eds.) ECCV 2018. LNCS, vol. 11133, pp. 334–355. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-11021-5_21

    Chapter  Google Scholar 

  2. Bosse, S., Maniry, D., Müller, K.R., Wiegand, T., Samek, W.: Deep neural networks for no-reference and full-reference image quality assessment. IEEE Trans. Image Process. 27(1), 206–219 (2017)

    Article  MathSciNet  Google Scholar 

  3. Cao, Y., Wan, Z., Ren, D., Yan, Z., Zuo, W.: Incorporating semi-supervised and positive-unlabeled learning for boosting full reference image quality assessment. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 5851–5861 (2022)

    Google Scholar 

  4. Cheon, M., Yoon, S.J., Kang, B., Lee, J.: Perceptual image quality assessment with transformers. In: IEEE Conference on Computer Vision and Pattern Recognition Workshops, pp. 433–442 (2021)

    Google Scholar 

  5. Delbracio, M., Talebi, H., Milanfar, P.: Projected distribution loss for image enhancement. arXiv preprint arXiv:2012.09289 (2020)

  6. Ding, K., Ma, K., Wang, S., Simoncelli, E.: Image quality assessment: unifying structure and texture similarity. IEEE Trans. Pattern Anal. Mach. Intell. (2020)

    Google Scholar 

  7. Dosovitskiy, A., et al.: An image is worth 16x16 words: transformers for image recognition at scale. In: International Conference on Learning Representations (2020)

    Google Scholar 

  8. Gao, F., Wang, Y., Li, P., Tan, M., Yu, J., Zhu, Y.: DeepSim: deep similarity for image quality assessment. Neurocomputing 257, 104–114 (2017)

    Article  Google Scholar 

  9. Golestaneh, S.A., Dadsetan, S., Kitani, K.M.: No-reference image quality assessment via transformers, relative ranking, and self-consistency. In: IEEE Winter Conference on Applications of Computer Vision, pp. 1220–1230 (2022)

    Google Scholar 

  10. Jinjin, G., Haoming, C., Haoyu, C., Xiaoxing, Y., Ren, J.S., Chao, D.: PIPAL: a large-scale image quality assessment dataset for perceptual image restoration. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, J.-M. (eds.) ECCV 2020. LNCS, vol. 12356, pp. 633–651. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-58621-8_37

    Chapter  Google Scholar 

  11. Gu, J., et al.: NTIRE 2021 challenge on perceptual image quality assessment. In: IEEE Conference on Computer Vision and Pattern Recognition Workshops, pp. 677–690 (2021)

    Google Scholar 

  12. Guo, H., Bin, Y., Hou, Y., Zhang, Q., Luo, H.: IQMA network: image quality multi-scale assessment network. In: IEEE Conference on Computer Vision and Pattern Recognition Workshops, pp. 443–452 (2021)

    Google Scholar 

  13. He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 770–778 (2016)

    Google Scholar 

  14. Krizhevsky, A., Sutskever, I., Hinton, G.E.: Imagenet classification with deep convolutional neural networks. Commun. ACM 60(6), 84–90 (2017)

    Article  Google Scholar 

  15. Lao, S., et al.: Attentions help CNNs see better: attention-based hybrid image quality assessment network. In: IEEE Conference on Computer Vision and Pattern Recognition Workshops, pp. 1140–1149 (2022)

    Google Scholar 

  16. Larson, E., Chandler, D.: Most apparent distortion: full-reference image quality assessment and the role of strategy. J. Electron. Imaging 19(1), 011006 (2010)

    Article  Google Scholar 

  17. Li, D., Jiang, T., Jiang, M.: Norm-in-norm loss with faster convergence and better performance for image quality assessment. In: ACM International Conference on Multimedia, pp. 789–797 (2020)

    Google Scholar 

  18. Liao, X., Chen, B., Zhu, H., Wang, S., Zhou, M., Kwong, S.: DeepWSD: projecting degradations in perceptual space to Wasserstein distance in deep feature space. In: ACM International Conference on Multimedia, pp. 970–978 (2022)

    Google Scholar 

  19. Lin, H., Hosu, V., Saupe, D.: KADID-10k: a large-scale artificially distorted IQA database. In: IEEE International Conference on Quality of Multimedia Experience, pp. 1–3. IEEE (2019)

    Google Scholar 

  20. Ma, C., Yang, C.Y., Yang, X., Yang, M.H.: Learning a no-reference quality metric for single-image super-resolution. Comput. Vis. Image Underst. 158, 1–16 (2017)

    Article  Google Scholar 

  21. Mittal, A., Soundararajan, R., Bovik, A.: Making a “completely blind” image quality analyzer. IEEE Sig. Process. Lett. 20(3), 209–212 (2012)

    Google Scholar 

  22. Ponomarenko, N., et al.: Image database TID2013: peculiarities, results and perspectives. Sig. Process. Image Commun. 30, 57–77 (2015)

    Article  Google Scholar 

  23. Prashnani, E., Cai, H., Mostofi, Y., Sen, P.: PieAPP: perceptual image-error assessment through pairwise preference. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 1808–1817 (2018)

    Google Scholar 

  24. Russakovsky, O., et al.: Imagenet large scale visual recognition challenge. Int. J. Comput. Vision 115(3), 211–252 (2015)

    Article  MathSciNet  Google Scholar 

  25. Sheikh, H.: Image and video quality assessment research at live (2003). http://live.ece.utexas.edu/research/quality

  26. Sheikh, H., Bovik, A.: Image information and visual quality. IEEE Trans. Image Process. 15(2), 430–444 (2006)

    Article  Google Scholar 

  27. Sheikh, H., Bovik, A., De Veciana, G.: An information fidelity criterion for image quality assessment using natural scene statistics. IEEE Trans. Image Process. 14(12), 2117–2128 (2005)

    Article  Google Scholar 

  28. Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556 (2014)

  29. Su, S., et al.: Blindly assess image quality in the wild guided by a self-adaptive hyper network. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 3667–3676 (2020)

    Google Scholar 

  30. Wang, Z., Bovik, A.: A universal image quality index. IEEE Sig. Process. Lett. 9(3), 81–84 (2002)

    Article  Google Scholar 

  31. Wang, Z., Bovik, A., Sheikh, H., Simoncelli, E.: Image quality assessment: from error visibility to structural similarity. IEEE Trans. Image Process. 13(4), 600–612 (2004)

    Article  Google Scholar 

  32. Wang, Z., Simoncelli, E., Bovik, A.: Multiscale structural similarity for image quality assessment. In: Asilomar Conference on Signals, Systems and Computers, vol. 2, pp. 1398–1402. IEEE (2003)

    Google Scholar 

  33. Xue, W., Mou, X., Zhang, L., Bovik, A., Feng, X.: Blind image quality assessment using joint statistics of gradient magnitude and Laplacian features. IEEE Trans. Image Process. 23(11), 4850–4862 (2014)

    Article  MathSciNet  Google Scholar 

  34. Yang, S., et al.: MANIQA: Multi-dimension attention network for no-reference image quality assessment. In: IEEE Conference on Computer Vision and Pattern Recognition Workshops, pp. 1191–1200 (2022)

    Google Scholar 

  35. Zamir, S.W., Arora, A., Khan, S., Hayat, M., Khan, F.S., Yang, M.H.: Restormer: efficient transformer for high-resolution image restoration. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 5728–5739 (2022)

    Google Scholar 

  36. Zhang, L., Shen, Y., Li, H.: VSI: a visual saliency-induced index for perceptual image quality assessment. IEEE Trans. Image Process. 23(10), 4270–4281 (2014)

    Article  MathSciNet  Google Scholar 

  37. Zhang, L., Zhang, L., Mou, X., Zhang, D.: FSIM: a feature similarity index for image quality assessment. IEEE Trans. Image Process. 20(8), 2378–2386 (2011)

    Article  MathSciNet  Google Scholar 

  38. Zhang, R., Isola, P., Efros, A., Shechtman, E., Wang, O.: The unreasonable effectiveness of deep features as a perceptual metric. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 586–595 (2018)

    Google Scholar 

  39. Zhang, Y., Li, K., Li, K., Wang, L., Zhong, B., Fu, Y.: Image super-resolution using very deep residual channel attention networks. In: Ferrari, V., Hebert, M., Sminchisescu, C., Weiss, Y. (eds.) ECCV 2018. LNCS, vol. 11211, pp. 294–310. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-01234-2_18

    Chapter  Google Scholar 

  40. Zhang, Z., Wang, R., Zhang, H., Chen, Y., Zuo, W.: Self-supervised learning for real-world super-resolution from dual zoomed observations. In: Avidan, S., Brostow, G., Cissé, M., Farinella, G.M., Hassner, T. (eds.) ECCV 2022. LNCS, vol. 13678, pp. 610–627. Springer, Cham (2022). https://doi.org/10.1007/978-3-031-19797-0_35

    Chapter  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. School of Computer Science and Technology, Harbin Institute of Technology, Harbin, China

    Shunyu Yao, Yue Cao, Yabo Zhang & Wangmeng Zuo

Authors
  1. Shunyu Yao
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Yue Cao
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Yabo Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Wangmeng Zuo
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Wangmeng Zuo .

Editor information

Editors and Affiliations

  1. Nanjing University of Information Science and Technology, Nanjing, China

    Qingshan Liu

  2. Xiamen University, Xiamen, China

    Hanzi Wang

  3. Beijing University of Posts and Telecommunications, Beijing, China

    Zhanyu Ma

  4. Sun Yat-sen University, Guangzhou, China

    Weishi Zheng

  5. Peking University, Beijing, China

    Hongbin Zha

  6. Chinese Academy of Sciences, Beijing, China

    Xilin Chen

  7. Chinese Academy of Sciences, Beijing, China

    Liang Wang

  8. Xiamen University, Xiamen, China

    Rongrong Ji

1 Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary material 1 (pdf 1246 KB)

Rights and permissions

Reprints and permissions

Copyright information

© 2024 The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd.

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Yao, S., Cao, Y., Zhang, Y., Zuo, W. (2024). Misalignment Insensitive Perceptual Metric for Full Reference Image Quality Assessment. In: Liu, Q., et al. Pattern Recognition and Computer Vision. PRCV 2023. Lecture Notes in Computer Science, vol 14435. Springer, Singapore. https://doi.org/10.1007/978-981-99-8552-4_35

Download citation

  • DOI: https://doi.org/10.1007/978-981-99-8552-4_35

  • Published:

  • Publisher Name: Springer, Singapore

  • Print ISBN: 978-981-99-8551-7

  • Online ISBN: 978-981-99-8552-4

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation