Skip to main content
SpringerLink
Log in

Image Quality Assessment Method Based on Cross-Modal

  • Conference paper
  • First Online:
PRICAI 2023: Trends in Artificial Intelligence (PRICAI 2023)

Part of the book series: Lecture Notes in Computer Science ((LNAI,volume 14327))

Included in the following conference series:

  • 501 Accesses

Abstract

Deep learning methods have achieved remarkable results in the direction of image quality assessment tasks. However, most of the related studies focus only on image unimodal, ignoring the potential advantages that come with the development of cross-modal techniques. Cross-modal models have implied a wealth of information, which provides new research directions and possibilities in the field of image quality assessment. In this paper, the feasibility of cross-modal models in image quality assessment is first explored for the image quality binary classification task. Subsequently, the optimization prompting method is combined with the tuning of the image encoder in the cross-modal model so that the cross-modal model can be used for image quality assessment scoring. To verify the feasibility of the cross-modal model on the image quality assessment task, an empirical analysis was conducted on the binary image quality dataset PQD, and it was found that the F1 score improved by 18% over the baseline model. Further, we propose an adaptive cross-modal image quality assessment method AC-IQA. On the image quality scoring dataset, compared with the previous optimal methods, AC-IQA improves the PLCC and SROCC metrics on the TID2013 dataset by 5.5% and 9.5%, respectively, and on the KADID dataset by 6.2% and 5.2%.

This work was supported by National Natural Science Foundation of China (62271359).

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 59.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 79.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Bosse, S., Maniry, D., Müller, K.R., Wiegand, T., Samek, W.: Deep neural networks for no-reference and full-reference image quality assessment. IEEE Trans. Image Process. 27(1), 206–219 (2017)

    Article  MathSciNet  MATH  Google Scholar 

  2. Dendi, S.V.R., Dev, C., Kothari, N., Channappayya, S.S.: Generating image distortion maps using convolutional autoencoders with application to no reference image quality assessment. IEEE Signal Process. Lett. 26(1), 89–93 (2018)

    Article  Google Scholar 

  3. Gao, P., et al.: CLIP-Adapter: better vision-language models with feature adapters. arXiv preprint arXiv:2110.04544 (2021)

  4. Golestaneh, S.A., Dadsetan, S., Kitani, K.M.: No-reference image quality assessment via transformers, relative ranking, and self-consistency. In: Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 1220–1230 (2022)

    Google Scholar 

  5. He, K., Zhang, X., Ren, S., Sun, J.: Delving deep into rectifiers: surpassing human-level performance on imagenet classification. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 1026–1034 (2015)

    Google Scholar 

  6. He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 770–778 (2016)

    Google Scholar 

  7. Jia, M., et al.: Visual prompt tuning. In: Avidan, S., Brostow, G., Ciss, M., Farinella, G.M., Hassner, T. (eds.) Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part XXXIII, vol. 13693, pp. 709–727. Springer, Cham (2022). https://doi.org/10.1007/978-3-031-19827-4_41

  8. Jin, X., et al.: ILGNet: inception modules with connected local and global features for efficient image aesthetic quality classification using domain adaptation. IET Comput. Vision 13(2), 206–212 (2019)

    Article  Google Scholar 

  9. Kim, J., Lee, S.: Fully deep blind image quality predictor. IEEE J. Sel. Top. Sign. Proces. 11(1), 206–220 (2017)

    Article  Google Scholar 

  10. Larson, E.C., Chandler, D.M.: Most apparent distortion: full-reference image quality assessment and the role of strategy. J. Electron. Imaging 19(1), 011006 (2010)

    Article  Google Scholar 

  11. Lin, H., Hosu, V., Saupe, D.: KADID-10k: a large-scale artificially distorted IQA database. In: 2019 Eleventh International Conference on Quality of Multimedia Experience (QoMEX), pp. 1–3. IEEE (2019)

    Google Scholar 

  12. Liu, X., Van De Weijer, J., Bagdanov, A.D.: RankIQA: learning from rankings for no-reference image quality assessment. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 1040–1049 (2017)

    Google Scholar 

  13. Lu, J., Batra, D., Parikh, D., Lee, S.: ViLBERT: pretraining task-agnostic visiolinguistic representations for vision-and-language tasks. In: Advances in Neural Information Processing Systems 32 (2019)

    Google Scholar 

  14. Luo, W., Wang, X., Tang, X.: Content-based photo quality assessment. In: 2011 International Conference on Computer Vision, pp. 2206–2213. IEEE (2011)

    Google Scholar 

  15. Mittal, A., Moorthy, A.K., Bovik, A.C.: No-reference image quality assessment in the spatial domain. IEEE Trans. Image Process. 21(12), 4695–4708 (2012)

    Article  MathSciNet  MATH  Google Scholar 

  16. Ponomarenko, N., et al.: Image database TID2013: peculiarities, results and perspectives. Sign. process. Image Commun. 30, 57–77 (2015)

    Article  Google Scholar 

  17. Radford, A., et al.: Learning transferable visual models from natural language supervision. In: International Conference on Machine Learning, pp. 8748–8763. PMLR (2021)

    Google Scholar 

  18. Rombach, R., Blattmann, A., Lorenz, D., Esser, P., Ommer, B.: High-resolution image synthesis with latent diffusion models. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10684–10695 (2022)

    Google Scholar 

  19. Su, S., et al.: Blindly assess image quality in the wild guided by a self-adaptive hyper network. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3667–3676 (2020)

    Google Scholar 

  20. Talebi, H., Milanfar, P.: NIMA: neural image assessment. IEEE Trans. Image Process. 27(8), 3998–4011 (2018)

    Article  MathSciNet  MATH  Google Scholar 

  21. Tan, H., Bansal, M.: LXMERT: learning cross-modality encoder representations from transformers. arXiv preprint arXiv:1908.07490 (2019)

  22. Yan, W., Li, Y., Yang, H., Huang, B., Pan, Z.: Semantic-aware multi-task learning for image aesthetic quality assessment. Connect. Sci. 34(1), 2689–2713 (2022)

    Article  Google Scholar 

  23. You, J., Korhonen, J.: Transformer for image quality assessment. In: 2021 IEEE International Conference on Image Processing (ICIP), pp. 1389–1393. IEEE (2021)

    Google Scholar 

  24. Zeng, H., Zhang, L., Bovik, A.C.: A probabilistic quality representation approach to deep blind image quality prediction. arXiv preprint arXiv:1708.08190 (2017)

  25. Zhang, H., Luo, Y., Zhang, L., Wu, Y., Wang, M., Shen, Z.: Considering three elements of aesthetics: multi-task self-supervised feature learning for image style classification. Neurocomputing 520, 262–273 (2023)

    Article  Google Scholar 

  26. Zhang, L., Zhang, L., Bovik, A.C.: A feature-enriched completely blind image quality evaluator. IEEE Trans. Image Process. 24(8), 2579–2591 (2015)

    Article  MathSciNet  MATH  Google Scholar 

  27. Zhang, R., et al.: Tip-Adapter: training-free adaption of clip for few-shot classification. In: Avidan, S., Brostow, G., Ciss, M., Farinella, G.M., Hassner, T. (eds.) Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part XXXV, vol. 13695, pp. 493–510. Springer, Cham (2022). https://doi.org/10.1007/978-3-031-19833-5_29

  28. Zhang, W., Ma, K., Yan, J., Deng, D., Wang, Z.: Blind image quality assessment using a deep bilinear convolutional neural network. IEEE Trans. Circuits Syst. Video Technol. 30(1), 36–47 (2018)

    Article  Google Scholar 

  29. Zhou, K., Yang, J., Loy, C.C., Liu, Z.: Conditional prompt learning for vision-language models. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 16816–16825 (2022)

    Google Scholar 

  30. Zhou, K., Yang, J., Loy, C.C., Liu, Z.: Learning to prompt for vision-language models. Int. J. Comput. Vision 130(9), 2337–2348 (2022)

    Article  Google Scholar 

  31. Zhu, H., Li, L., Wu, J., Dong, W., Shi, G.: MetaIQA: deep meta-learning for no-reference image quality assessment. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14143–14152 (2020)

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. School of Computer Science and Technology, Wuhan University of Science and Technology, Wuhan, 430065, Hubei, China

    Teng Long, Li Chen & Yubo Wang

  2. Hubei Province Key Laboratory of Intelligent Information Processing and Real-time Industrial System, Wuhan University of Science and Technology, Wuhan, 430065, Hubei, China

    Teng Long, Li Chen & Yubo Wang

Authors
  1. Teng Long
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Li Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Yubo Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Li Chen .

Editor information

Editors and Affiliations

  1. Tsinghua University, Beijing, China

    Fenrong Liu

  2. SEEK Limited, Cremorne, NSW, Australia

    Arun Anand Sadanandan

  3. MIMOS Berhad, Kuala Lumpur, Malaysia

    Duc Nghia Pham

  4. Universitas Indonesia, Depok, Indonesia

    Petrus Mursanto

  5. Tabcorp Holdings Limited, Melbourne, VIC, Australia

    Dickson Lukose

Rights and permissions

Reprints and permissions

Copyright information

© 2024 The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd.

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Long, T., Chen, L., Wang, Y. (2024). Image Quality Assessment Method Based on Cross-Modal. In: Liu, F., Sadanandan, A.A., Pham, D.N., Mursanto, P., Lukose, D. (eds) PRICAI 2023: Trends in Artificial Intelligence. PRICAI 2023. Lecture Notes in Computer Science(), vol 14327. Springer, Singapore. https://doi.org/10.1007/978-981-99-7025-4_10

Download citation

  • DOI: https://doi.org/10.1007/978-981-99-7025-4_10

  • Published:

  • Publisher Name: Springer, Singapore

  • Print ISBN: 978-981-99-7024-7

  • Online ISBN: 978-981-99-7025-4

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation