Skip to main content
SpringerLink
Log in

Saliency-based feature fusion convolutional network for blind image quality assessment

  • Original Paper
  • Published:
Signal, Image and Video Processing Aims and scope Submit manuscript

Abstract

Quality assessment plays an important role in promoting the prevalence of digital imaging technology as well as the associated products. Since the human being is the ultimate assessor of image quality, the human visual system model has received much attention. In this paper, we present a novel IQA approach via analysis of human visual characteristics. Given that salient regions have greater impacts on subjects’ judgments of image quality, a saliency-based filtering model is first designed to collect saliency patches, and a saliency weighting matrix is obtained to represent their priority. Second, to learn more effective feature representations, we design a sub-network with up-sampling layers to capture features from different levels. Features are synthesized by utilizing a feature fusion convolutional network with two-stream structure. Features from different levels are mapped to a local score. Finally, the local score of each salient patch is summarized by a saliency-weighting model to work out the final predicted score. The experimental results on a series of publicly available databases, e.g., LIVE, CISQ and TID2013 demonstrate that the proposed method outperforms other state-of-the-art methods.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2

Similar content being viewed by others

References

  1. Yue, G., Hou, C., Jiang, Q., Yang, Y.: Blind stereoscopic 3D image quality assessment via analysis of naturalness, structure, and binocular asymmetry. Signal Process. 150, 204 (2018)

    Article  Google Scholar 

  2. Shi, Z., Zhang, J., Cao, Q., Pang, K., Luo, T.: Full-reference image quality assessment based on image segmentation with edge feature. Signal Process. 145, 99 (2018)

    Article  Google Scholar 

  3. Yue, G., Hou, C., Gu, K., Zhou, T., Zhai, G.: Combining local and global measures for DIBR-synthesized image quality evaluation. IEEE Trans. Image Process. 28(4), 2075 (2018)

    Article  MathSciNet  Google Scholar 

  4. Yue, G., Yan, W., Zhou, T.: Referenceless quality evaluation of tone-mapped hdr and multi-exposure fused images. IEEE Trans. Ind. Inform. (2019)

  5. Yue, G., Hou, C., Gu, K., Zhou, T., Liu, H.: No-reference quality evaluator of transparently encrypted images. IEEE Trans. Multimed. 21(9), 2184 (2019)

    Article  Google Scholar 

  6. He, S., Liu, Z.: Image quality assessment based on adaptive multiple Skyline query. Signal Process. Image Commun. 80, 115676 (2020)

    Article  Google Scholar 

  7. Tang, Y., Jiang, S., Xu, S., Liu, T., Li, C.: Blind image quality assessment based on multi-window method and HSV color space. Appl. Sci. 9(12), 2499 (2019)

    Article  Google Scholar 

  8. Mittal, A., Soundararajan, R., Bovik, A.C.: Making a completely blind image quality analyzer. IEEE Signal Process. Lett. 20(3), 209 (2012)

    Article  Google Scholar 

  9. Moorthy, A.K., Bovik, A.C.: Blind image quality assessment: from natural scene statistics to perceptual quality. IEEE Trans. Image Process. 20(12), 3350 (2011)

    Article  MathSciNet  Google Scholar 

  10. Ye, P., Doermann, D.: No-reference image quality assessment using visual codebooks. IEEE Trans. Image Process. 21(7), 3129 (2012)

    Article  MathSciNet  Google Scholar 

  11. Zhou, W., Chen, Z., Li, W.: Dual-stream interactive networks for no-reference stereoscopic image quality assessment. IEEE Trans. Image Process. 28(8), 3946 (2019)

    Article  MathSciNet  Google Scholar 

  12. Kim, J., Zeng, H., Ghadiyaram, D., Lee, S., Zhang, L., Bovik, A.C.: Deep convolutional neural models for picture-quality prediction: challenges and solutions to data-driven image quality assessment. IEEE Signal Process. Mag. 34(6), 130 (2017)

    Article  Google Scholar 

  13. Ma, K., Liu, W., Zhang, K., Duanmu, Z., Wang, Z., Zuo, W.: End-to-end blind image quality assessment using deep neural networks. IEEE Trans. Image Process. 27(3), 1202 (2017)

    Article  MathSciNet  Google Scholar 

  14. Bosse, S., Maniry, D., Müller, K.R., Wiegand, T., Samek, W.: Deep neural networks for no-reference and full-reference image quality assessment. IEEE Trans. Image Process. 27(1), 206 (2017)

    Article  MathSciNet  Google Scholar 

  15. Bianco, S., Celona, L., Napoletano, P., Schettini, R.: On the use of deep learning for blind image quality assessment. SIViP 12(2), 355 (2018)

    Article  Google Scholar 

  16. Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., et al.: Imagenet large scale visual recognition challenge. Int. J. Comput. Vis. 115(3), 211 (2015)

    Article  MathSciNet  Google Scholar 

  17. Zhang, W., Ma, K., Yan, J., Deng, D., Wang, Z.: Blind image quality assessment using a deep bilinear convolutional neural network. IEEE Trans. Circuits Syst. Video Technol. (2018)

  18. Kang, L., Ye, P., Li, Y., Doermann, D.: Convolutional neural networks for no-reference image quality assessment. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1733–1740 (2014)

  19. Bare, B., Li, K., Yan, B.: An accurate deep convolutional neural networks model for no-reference image quality assessment. In: 2017 IEEE International Conference on Multimedia and Expo (ICME), pp. 1356–1361. IEEE (2017)

  20. Yan, Q., Gong, D., Zhang, Y.: Two-stream convolutional networks for blind image quality assessment. IEEE Trans. Image Process. 28(5), 2200 (2018)

    Article  MathSciNet  Google Scholar 

  21. Bosse, S., Maniry, D., Wiegand, T., Samek, W.: A deep neural network for image quality assessment. In: 2016 IEEE International Conference on Image Processing (ICIP), pp. 3773–3777. IEEE (2016)

  22. Kang, L., Ye, P., Li, Y., Doermann, D.: Simultaneous estimation of image quality and distortion via multi-task convolutional neural networks. In: 2015 IEEE International Conference on Image Processing (ICIP), pp. 2791–2795. IEEE (2015)

  23. Kim, J., Lee, S.: Fully deep blind image quality predictor. IEEE J. Sel. Top. Signal Process. 11(1), 206 (2016)

    Article  Google Scholar 

  24. Wang, Z., Bovik, A.C., Sheikh, H.R., Simoncelli, E.P.: Image quality assessment: from error visibility to structural similarity. IEEE Trans. Image Process. 13(4), 600 (2004)

    Article  Google Scholar 

  25. Liu, H., Heynderickx, I.: Visual attention in objective image quality assessment: based on eye-tracking data. IEEE Trans. Circuits Syst. Video Technol. 21(7), 971 (2011)

    Article  Google Scholar 

  26. Le Meur, O., Ninassi, A., Le Callet, P., Barba, D.: Overt visual attention for free-viewing and quality assessment tasks: impact of the regions of interest on a video quality metric. Signal Process. Image Commun. 25(7), 547 (2010)

    Article  Google Scholar 

  27. Engelke, U., Kaprykowsky, H., Zepernick, H.J., Ndjiki-Nya, P.: Visual attention in quality assessment. IEEE Signal Process. Mag. 28(6), 50 (2011)

    Article  Google Scholar 

  28. Liu, H., Engelke, U., Wang, J., Le Callet, P., Heynderickx, I.: How does image content affect the added value of visual attention in objective image quality assessment? IEEE Signal Process. Lett. 20(4), 355 (2013)

    Article  Google Scholar 

  29. Ma, Q., Zhang, L.: Image quality assessment with visual attention. In: 2008 19th International Conference on Pattern Recognition, pp. 1–4. IEEE (2008)

  30. Wang, Z., Sheikh, H.R., Bovik, A.C.: No-reference perceptual quality assessment of JPEG compressed images. In: Proceedings. International Conference on Image Processing, vol. 1, pp. I–I. IEEE (2002)

  31. Ruderman, D.L.: The statistics of natural images. Netw. Comput. Neural Syst. 5(4), 517 (1994)

    Article  Google Scholar 

  32. Mittal, A., Moorthy, A.K., Bovik, A.C.: No-reference image quality assessment in the spatial domain. IEEE Trans. Image Process. 21(12), 4695 (2012)

    Article  MathSciNet  Google Scholar 

  33. Saad, M.A., Bovik, A.C., Charrier, C.: Blind image quality assessment: a natural scene statistics approach in the DCT domain. IEEE Trans. Image Process. 21(8), 3339 (2012)

    Article  MathSciNet  Google Scholar 

  34. Moorthy, A.K., Bovik, A.C.: A two-step framework for constructing blind image quality indices. IEEE Signal Process. Lett. 17(5), 513 (2010)

    Article  Google Scholar 

  35. Xue, W., Mou, X., Zhang, L., Bovik, A.C., Feng, X.: Blind image quality assessment using joint statistics of gradient magnitude and Laplacian features. IEEE Trans. Image Process. 23(11), 4850 (2014)

    Article  MathSciNet  Google Scholar 

  36. Ye, P., Kumar, J., Kang, L., Doermann, D.: Unsupervised feature learning framework for no-reference image quality assessment. In: 2012 IEEE Conference on Computer Vision and Pattern Recognition, pp. 1098–1105. IEEE (2012)

  37. Xu, J., Ye, P., Li, Q., Du, H., Liu, Y., Doermann, D.: Blind image quality assessment based on high order statistics aggregation. IEEE Trans. Image Process. 25(9), 4444 (2016)

    Article  MathSciNet  Google Scholar 

  38. Gu, K., Zhai, G., Yang, X., Zhang, W.: Using free energy principle for blind image quality assessment. IEEE Trans. Multimed. 17(1), 50 (2014)

    Article  Google Scholar 

  39. Li, Q., Lin, W., Fang, Y.: No-reference quality assessment for multiply-distorted images in gradient domain. IEEE Signal Process. Lett. 23(4), 541 (2016)

    Article  Google Scholar 

  40. Yang, Y., Ming, J.: Image quality assessment based on the space similarity decomposition model. Signal Process. 120, 797 (2016)

    Article  Google Scholar 

  41. Zhang, P., Zhou, W., Wu, L., Li, H.: SOM: semantic obviousness metric for image quality assessment. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2394–2402 (2015)

  42. Zhang, W., Borji, A., Wang, Z., Le Callet, P., Liu, H.: The application of visual saliency models in objective image quality assessment: a statistical evaluation. IEEE Trans. Neural Netw. Learn. Syst. 27(6), 1266 (2015)

  43. Li, L., Lin, W., Wang, X., Yang, G., Bahrami, K., Kot, A.C.: No-reference image blur assessment based on discrete orthogonal moments. IEEE Trans. Cybern. 46(1), 39 (2015)

  44. Oszust, M.: No-Reference quality assessment of noisy images with local features and visual saliency models. Inf. Sci. 482, 334 (2019)

    Article  Google Scholar 

  45. Gao, F., Yu, J.: Biologically inspired image quality assessment. Signal Process. 124, 210 (2016)

    Article  Google Scholar 

  46. Kim, J., Kim, W., Lee, S.: Deep blind image quality assessment by learning sensitivity map. In: 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 6727–6731. IEEE (2018)

  47. Kim, J., Nguyen, A.D., Lee, S.: Deep CNN-based blind image quality predictor. IEEE Trans. Neural Netw. Learn. Syst. 30(1), 11 (2018)

    Article  Google Scholar 

  48. Zeng, H., Zhang, L., Bovik, A.C.: Blind image quality assessment with a probabilistic quality representation. In: 2018 25th IEEE International Conference on Image Processing (ICIP), pp. 609–613. IEEE (2018)

  49. Lin, K.Y., Wang, G.: Hallucinated-iqa: no-reference image quality assessment via adversarial learning. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 732–741 (2018)

  50. Jia, S., Zhang, Y.: Saliency-based deep convolutional neural network for no-reference image quality assessment. Multimed. Tools Appl. 77(12), 14859 (2018)

    Article  Google Scholar 

  51. Harel, J., Koch, C., Perona, P.: Graph-based visual saliency. Adv. Neural Inf. Process. Syst. 19, 545552 (2006)

  52. Gao, F., Yu, J., Zhu, S., Huang, Q., Tian, Q.: Blind image quality prediction by exploiting multi-level deep representations. Pattern Recogn. 81, 432 (2018)

    Article  Google Scholar 

  53. Sheikh, H.R., Sabir, M.F., Bovik, A.C.: A statistical evaluation of recent full reference image quality assessment algorithms. IEEE Trans. Image Process. 15(11), 3440 (2006)

    Article  Google Scholar 

  54. Larson, E.C., Chandler, D.M.: Most apparent distortion: full-reference image quality assessment and the role of strategy. J. Electron. Imaging 19(1), 011006 (2010)

    Article  Google Scholar 

  55. Ponomarenko, N., Ieremeiev, O., Lukin, V., Egiazarian, K., Jin, L., Astola, J., Vozel, B., Chehdi, K., Carli, M., Battisti, F., et al.: Color image database TID2013: Peculiarities and preliminary results. In: European Workshop on Visual Information Processing (EUVIP), pp. 106–111. IEEE (2013)

Download references

Author information

Authors and Affiliations

  1. School of Electrical and Information Engineering, Tianjin University, Tianjin, 300072, China

    Lili Shen, Chuhe Zhang & Chunping Hou

Authors
  1. Lili Shen
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Chuhe Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Chunping Hou
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Lili Shen.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Shen, L., Zhang, C. & Hou, C. Saliency-based feature fusion convolutional network for blind image quality assessment. SIViP 16, 419–427 (2022). https://doi.org/10.1007/s11760-021-01958-7

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11760-021-01958-7

Keywords

Search

Navigation