Skip to main content
SpringerLink
Log in

An End-to-End Perceptual Quality Assessment Method via Score Distribution Prediction

  • Published:
Neural Processing Letters Aims and scope Submit manuscript

Abstract

Image quality assessment (IQA) has become a rapidly growing field of technology as it automatically predicts the perceptual quality, which is of vital importance for consumer-centric services. However, most existing IQA algorithms focus on predicting the mean opinion score regardless of the inevitable opinion diversity. To address this shortcoming, in this paper, we propose to predict the distribution of opinion scores via an end-to-end convolutional neural network. The network is based on a pre-trained ResNet with 50 layers and a novel Statistical Region-of-Interest (ROI) Pooling layer is introduced for lower model complexity, which enables effective training with few datum. Meanwhile, instead of using traditional mean-square-error as loss function, our model is trained with cross-entropy loss, which is more suitable for probability distribution learning. Extensive experiments have been carried out on ESPL-LIVE HDR datasets with highly diverse opinion scores. It is shown that the statistical ROI Pooling is more efficient than traditional ROI Pooling layers and classical dimensionality reduction of principle component analysis. And the proposed algorithm achieves superior performance than state-of-the-art label distribution learning methods in terms of six representative evaluation metrics.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6

Similar content being viewed by others

References

  1. Zhao S, Yao H, Gao Y, Ding G, Chua T (2018) Predicting personalized image emotion perceptions in social networks. IEEE Trans Affect Comput 9(4):526–540

    Article  Google Scholar 

  2. Jing P, Su Y, Nie L, Bai X, Liu J, Wang M (2018) Low-rank multi-view embedding learning for micro-video popularity prediction. IEEE Trans Knowl Data Eng 30:1519–1532

    Article  Google Scholar 

  3. Jing P, Su Y, Nie L, Gu H, Liu J, Wang M (2018) A framework of joint low-rank and sparse regression for image memorability prediction. IEEE Trans Circuits Syst Video Technol. https://doi.org/10.1109/TCSVT.2018.2832095

  4. Liu A, Shi Y, Jing P, Liu J, Su Y (2018) Low-rank regularized multi-view inverse-covariance estimation for visual sentiment distribution prediction. J Vis Commun Image Represent 57:243–252

    Article  Google Scholar 

  5. Liu A, Wang J, Liu J, Su Y (2018) Comprehensive image quality assessment via predicting the distribution of opinion score. Multimed Tools Appl. https://doi.org/10.1007/s11042-018-6985-2

  6. Ma S, Liu J, Chen W (2017) A-Lamp: adaptive layout-aware multi-patch deep convolutional neural network for photo aesthetic assessment. In: IEEE conference on computer vision and pattern recognition

  7. Min X, Gu K, Zhai G, Liu J, Yang X, Chen CW (2018) Blind quality assessment based on pseudo-reference image. IEEE Trans Multimed 20:2049–2062

    Article  Google Scholar 

  8. Liu J, Zhai G, Yang X, Chen L (2014) Lossless predictive coding for images with Bayesian treatment. IEEE Trans Image Process 23(12):5519–5530

    Article  MathSciNet  Google Scholar 

  9. Liu J, Yang X, Zhai G, Chen L (2013) Hybrid image interpolation with soft-decision kernel regression. In: IEEE international symposium on circuits and systems, Beijing, pp 765–768

  10. Liu J, Zhai G, Yang X, Yang B, Chen L (2015) Spatial error concealment with an adaptive linear predictor. IEEE Trans Circuits Syst Video Technol 25(3):353–366

    Article  Google Scholar 

  11. Xu H, Zhai G, Yang X (2013) Single image super-resolution with detail enhancement based on local fractal analysis of gradient. IEEE Trans Circuits Syst Video Technol 23(10):1740–1754

    Article  Google Scholar 

  12. Liu J, Zhai G, Liu A, Yang X, Zhao X, Chen CW (2018) IPAD: intensity potential for adaptive De-quantization. IEEE Trans Image Process 27(10):4860–4872

    Article  MathSciNet  Google Scholar 

  13. Liu J, Liu P, Su Y, Jing P, Yang X (2019) Spatiotemporal symmetric convolutional neural network for video bit-depth enhancement. IEEE Trans Multimed. https://doi.org/10.1109/TMM.2019.2897909

  14. Xu H, Zhai G, Wu X, Yang X (2014) Generalized equalization model for image enhancement. IEEE Trans Multimed 16(1):68–82

    Article  Google Scholar 

  15. Zhu W, Zhai G, Hu M, Liu J, Yang X (2018) Arrow’s impossibility theorem inspired subjective image quality assessment approach. Signal Process 145:193–201

    Article  Google Scholar 

  16. Sheikh H, Bovik A, De V (2005) An information fidelity criterion for image quality assessment using natural scene statistics. IEEE Trans Image Process 14(12):2117–2128

    Article  Google Scholar 

  17. Min X, Gu K, Zhai G, Hu M, Yang X (2018) Saliency-induced reduced reference quality index for natural scene and screen content images. Signal Process 145:127–136

    Article  Google Scholar 

  18. Min X, Ma K, Gu K, Zhai G (2017) Unified blind quality assessment compressed natural, graphic, and screen content images. IEEE Trans Image Process 26(11):5462–5474

    Article  MathSciNet  Google Scholar 

  19. Min X, Zhai G, Gu K, Liu Y, Yang X (2018) Blind image quality estimation via distortion aggravation. IEEE Trans Broadcast 64(2):508–517

    Article  Google Scholar 

  20. Geng X, Chao Y, Zhou ZH (2013) Facial age estimation by learning from label distributions. IEEE Trans Pattern Anal Mach Intell 35(10):2401–2412

    Article  Google Scholar 

  21. Geng X, Ji R (2013) Label distribution learning. IEEE Trans Knowl Data Eng 28(7):1734–1748

    Article  Google Scholar 

  22. Gao H, Lin S, Li C, Yang Y (2018) Application of hyperspectral image classification based on overlap pooling. Neural Process Lett 49(3):1335–1354

    Article  Google Scholar 

  23. Zhang X, Xiong H, Zhou W, Lin W, Tian Q (2017) Picking neural activations for fine-grained recognition. IEEE Trans Multimed 19(12):2736–2750

    Google Scholar 

  24. Ding P, Zhang Y, Jia P, Chang X (2018) A comparison: different DCNN models for intelligent object detection in remote sensing images. Neural Process Lett 1:1–11

    Google Scholar 

  25. Zhang X, Feng J, Xiong H, Tian Q (2018) Zigzag learning for weakly supervised object detection. In: IEEE conference on computer vision and pattern recognition, pp 4262–4270

  26. Liu J, Sun W, Su Y, Jing P, Yang X (2019) BE-CALF: bit-depth enhancement by concatenating all level features of DNN. IEEE Trans Image Process. https://doi.org/10.1109/TIP.2019.2912294

  27. Su Y, Sun W, Liu J, Zhai G, Jing P (2019) Photo-realistic image bit-depth enhancement via residual transposed convolutional neural network. Neurocomputing 347:200–211

    Article  Google Scholar 

  28. Ciresan D, Meier U, Schmidhuber J (2012) Multi-column deep neural networks for image classification. In: IEEE conference on computer vision and pattern recognition, pp 3642–3649

  29. Barker A, Varghese B, Ward JS, Sommerville I (2014) Academic cloud computing research: five pitfalls and five opportunities. In: 6th {USENIX} Workshop on Hot Topics in Cloud Computing (HotCloud 14)

  30. He K, Zhang X, Ren S, Sun J (2016) Deep residual learning for image recognition. In: IEEE conference on computer vision and pattern recognition, pp 770–778

  31. He K, Zhang X, Ren S, Sun J (2015) Spatial pyramid pooling in deep convolutional networks for visual recognition. IEEE Trans Pattern Anal Mach Intell 37(9):1904–1916

    Article  Google Scholar 

  32. Kong S, Shen X, Lin Z, Mech R, Fowlkes C (2016) Photo aesthetics ranking network with attributes and content adaptation. In: European conference on computer vision, Springer, pp 662–679

  33. Jin X, Wu L, Li X, Chen S, Peng S, Chi J, Ge S, Song C, Zhao G (2018) Predicting aesthetic score distribution through cumulative jensen-shannon divergence. In: Thirty-Second AAAI Conference on Artificial Intelligence, 28 April

  34. Bishop CM (1995) Neural networks for pattern recognition. Oxford University Press, Oxford

    MATH  Google Scholar 

  35. Murphy KP (2012) Machine learning: a probabilistic perspective. MIT Press, Cambridge

    MATH  Google Scholar 

  36. Ponomarenko N, Ieremeiev O, Lukin V, Egiazarian K, Jin L, Astola J, Vozel B, Chehdi K, Carli M, Battisti F (2013) Color image database TID2013: peculiarities and preliminary results. In: European workshop on visual information processing, pp 106–111

  37. Kundu D, Ghadiyaram D, Bovik A, Evans B (2017) No-reference quality assessment of tone-mapped HDR pictures. IEEE Trans Image Process 26(6):2957–2971

    Article  MathSciNet  Google Scholar 

  38. Larson GW, Rushmeier H, Piatko C (1997) How to assess image quality within a workflow chain: an overview. International journal on digital libraries. IEEE Trans Vis Comput Graph 3(4):291–306

    Article  Google Scholar 

  39. Fattal R, Lischinski D, Werman M (2002) Gradient domain high dynamic range compression. ACM Trans Graph 21(3):249–256

    Article  Google Scholar 

  40. Durand F, Dorsey J (2002) Fast bilateral filtering for the display of high dynamic range images. In: ACM SIGGRAPH, pp 257–266

  41. Reinhard E, Stark M, Shirley P, Ferwerda J (2002) Photographic tone reproduction for digital images. ACM Trans Graph 21(3):267–276

    Article  Google Scholar 

  42. Paul S, Sevcenco I, Agathoklis P (2016) Multi-exposure and multi-focus image fusion in gradient domain. J Circuits Syst Comput 25(10):1650123

    Article  Google Scholar 

  43. Pece F, Kautz J, Agathoklis P (2010) Bitmap movement detection: HDR for dynamic scenes. In: Proceedings of the conference on visual media production, pp 1–8

  44. Raman S, Chaudhuri S (2009) Bilateral filter based compositing for variable exposure photography. In: Eurographics - short papers, pp 1–4

  45. Cover TM, Thomas JA (2012) Elements of information theory. Wiley, Hoboken

    MATH  Google Scholar 

  46. Vedaldi A, Lenc K (2015) MatConvNet—convolutional neural network for MATLAB. In: ACM international conference on multimedia

  47. Deng J, Dong W, Socher R, Li LJ, Kai L, Li F (2009) ImageNet: a large-scale hierarchical image database. In: IEEE conference on computer vision and pattern recognition, pp 248–255

  48. Cha S-H (2007) Comprehensive survey on distance/similarity measures between probability density functions. Int J Math Models Methods Appl Sci 1:300–307

    Google Scholar 

  49. Hou L, Yu CP, Samaras D (2016) Squared earth mover’s distance-based loss for training deep neural networks. Arxiv Preprint, arxiv:1611.05916

  50. Shalev-Shwartz S, Tewari A (2011) Stochastic methods for l1-regularized loss minimization. J Mach Learn Res 12:1865–1892

    MathSciNet  MATH  Google Scholar 

  51. Kuhn HW, Tucker AW (2014) Nonlinear programming. In: Traces and emergence of nonlinear programming. Birkhäuser, Basel, pp 247–258

Download references

Author information

Authors and Affiliations

  1. School of Electrical and Information Engineering, Tianjin University, Tianjin, China

    Jing Liu, Jingting Wang, Weizhi Nie, Yuting Su & Anan Liu

Authors
  1. Jing Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Jingting Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Weizhi Nie
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Yuting Su
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Anan Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding authors

Correspondence to Weizhi Nie or Anan Liu.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Liu, J., Wang, J., Nie, W. et al. An End-to-End Perceptual Quality Assessment Method via Score Distribution Prediction. Neural Process Lett 51, 2123–2137 (2020). https://doi.org/10.1007/s11063-019-10057-1

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11063-019-10057-1

Keywords

Search

Navigation