Skip to main content

Advertisement

SpringerLink
Log in

Quality assessment of 3D synthesized images based on structural and textural distortion

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

Depth image based rendering (DIBR) is a popular technique for rendering virtual 3D views in stereoscopic and autostereoscopic displays. The quality of DIBR-synthesized images may decrease due to various factors, e.g., imprecise depth maps, poor rendering techniques, inaccurate camera parameters. The quality of synthesized images is important as it directly affects the overall user experience. Therefore, the need arises for designing algorithms to estimate the quality of the DIBR-synthesized images. The existing 2D image quality assessment metrics are found to be insufficient for 3D view quality estimation because the 3D views not only contain color information but also make use of disparity to achieve the real depth sensation. In this paper, we present a new algorithm for evaluating the quality of DIBR generated images in the absence of the original references. The human visual system is sensitive to structural information; any deg radation in structure or edges affects the visual quality of the image and is easily noticeable for humans. In the proposed metric, we estimate the quality of the synthesized view by capturing the structural and textural distortion in the warped view. The structural and textural information from the input and the synthesized images is estimated and used to calculate the image quality. The performance of the proposed quality metric is evaluated on the IRCCyN IVC DIBR images dataset. Experimental evaluations show that the proposed metric outperforms the existing 2D and 3D image quality metrics by achieving a high correlation with the subjective ratings.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8

Similar content being viewed by others

References

  1. Battisti F, Bosc E, Carli M, Le Callet P, Perugia S (2015) Objective image quality assessment of 3D synthesized views. Signal Process: Image Comm 30(0):78–88

    Google Scholar 

  2. Banitalebi-Dehkordi A, Nasiopoulos P (2018) Saliency inspired quality assessment of stereoscopic 3d video. Multimed Tools Appl 77(19):26055–26082

    Article  Google Scholar 

  3. Benoit A, Le Callet P, Campisi P, Cousseau R (2009) Quality assessment of stereoscopic images. EURASIP J Image Video Process. 2008(1):659024

    Google Scholar 

  4. Bosc E, Pepion R, Le Callet P, Koppel M, Ndjiki-Nya P, Pressigout M, Morin L (2011) Towards a new quality metric for 3-d synthesized view assessment. IEEE J Sel Topics Signal Process. 5(7):1332–1343

    Article  Google Scholar 

  5. Bosc E, Le Callet P, Morin L, Pressigout M (2012) An edge-based structural distortion indicator for the quality assessment of 3D synthesized views. In Proc Pict Coding Symp (PCS)249-252

  6. Campisi P, Le Callet P, Marini E (2007) Stereoscopic images quality assessment. In 15th Europ Signal Process Conf pages 2110–2114

  7. Canny J (1986) A computational approach to edge detection. IEEE Trans Pattern Anal Mach Intell 8(6):679–698

    Article  Google Scholar 

  8. Chandler DM, Hemami SS (2007) VSNR: A Wavelet-Based Visual Signal-to-Noise Ratio for Natural Images. IEEE Trans Image Process 16(9):2284–2298

    Article  MathSciNet  Google Scholar 

  9. Chen L, Zhao J (2019) Perceptual quality assessment of stereoscopic images based on local and global visual characteristics. Multimed Tools Appl 78(9):12139–12156

    Article  Google Scholar 

  10. Chen M-J, Su C-C, Kwon D-K, Cormack LK, Bovik AC (2013) Full-reference quality assessment of stereopairs accounting for rivalry. Signal Process-Image Commun 28(9):1143 – 1155

  11. Criminisi A, Perez P, Toyama K (2004) Region filling and object removal by exemplar-based image inpainting. IEEE Trans Image Process 13(9):1200–1212

    Article  Google Scholar 

  12. Domański M, Stankiewicz O, Wegner K, Kurc M, Konieczny J, Siast J, Stankowski J, Ratajczak R, Grajek T (2013) High efficiency 3D video coding using new tools based on view synthesis. IEEE Trans. Image Process. 22(9):3517–3527

    Article  Google Scholar 

  13. Fan Y, Larabi M, Alaya Cheikh F, Fernandez-Maloigne C (2019) A survey of stereoscopic 3D just noticeable difference models. IEEE Access 7:8621–8645

  14. Farid MS, Lucenteforte M, Grangetto M (2013) Edges shape enforcement for visual enhancement of depth image based rendering. In Proc Int Workshop Multimed Signal Process (MMSP) 406–411

  15. Farid MS, Lucenteforte M, Grangetto M (2014) Edge enhancement of depth based rendered images. In Proc. Int Conf Image Process (ICIP) 5452–5456

  16. Farid MS, Lucenteforte M, Grangetto M (2015) Objective quality metric for 3D virtual views. In Proc Int Conf Image Process (ICIP) 3720–3724

  17. Farid MS, Lucenteforte M, Grangetto M (2015) Panorama view with spatiotemporal occlusion compensation for 3D video coding. IEEE Trans. Image Process. 24(1):205–219

    Article  MathSciNet  Google Scholar 

  18. Farid MS, Lucenteforte M, Grangetto M (2017) Perceptual quality assessment of 3D synthesized images. In Proc IEEE Int Conf Multimed and Expo (ICME) 505–510

  19. Farid MS, Lucenteforte M, Grangetto M (2018) Evaluating virtual image quality using the side-views information fusion and depth maps. Inf Fusion 43:47 – 56

  20. Fehn C (2004) Depth-image-based rendering (DIBR), compression, and transmission for a new approach on 3D-TV. In Proc SPIE 5291:93–104

    Article  Google Scholar 

  21. Farid MS, Lucenteforte M, Grangetto M (2013) Depth image based rendering with inverse mapping. In Proc. Int Workshop Multimed Signal Process. (MMSP) 135–140

  22. Farid MS, Lucenteforte M, Grangetto M (2020) No-reference quality metric for hevc compression distortion estimation in depth maps. Signal Image Vid Process 14(1):195–203

    Article  Google Scholar 

  23. Gorley P, Holliman N (2008) Stereoscopic image quality metrics and compression. In Proc SPIE Stereos Displays and Apps XIX 6803, 680305–12

  24. Guan-Ming S, Yu-Chi L, Andres K, Haohong W (2011) 3D video communications: Challenges and opportunities. Int J Comm Sys 24(10):1261–1281

  25. Heo YS, Lee KM, Lee SU (2013) Joint depth map and color consistency estimation for stereo images with different illuminations and cameras. IEEE Trans Pattern Anal Mach Intell 35(5):1094–1106

    Article  Google Scholar 

  26. Joveluro P, Malekmohamadi H, Fernando WAC, Kondoz AM (2010) Perceptual video quality metric for 3d video quality assessment. In Proc 3DTV Conf True Vis-Capture Transmiss Display 3D Video (3DTV-CON) 1–4

  27. Julesz B (1972) Cyclopean perception and neurophysiology. Investigative Ophthalmol & Vis Sci 11(6), 540–548

    Google Scholar 

  28. Karimi M, Soltanian N, Samavi S, Najarian K, Karimi N, Reza SM (2019) Soroushmehr. Blind stereo image quality assessment inspired by brain sensory-motor fusion. Digital Signal Process 91:91 – 104. Quality Perception of Advanced Multimedia Systems

  29. Kim D, Ryu S, Sohn K (2012) Depth perception and motion cue based 3D video quality assessment. In Proc IEEE Int Symp.Broadband Multimed Sys Broadcast. (BMSB) pages 1–4

  30. Koppel M, Ndjiki-Nya P, Doshkov D, Lakshman H, Merkle P, Müller K, Wiegand T (2010) Temporally consistent handling of disocclusions with texture synthesis for depth-image-based rendering. In Proc Int Conf Image Process (ICIP) 1809–1812

  31. Ling S, Le Callet P (2017) Image quality assessment for free viewpoint video based on mid-level contours feature. In Proc IEEE Int Conf Multimed and Expo (ICME) 79–84

  32. Liu X, Zhang Y, Hu S, Kwong S, Kuo CCJ, Peng Q (2015) Subjective and objective video quality assessment of 3D synthesized views with texture/depth compression distortion. IEEE Trans on Image Process 24(12):4847–4861

    Article  MathSciNet  Google Scholar 

  33. Mitsa T, Varkur KL (1993) Evaluation of contrast sensitivity functions for the formulation of quality measures incorporated in halftoning algorithms. In IEEE ICASSP 5:301–304

  34. Mori Y, Fukushima N, Yendo T, Fujii T, Tanimoto M (2009) View generation with 3D warping using depth information for ftv. Signal Process-Image Commun 24(1):65–72. Special issue on advances in three-dimensional television and video

  35. Muller K, Merkle P, Tech G, Wiegand T (2010) 3D video formats and coding methods. In Proc. Int. Conf. Image Process. (ICIP) 2389–2392

  36. Ndjiki-Nya P, Koppel M, Doshkov D, Lakshman H, Merkle P, Müller K, Wiegand T (2010) Depth image based rendering with advanced texture synthesis. In Proc IEEE Int Conf Multimed and Expo (ICME) 424–429

  37. Ndjiki-Nya P, Koppel M, Doshkov D, Lakshman H, Merkle P, Muller K, Wiegand T (2011) Depth image-based rendering with advanced texture synthesis for 3-d video. IEEE Trans Multimed 13(3):453–465

    Article  Google Scholar 

  38. Rahaman DMM, Paul M (2018) Virtual view synthesis for free viewpoint video and multiview video compression using gaussian mixture modelling. IEEE Trans Image Process 27(3):1190–1201

    Article  MathSciNet  Google Scholar 

  39. Ryu S, Hyun Kim D, Sohn K (2012) Stereoscopic image quality metric based on binocular perception model. In Proc Int Conf Image Process (ICIP)609–612

  40. Sandic-Stankovic D, Kukolj D, Le Callet P (2015) Dibr synthesized image quality assessment based on morphological wavelets. In 2015 Seventh Int Workshop on Qual of Multimed Exp (QoMEX) 1–6

  41. Sandić-Stanković D, Battisti F, Kukolj D, Le Callet P, Carli M (2016) Free viewpoint video quality assessment based on morphological multiscale metrics. In 2016 Eighth Int Conf Qual Multimed Exp (QoMEX) 1–6

  42. Sandić-Stanković D, Kukolj D, Le Callet P (2016) Dibr-synthesized image quality assessment based on morphological multi-scale approach. EURASIP J Image and Vid Process 2017(1):4

    Article  Google Scholar 

  43. Sheikh HR, Bovik AC, de Veciana G (2005) An information fidelity criterion for image quality assessment using natural scene statistics. IEEE Trans Image Process 14(12):2117–2128

    Article  Google Scholar 

  44. Sheikh HR, Bovik AC (2006) Image information and visual quality. IEEE Trans Image Process 15(2):430–444

    Article  Google Scholar 

  45. Sheikh HR, Sabir MF, Bovik AC (2006) A statistical evaluation of recent full reference image quality assessment algorithms. IEEE Trans on Image Process 15(11):3440–3451

    Article  Google Scholar 

  46. Shao F, yi Jiang G, Yu M, Li M, Peng Z, Fu R (2014) Binocular energy response based quality assessment of stereoscopic images. Digital Signal Process 29:45 – 53

  47. Shao F, Chen W, Lin W, Jiang Q, Jiang G (2016) Simulating receptive fields of human visual cortex for 3D image quality prediction. Appl Opt 55(21):5488–5496

    Article  Google Scholar 

  48. Tian S, Zhang L, Morin L, Deforges O (2017) Niqsv: A no reference image quality assessment metric for 3D synthesized views. In Proc Int Conf Acoust Speech and Signal Process (ICASSP) 1248–1252

  49. Tsai C-T, Hang H-M (2013) Quality assessment of 3D synthesized views with depth map distortion. In Proc Int. Conf Vis Commun Image Process (VCIP) 1–6

  50. Telea A (2004) An image inpainting technique based on the fast marching method. J Graphics Tools 9(1):23–34

    Article  Google Scholar 

  51. Video Quality Expert Group (2003) Final report from the video quality experts group on the validation of objective models of video quality assessment, phase II. http://www.its.bldrdoc.gov/vqeg/projects/frtv-phase-ii/frtv-phase-ii.aspx

  52. Voo KHB, Bong DBL (2018) Quality assessment of stereoscopic image by 3d structural similarity. Multimed Tools Appl 77(2):2313–2332

    Article  Google Scholar 

  53. Wan Z, Qi F, Liu Y, Zhao D (2017) Reduced reference stereoscopic image quality assessment based on entropy of classified primitives. In Proc IEEE Int Conf Multimed and Expo (ICME) 73–78

  54. Wang Z, Bovik AC (2002) A universal image quality index. IEEE Signal Process Lett 9(3):81–84

  55. Wang Z, Bovik AC, Sheikh HR, Simoncelli EP (2004) Image quality assessment: from error visibility to structural similarity. IEEE Trans Image Process 13(4):600–612

  56. Wang J, Rehman A, Zeng K, Wang S, Wang Z (2015) Quality prediction of asymmetrically distorted stereoscopic 3D images. IEEE Trans Image Process 24(11):3400–3414

    Article  MathSciNet  Google Scholar 

  57. Wang Z, Simoncelli EP, Bovik AC (2003) Multiscale structural similarity for image quality assessment. In The Thrity-Seventh Asilomar Conf on Signals, Sys Comp 2003 2:1398–1402

  58. You J, Xing L, Perkis A, Wang X (2010) Perceptual quality assessment for stereoscopic images based on 2D image quality metrics and disparity analysis. In Proc Int Workshop Vid Process Qual Metrics Consum Electron 1–5

  59. Zijin G, Ding Y, Deng R, Chen X, Krylov AS (2019) Multiple just-noticeable-difference-based no-reference stereoscopic image quality assessment. Appl Opt 58(2):340–352

    Article  Google Scholar 

  60. Zhou W, Yu L (2016) Binocular responses for no-reference 3D image quality assessment. IEEE Trans Broadcast 18(6):1077–1084

    MathSciNet  Google Scholar 

  61. Zhou J, Wang L, Yin H, Bovik AC (2019) Eye movements and visual discomfort when viewing stereoscopic 3d content. Digital Signal Process 91:41 – 53 Quality Perception of Advanced Multimedia Systems

  62. Zhou W, Zhou Y, Qiu W, Luo T, Zhai Z (2019) Perceived quality measurement of stereoscopic 3d images based on sparse representation and binocular combination. Digital Signal Process 93:128–137

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of Computer Science, University of the Punjab, Lahore, 54000, Pakistan

    Tehreem Fatima & Muhammad Shahid Farid

Authors
  1. Tehreem Fatima
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Muhammad Shahid Farid
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Muhammad Shahid Farid.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Fatima, T., Farid, M.S. Quality assessment of 3D synthesized images based on structural and textural distortion. Multimed Tools Appl 80, 36443–36463 (2021). https://doi.org/10.1007/s11042-021-11382-1

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-021-11382-1

Keywords

Search

Navigation