Skip to main content
SpringerLink
Log in

Hole filling using multiple frames and iterative texture synthesis with illumination compensation

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

In this paper, we propose a novel freeview generation system. We first discuss the problem of misalignment of the edges between color frame and depth frame, and propose an optimization method to solve it. Then an original frame is separated into two parts, depending on the sizes of the resulting dis-occluded areas in the virtual view. Illumination changes between current frame and the information from previous frames are modeled and compensated. In the virtual view, the missing pixels are recovered by the proposed inpainting method. Using depth information, we compute the priority of the recovering order among the missing pixels. The recovering process accounts for pixel consistency, and the process iterates between virtual color frame and depth frame. Experimental results show that, compared with a state-of-the-art work, the proposed method has better subjective and objective performances; the PSNR gain can be up to 2.9575 dB.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13

Similar content being viewed by others

References

  1. Bertalmio M, Sapiro G, Caselles V, Ballester C (2000) Image inpainting. In: SIGGRAPH '00 Proceedings of the 27th Annual Conference on Computer Graphics and Interactive Techniques, ACM Press/Addison-Wesley Publishing Co., New York, pp 417–424

  2. Criminisi A, Pérez P, Toyama K (2004) Region filling and object removal by exemplar-based image inpainting. IEEE Trans Image Process 13:1200–1212

    Article  Google Scholar 

  3. Dong T, Lai PL, Patrick L, Gomila C (2009) View synthesis techniques for 3D video. In: Proc. SPIE 7443, Applications of Digital Image Processing XXXII, 74430T. doi:10.1117/12.829372

  4. Fehn C (2004) Depth-image-based rendering (DIBR), compression, and transmission for a new approach on 3D-TV. In: Proc. SPIE 5291, Stereoscopic Displays and Virtual Reality Systems XI, 93. doi:10.1117/12.524762

  5. Hervieu A, Papadakis N, Bugeau A, Gargallo P, Caselles V (2010) Stereoscopic image inpainting: distinct depth maps and images inpainting. Pattern Recognition (ICPR), 2010 20th International Conference on, pp 4101–4104. doi:10.1109/ICPR.2010.997

  6. Kauff P, Atzpadin N, Fehn C, Müller M, Smolic A, Tanger R (2007) Depth map creation and image-based rendering for advanced 3DTV services providing interoperability and scalability. Sig Process Image Commun Spec Issue Three-Dim VIdeo Telev 22:217–234

    Google Scholar 

  7. Ndjiki-Nya P, Koppel M, Doshkov D, Lakshman H, Merkle P, Muller K, Wiegand T (2011) Depth image based rendering with advanced texture synthesis for 3-D video. IEEE Trans Multimed 13:453–465

    Article  Google Scholar 

  8. Schmeing M, Jiang X (2010) Depth Image Based Rendering: A faithful approach for the disocclusion problem, 3DTV-Conference: The True Vision - Capture, Transmission and Display of 3D Video (3DTV-CON), pp 1-4. doi:10.1109/3DTV.2010.5506596

  9. Smolic A, Mueller K, Merkle P (2006) 3D video and free viewpoint video technologies, applications and MPEG standards. IEEE Int Conf Multimed Expo 53:2161–2164

    Google Scholar 

  10. Smolic A, Muller K, Dix K, Merkle P, Kauff P, Wiegand T (2008) Intermediate view interpolation based on multiview video plus depth for advanced 3D video systems. IEEE Int Conf Image Process 11:2448–2451

    Google Scholar 

  11. Solh M, AlRegib G (2012) Hierarchical Hole-Filling For Depth-Based View Synthesis in FTV and 3D Video. IEEE Journal of Selected Topics in Signal Processing 6(5):495–504

  12. Wang L, Jin H, Yang R, Gong M (2008) Stereoscopic inpainting: joint color and depth completion from stereo images. IEEE Conf Comput Vis Pattern Recog 10:1–8

    MATH  Google Scholar 

  13. Zhang L, Tam W (2005) Stereoscopic image generation based on depth images for 3D TV. IEEE Trans Broadcast 51:191–199

    Article  Google Scholar 

Download references

Acknowledgments

This research is supported by the National Science Council, Taiwan, under Grants NSC 101-2221-E-033-036, NSC 102-2221-E-033-018, NSC-102-2221-E-033-063, by the Ministry of Science and Technology, Taiwan under Grant MOST 103-2221-E-033-020, MOST-103-2221-E-033-070, MOST-103-2622-E-033-001-CC2, and by the College of Electrical Engineering and Computer Science in Chung Yuan Christian University, Taiwan under Grant CYCU-EECS-10301.

Author information

Authors and Affiliations

  1. Department of Electronic Engineering, Chung Yuan Christian University, 200 Zhongbei Rd., Zhongli City, Taoyuan County, 320, Taiwan, Republic of China

    Ting-Lan Lin, Chi-Chan Chou & Shih-Lun Chen

  2. Institut für Nachrichtentechnik, RWTH Aachen University, Aachen, Germany

    Uday Singh Thakur

Authors
  1. Ting-Lan Lin
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Uday Singh Thakur
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Chi-Chan Chou
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Shih-Lun Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Ting-Lan Lin.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Lin, TL., Thakur, U.S., Chou, CC. et al. Hole filling using multiple frames and iterative texture synthesis with illumination compensation. Multimed Tools Appl 75, 1899–1921 (2016). https://doi.org/10.1007/s11042-014-2379-2

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-014-2379-2

Keywords

Search

Navigation