Abstract
In this paper, we propose a novel algorithm to generate multiple virtual views from a video-plus-depth sequence for modern autostereoscopic displays. To synthesize realistic content in the disocclusion regions from the virtual views is the main challenging problem in this task. In order to produce perceptually satisfactory images, our proposed algorithm takes advantage of spatial coherence and temporal consistency to handle the uncertain pixels in the disocclusion regions. On the one hand, regarding the spatial coherence, we incorporate the intensity gradient strength with the depth information to determine the filling priority for inpainting the disocclusion regions, so that the continuity of image structures can be preserved. On the other hand, the temporal consistency is enforced by considering the intensities in the disocclusion regions across the adjacent frames through an optimization process. We propose an iterative re-weighted framework to jointly consider intensity and depth consistency in the adjacent frames, which not only imposes temporal consistency but also reduces noise disturbance. Finally, for accelerating the multi-view synthesis process, we apply the proposed view synthesis algorithm to generate the images plus depth at the leftmost and rightmost viewpoints, so that the intermediate views are efficiently interpolated through image warping according to the associated depth maps between the two views. In the experimental validation, we perform quantitative evaluation on synthetic data as well as subjective assessment on real video data with comparisons to some previous representative methods to demonstrate the superior performance of the proposed method.
Keywords
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
Criminisi, A., Pérez, P., Toyama, K.: Object removal by exemplar-based inpainting. In: IEEE Int’l Conf. on Computer Vision and Pattern Recognition, Wisconsin, vol. 2(2), pp. 721–728 (2003)
Tomasi, C., Manduchi, R.: Bilateral Filtering for Gray and Color Images. In: IEEE Int’l Conf. on Computer Vision, Washington, DC, pp. 839–846 (1998)
Berretty, R.-P.M., Peters, F.J., Volleberg, G.T.G.: Real Time Rendering for Multiview Autostereoscopic Displays. In: SPIE Proc. on Stereoscopic Displays and Virtual Reality Systems XIII, San Jose, vol. 6055, pp. 208–219 (2006)
Christoph, F.: A 3D-TV System Based on Video plus Depth Information. In: Conference Record of the 37th Asilomar Conf. on Signal, System and Computers, vol. 2, pp. 1529–1533 (2003)
Zhang, L., Tam, W.J.: Stereoscopic Image Generation Based on Depth Images for 3D TV. IEEE Trans. Broadcasting 51(2), 191–199 (2005)
Patwardhan, K.A., Sapiro, G., Bertalmio, M.: Video Inpainting under Constrained Camera Motion. IEEE Trans. Image Processing 16, 545–553 (2007)
Atzpadin, N., Kauff, P., Schreer, O.: Stereo Analysis by Hybrid Recursive Matching for Real-Time Immersive Video Conferencing. IEEE Trans. Circuits and Systems for Video Technology 14(3), 321–334 (2004)
Patwardhan, K.A., Sapiro, G., Bertalmio, M.: Video Inpainting of Occluding and Occluded Objects. In: IEEE Int’l Conf. Image Processing, Italy, vol. 2, pp. 69–72 (2005)
Zhang, G., Jia, J., Wong, T.T., Bao, H.: Consistent Depth Maps Recovery from a Video Sequence. IEEE Trans. Pattern Analysis and Machine Intelligence 31(6), 974–988 (2009)
Mobile 3DTV, http://sp.cs.tut.fi/mobile3dtv/stereo-video/
Scharstein, D., Szeliski, R.: A Taxonomy and Evaluation of Dense Two-frame Stereo Correspondence Algorithms. Int’l Journal of Computer Vision 47, 7–42 (2002)
Mark, W., Mcmillan, L., Bishop, G.: Post-rendering 3D Warping. In: Symposium on Interactive 3D Graphics, pp. 7–16. ACM Press, New York (1997)
McMillan, L.: An Image-based Approach on Three-dimensional Computer Graphics. Ph.D. Dissertation, University of North Carolina at Chapel Hill (1997)
Bertalmio, M., Sapiro, G., Ballester, C., Caselles, V.: Image Inpainting. In: SIGGRAPH 2000: Proceedings of the 27th Annual Conf. Computer Graphics and Interactive Techniques, pp. 417–424. ACM Press, New York (2000)
Telea, A.: An Image Inpainting Technique Based on the Fast Marching Method. Journal of Graphics, GPU and Game Tools 9(1), 23–34 (2004)
Wang, L., Jin, H., Yang, R., Gong, M.: Stereoscopic Inpainting: Joint Color and Depth Completion from Stereo Images. In: IEEE Int’l Conf. Computer Vision and Pattern Recognition, Alaska, pp. 1–8 (2008)
Cheng, C.M., Lin, S.J., Lai, S.H., Yang, J.C.: Improved Novel View Synthesis from Depth Image with Large Baseline. In: IEEE Int’l Conf. Pattern Recognition, Tampa, pp. 1–4 (2008)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2009 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Lin, SJ., Cheng, CM., Lai, SH. (2009). Spatio-temporally Consistent Multi-view Video Synthesis for Autostereoscopic Displays. In: Muneesawang, P., Wu, F., Kumazawa, I., Roeksabutr, A., Liao, M., Tang, X. (eds) Advances in Multimedia Information Processing - PCM 2009. PCM 2009. Lecture Notes in Computer Science, vol 5879. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-10467-1_47
Download citation
DOI: https://doi.org/10.1007/978-3-642-10467-1_47
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-642-10466-4
Online ISBN: 978-3-642-10467-1
eBook Packages: Computer ScienceComputer Science (R0)