ABSTRACT
The paper proposes an algorithm to robustly reconstruct an accurate billboard model of an individual object including an occluded one in each camera. Each billboard model is utilized to synthesize high-quality, free-viewpoint video especially for outdoor sport scenes in which roughly calibrated cameras are sparsely placed. The two main contributions of the proposed algorithm are (1) robustness to occlusions caused by overlaps of multiple objects in every camera, that is one of the biggest issues for billboard-based method, and (2) applicability to challenging shooting conditions in which accurate 3D model cannot be reconstructed because of calibration errors, small number of cameras and so on. In order to achieve the contributions above, the algorithm does not try to reproduce an accurate 3D model of each object but utilize a "rough 3D model". The algorithm precisely extracts an individual object region in every camera by reconstructing a "rough 3D model" of each object and back-projecting it to every camera. The 3D coordinate for each billboard to be located is calculated based on the position of a rough 3D model. Experimental results compare the visual quality of free-viewpoint videos synthesized with our proposed method and conventional methods and show the effectiveness of our proposed method in terms of the naturalness of positional relationships and the fineness of the surface textures of all the objects.
- T. Kanade, P. W. Rander, and P. J. Narayanan, "Virtualized Reality: Constructing Virtual Worlds from Real Scenes," IEEE Multimedia, Vol. 4, No. 1, pp. 34--47, 1997. Google ScholarDigital Library
- T. Fujii and M. Tanimoto, "Free Viewpoint TV System based on Ray-space Representation," in Proc. ITCom 2002: The Convergence of Information Technologies and Communications, pp. 175--189, 2002.Google Scholar
- A. Ishikawa, M. P. Tehrani, S. Naito, S. Sakazawa, and A. Koike, "Free Viewpoint Video Generation for Walk-through Experience using Image-based Rendering," In Proc of the 16th ACM international conference on Multimedia, pp.1007--1008, 2008. Google ScholarDigital Library
- G. K. M. Cheung, T. Kanade, J. Y. Bouguet, and M. Holler, "A Real Time System for Robust 3d Voxel Reconstruction of Human Motions," IEEE conference on Computer Vision and Pattern Recognition, Vol. 2, pp. 714--720, 2000.Google Scholar
- T. Matsuyama, X. Wu, T. Takai, and T. Wada, "Real-Time Dynamic 3D Object Shape Reconstruction and High-FidelityTexture Mapping for 3D Video," IEEE Trans. on Circuits and Systems for Video Technology, Vol. CSVT-14, No. 3, pp. 357--369, 2004. Google ScholarDigital Library
- J. Starck, and A. Hilton, "Surface Capture for Performance-Based Animation," IEEE Computer Graphics and Applications, Vol. 27, No. 3, pp. 21--31, 2007. Google ScholarDigital Library
- T. Kanade et al., ''Eye Vision,''http://www.ri.cmu.edu/events/sb35/tksuperbowl.htmlGoogle Scholar
- C. Zhang and T. Chen, "A Survey on Image-based Rendering - Representation, Sampling and Compression," Signal Processing: Image Communication, Vol. 19, No. 1, pp. 1--28, 2004.Google ScholarDigital Library
- N. Inamoto and H. Saito, "Virtual Viewpoint Replay for a Soccer Match by View Interpolation from Multiple Cameras," IEEE trans. Multimedia, Vol. 9, No. 6, pp.1155--1166, 2007. Google ScholarDigital Library
- H. Y. Shum, S. C. Chan, and S. B. Kang, "Image-Based Rendering," Springer, 2008. Google ScholarDigital Library
- K. Hayashi, and H. Saito, "Synthesizing Free-viewpoint Images from Multiple View Videos in Soccer Stadium," In Proc. IEEE Conference Computer Graphics, Imaging and Visualization (CGIV 2006), pp. 220--225, 2006. Google ScholarDigital Library
- Y. Ohta, I. Kitahara, Y. Kameda, H. Ishikawa, and T. Koyama, "Live 3D Video in Soccer Stadium," International Journal of Computer Vision (IJCV), Vol. 75, No. 1, pp. 173--187, 2007. Google ScholarDigital Library
- J-Y. Guillemaut, J. Kilner, and A. Hilton, "Robust Graph-cut Scene Segmentation and Reconstruction for Free-viewpoint Video of Complex Dynamic Scenes," 2009 IEEE Conference on Computer Vision (ICCV), pp. 809 - 816, 2009.Google Scholar
- A. Hilton, J-Y. Guillemaut, J. Kilner, O. Grau, and T. Graham, " 3D-TV Production from Conventional Cameras for Sports Broadcast," IEEE trans. Broadcasting, Vol. 57, No. 2, pp. 462--476, 2011.Google ScholarCross Ref
- M. Germann, A. Hornung, R. Keiser, R. Ziegler, S. Wurmlin, and M. Gross, "Articulated Billboards for Video-based Rendering," In Proc. EUROGRAPHICS, pp. 585--594, 2010.Google Scholar
- K. Yamada, H. Sankoh, M. Sugano, and S. Naito, "Occlusion Robust Free-viewpoint Video Synthesis based on Inter-Camera/-Frame Interpolation," In Proc. 2013 IEEE International Conference on Image Processing (ICIP), pp. 2072 - 2076 (2013).Google ScholarCross Ref
- D. Farin, J. Han, P. H. N. de With, "Fast Camera Calibration for the Analysis of Sport Sequences," In Proc. IEEE International Conference on Multimedia and Expo (ICME 2005), 2005.Google ScholarCross Ref
- J. Han, D. Farin, and P. H. N. de With, "Broadcast Court-Net Sports Video Analysis Using Fast 3-D Camera Modeling," IEEE trans. Circuits and Systems for Video Technology (TCSVT), Vol. 18, No. 11, pp. 1628--1638, 2008. Google ScholarDigital Library
- H. Sankoh, M. Sugano, and S. Naito "Dynamic Camera Calibration Method for Free-viewpoint Experience in Sport Videos," In Proc. ACM Conference on Multimedia (MM'12), pp. 1125 - 1128, 2012. Google ScholarDigital Library
- W. N. Martin and J. K. Aggarwal, "Volumetric Description of Objects from Multiple Views," IEEE Trans. Pattern Analysis and Machine Intelligence, Vol. 5, No. 2, pp. 150--158, 1983. Google ScholarDigital Library
- S. M. Seitz, and C. R. Dyer, "View Morphing," In Proc. ACM SIGGRAPH, pp. 21--30, 1996. Google ScholarDigital Library
- K. N. Kutulakos and S. M. Seitz, "A Theory of Shape by Space Carving," Int. J. Comput. Vis., Vol. 38, No. 3, pp. 192--218, 2000. Google ScholarDigital Library
- W. E. Lorensen and H. E. Cline, "Marching Cubes: A High Resolution 3d Surface Construction Algorithm," In Proc. ACM SIGGRAPH, Vol. 21, No. 4, pp. 163--169, 1987. Google ScholarDigital Library
- Z. Wang, A. C. Bovik, H. R. Sheikh, and E. P. Simoncelli, "Image quality assessment: from error visibility to structural similarity,'' IEEE Transactions on Image Processing, Vol. 13, No. 4, pp. 600--612, April 2004. Google ScholarDigital Library
Index Terms
Robust Billboard-based, Free-viewpoint Video Synthesis Algorithm to Overcome Occlusions under Challenging Outdoor Sport Scenes
Recommendations
Dynamic camera calibration method for free-viewpoint experience in sport videos
MM '12: Proceedings of the 20th ACM international conference on MultimediaIn this paper, we propose a dynamic camera calibration and object extraction method for sport videos captured with a moving pan-tilt-zoom camera. Such technology realizes an immersive free-viewpoint experience whereby audiences can see real sport scenes ...
A Point-Cloud-Based Multiview Stereo Algorithm for Free-Viewpoint Video
This paper presents a robust multiview stereo (MVS) algorithm for free-viewpoint video. Our MVS scheme is totally point-cloud-based and consists of three stages: point cloud extraction, merging, and meshing. To guarantee reconstruction accuracy, point ...
Multi-cue Based Visual Tracking in Clutter Scenes with Occlusions
AVSS '09: Proceedings of the 2009 Sixth IEEE International Conference on Advanced Video and Signal Based SurveillanceObject tracking is important for video analysis applications. However, tracking through occlusions is a difficult task due to significant appearance changes of the objects.Approaches based on either global features or one kind of local features can not ...
Comments