skip to main content
10.1145/3240508.3240514acmconferencesArticle/Chapter ViewAbstractPublication PagesmmConference Proceedingsconference-collections
research-article

Robust Billboard-based, Free-viewpoint Video Synthesis Algorithm to Overcome Occlusions under Challenging Outdoor Sport Scenes

Published:15 October 2018Publication History

ABSTRACT

The paper proposes an algorithm to robustly reconstruct an accurate billboard model of an individual object including an occluded one in each camera. Each billboard model is utilized to synthesize high-quality, free-viewpoint video especially for outdoor sport scenes in which roughly calibrated cameras are sparsely placed. The two main contributions of the proposed algorithm are (1) robustness to occlusions caused by overlaps of multiple objects in every camera, that is one of the biggest issues for billboard-based method, and (2) applicability to challenging shooting conditions in which accurate 3D model cannot be reconstructed because of calibration errors, small number of cameras and so on. In order to achieve the contributions above, the algorithm does not try to reproduce an accurate 3D model of each object but utilize a "rough 3D model". The algorithm precisely extracts an individual object region in every camera by reconstructing a "rough 3D model" of each object and back-projecting it to every camera. The 3D coordinate for each billboard to be located is calculated based on the position of a rough 3D model. Experimental results compare the visual quality of free-viewpoint videos synthesized with our proposed method and conventional methods and show the effectiveness of our proposed method in terms of the naturalness of positional relationships and the fineness of the surface textures of all the objects.

References

  1. T. Kanade, P. W. Rander, and P. J. Narayanan, "Virtualized Reality: Constructing Virtual Worlds from Real Scenes," IEEE Multimedia, Vol. 4, No. 1, pp. 34--47, 1997. Google ScholarGoogle ScholarDigital LibraryDigital Library
  2. T. Fujii and M. Tanimoto, "Free Viewpoint TV System based on Ray-space Representation," in Proc. ITCom 2002: The Convergence of Information Technologies and Communications, pp. 175--189, 2002.Google ScholarGoogle Scholar
  3. A. Ishikawa, M. P. Tehrani, S. Naito, S. Sakazawa, and A. Koike, "Free Viewpoint Video Generation for Walk-through Experience using Image-based Rendering," In Proc of the 16th ACM international conference on Multimedia, pp.1007--1008, 2008. Google ScholarGoogle ScholarDigital LibraryDigital Library
  4. G. K. M. Cheung, T. Kanade, J. Y. Bouguet, and M. Holler, "A Real Time System for Robust 3d Voxel Reconstruction of Human Motions," IEEE conference on Computer Vision and Pattern Recognition, Vol. 2, pp. 714--720, 2000.Google ScholarGoogle Scholar
  5. T. Matsuyama, X. Wu, T. Takai, and T. Wada, "Real-Time Dynamic 3D Object Shape Reconstruction and High-FidelityTexture Mapping for 3D Video," IEEE Trans. on Circuits and Systems for Video Technology, Vol. CSVT-14, No. 3, pp. 357--369, 2004. Google ScholarGoogle ScholarDigital LibraryDigital Library
  6. J. Starck, and A. Hilton, "Surface Capture for Performance-Based Animation," IEEE Computer Graphics and Applications, Vol. 27, No. 3, pp. 21--31, 2007. Google ScholarGoogle ScholarDigital LibraryDigital Library
  7. T. Kanade et al., ''Eye Vision,''http://www.ri.cmu.edu/events/sb35/tksuperbowl.htmlGoogle ScholarGoogle Scholar
  8. C. Zhang and T. Chen, "A Survey on Image-based Rendering - Representation, Sampling and Compression," Signal Processing: Image Communication, Vol. 19, No. 1, pp. 1--28, 2004.Google ScholarGoogle ScholarDigital LibraryDigital Library
  9. N. Inamoto and H. Saito, "Virtual Viewpoint Replay for a Soccer Match by View Interpolation from Multiple Cameras," IEEE trans. Multimedia, Vol. 9, No. 6, pp.1155--1166, 2007. Google ScholarGoogle ScholarDigital LibraryDigital Library
  10. H. Y. Shum, S. C. Chan, and S. B. Kang, "Image-Based Rendering," Springer, 2008. Google ScholarGoogle ScholarDigital LibraryDigital Library
  11. K. Hayashi, and H. Saito, "Synthesizing Free-viewpoint Images from Multiple View Videos in Soccer Stadium," In Proc. IEEE Conference Computer Graphics, Imaging and Visualization (CGIV 2006), pp. 220--225, 2006. Google ScholarGoogle ScholarDigital LibraryDigital Library
  12. Y. Ohta, I. Kitahara, Y. Kameda, H. Ishikawa, and T. Koyama, "Live 3D Video in Soccer Stadium," International Journal of Computer Vision (IJCV), Vol. 75, No. 1, pp. 173--187, 2007. Google ScholarGoogle ScholarDigital LibraryDigital Library
  13. J-Y. Guillemaut, J. Kilner, and A. Hilton, "Robust Graph-cut Scene Segmentation and Reconstruction for Free-viewpoint Video of Complex Dynamic Scenes," 2009 IEEE Conference on Computer Vision (ICCV), pp. 809 - 816, 2009.Google ScholarGoogle Scholar
  14. A. Hilton, J-Y. Guillemaut, J. Kilner, O. Grau, and T. Graham, " 3D-TV Production from Conventional Cameras for Sports Broadcast," IEEE trans. Broadcasting, Vol. 57, No. 2, pp. 462--476, 2011.Google ScholarGoogle ScholarCross RefCross Ref
  15. M. Germann, A. Hornung, R. Keiser, R. Ziegler, S. Wurmlin, and M. Gross, "Articulated Billboards for Video-based Rendering," In Proc. EUROGRAPHICS, pp. 585--594, 2010.Google ScholarGoogle Scholar
  16. K. Yamada, H. Sankoh, M. Sugano, and S. Naito, "Occlusion Robust Free-viewpoint Video Synthesis based on Inter-Camera/-Frame Interpolation," In Proc. 2013 IEEE International Conference on Image Processing (ICIP), pp. 2072 - 2076 (2013).Google ScholarGoogle ScholarCross RefCross Ref
  17. D. Farin, J. Han, P. H. N. de With, "Fast Camera Calibration for the Analysis of Sport Sequences," In Proc. IEEE International Conference on Multimedia and Expo (ICME 2005), 2005.Google ScholarGoogle ScholarCross RefCross Ref
  18. J. Han, D. Farin, and P. H. N. de With, "Broadcast Court-Net Sports Video Analysis Using Fast 3-D Camera Modeling," IEEE trans. Circuits and Systems for Video Technology (TCSVT), Vol. 18, No. 11, pp. 1628--1638, 2008. Google ScholarGoogle ScholarDigital LibraryDigital Library
  19. H. Sankoh, M. Sugano, and S. Naito "Dynamic Camera Calibration Method for Free-viewpoint Experience in Sport Videos," In Proc. ACM Conference on Multimedia (MM'12), pp. 1125 - 1128, 2012. Google ScholarGoogle ScholarDigital LibraryDigital Library
  20. W. N. Martin and J. K. Aggarwal, "Volumetric Description of Objects from Multiple Views," IEEE Trans. Pattern Analysis and Machine Intelligence, Vol. 5, No. 2, pp. 150--158, 1983. Google ScholarGoogle ScholarDigital LibraryDigital Library
  21. S. M. Seitz, and C. R. Dyer, "View Morphing," In Proc. ACM SIGGRAPH, pp. 21--30, 1996. Google ScholarGoogle ScholarDigital LibraryDigital Library
  22. K. N. Kutulakos and S. M. Seitz, "A Theory of Shape by Space Carving," Int. J. Comput. Vis., Vol. 38, No. 3, pp. 192--218, 2000. Google ScholarGoogle ScholarDigital LibraryDigital Library
  23. W. E. Lorensen and H. E. Cline, "Marching Cubes: A High Resolution 3d Surface Construction Algorithm," In Proc. ACM SIGGRAPH, Vol. 21, No. 4, pp. 163--169, 1987. Google ScholarGoogle ScholarDigital LibraryDigital Library
  24. Z. Wang, A. C. Bovik, H. R. Sheikh, and E. P. Simoncelli, "Image quality assessment: from error visibility to structural similarity,'' IEEE Transactions on Image Processing, Vol. 13, No. 4, pp. 600--612, April 2004. Google ScholarGoogle ScholarDigital LibraryDigital Library

Index Terms

  1. Robust Billboard-based, Free-viewpoint Video Synthesis Algorithm to Overcome Occlusions under Challenging Outdoor Sport Scenes

      Recommendations

      Comments

      Login options

      Check if you have access through your login credentials or your institution to get full access on this article.

      Sign in
      • Published in

        cover image ACM Conferences
        MM '18: Proceedings of the 26th ACM international conference on Multimedia
        October 2018
        2167 pages
        ISBN:9781450356657
        DOI:10.1145/3240508

        Copyright © 2018 ACM

        Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

        Publisher

        Association for Computing Machinery

        New York, NY, United States

        Publication History

        • Published: 15 October 2018

        Permissions

        Request permissions about this article.

        Request Permissions

        Check for updates

        Qualifiers

        • research-article

        Acceptance Rates

        MM '18 Paper Acceptance Rate209of757submissions,28%Overall Acceptance Rate995of4,171submissions,24%

        Upcoming Conference

        MM '24
        MM '24: The 32nd ACM International Conference on Multimedia
        October 28 - November 1, 2024
        Melbourne , VIC , Australia

      PDF Format

      View or Download as a PDF file.

      PDF

      eReader

      View online with eReader.

      eReader