Skip to main content
SpringerLink
Log in

Multi-view 4D Reconstruction of Human Action for Entertainment Applications

  • Chapter
Book cover Visual Analysis of Humans

Abstract

Multi-view 4D reconstruction of human action has a number of applications in entertainment. This chapter describes a selection of application areas that are of interest to the broadcast, movie and gaming industries. In particular, free-viewpoint video techniques for special effects and sport post-match analysis are discussed. The appearance of human action is captured as 4D data represented by 3D volumetric or surface data over time. A review of recent approaches identifies two major classes: 4D reconstruction and model-based tracking. The second part of the chapter describes aspects of a practical implementation of a 4D reconstruction pipeline. Implementations of the popular visual hull are discussed, as a building block in many free-viewpoint video systems.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 129.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 169.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info
Hardcover Book
USD 169.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    Result from the TSB iview project [12].

  2. 2.

    This is called ‘ingest’ in broadcast.

  3. 3.

    Volumetric elements.

References

  1. Time slice films. http://www.timeslicefilms.com/

  2. Bai, X., Wang, J., Simons, D., Sapiro, G.: Video SnapCut: Robust video object cutout using localized classifiers. In: ACM SIGGRAPH 2009 papers, pp. 1–11. ACM, New York (2009)

    Chapter  Google Scholar 

  3. Cagniart, C., Boyer, E., Ilic, S.: Probabilistic deformable surface tracking from multiple videos. In: ECCV 2010, pp. 326–339 (2010)

    Chapter  Google Scholar 

  4. Carranza, J., Theobalt, C., Magnor, M., Seidel, H.-P.: Free-viewpoint video of human actors. ACM Trans. Graph. 22(3), 569–577 (2003)

    Article  Google Scholar 

  5. Chuang, Y.-Y., Curless, B., Salesin, D.H., Szeliski, R.: A Bayesian approach to digital matting. In: Proceedings of IEEE CVPR 2001, vol. 2, pp. 264–271. IEEE Comput. Soc., Los Alamitos (December 2001)

    Google Scholar 

  6. Easterbrook, J., Grau, O., Schübel, P.: A system for distributed multi-camera capture and processing. In: Proc. of CVMP (2010)

    Google Scholar 

  7. Furukawa, Y., Ponce, J.: Carved visual hulls for image-based modeling. Int. J. Comput. Vis. 81(1), 53–67 (2009)

    Article  Google Scholar 

  8. Grau, O.: 3D sequence generation from multiple cameras. In: Proc. of IEEE, International Workshop on Multimedia Signal Processing 2004, Siena, Italy (September 2004)

    Google Scholar 

  9. Grau, O.: A 3D production pipeline for special effects in TV and film. In: Mirage 2005, Computer Vision/Computer Graphics Collaboration Techniques and Applications, Rocquencourt, France. INRIA, Rocquencourt (March 2005)

    Google Scholar 

  10. Grau, O., Easterbrook, J.: Effects of camera aperture correction on keying of broadcast video. In: Proc. of the 5th European Conference on Visual Media Production (CVMP) (2008)

    Google Scholar 

  11. Grau, O., Pullen, T., Thomas, G.A.: A combined studio production system for 3-d capturing of live action and immersive actor feedback. IEEE Trans. Circuits Syst. Video Technol. 14(3), 370–380 (2004)

    Article  Google Scholar 

  12. Grau, O., Thomas, G.A., Hilton, A., Kilner, J., Starck, J.: A robust free-viewpoint video system for sport scenes. In: Proc. of 3DTV-Conference, Kos island, Greece (May 2007)

    Google Scholar 

  13. Guillemaut, J.Y., Kilner, J., Hilton, A.: Robust graph-cut scene segmentation and reconstruction for free-viewpoint video of complex dynamic scenes. In: Computer Vision, 2009 IEEE 12th International Conference on, pp. 809–816. IEEE Comput. Soc., Los Alamitos (2010)

    Google Scholar 

  14. Hartley, R.I., Zisserman, A.: Multiple View Geometry in Computer Vision. Cambridge University Press, Cambridge (2000)

    MATH  Google Scholar 

  15. Hernández Esteban, C., Schmitt, F.: Silhouette and stereo fusion for 3D object modeling. Comput. Vis. Image Underst. 96(3), 367–392 (2004)

    Article  Google Scholar 

  16. Hillman, P., Hannah, J., and Renshaw, D.. Foreground/background segmentation of motion picture images and image sequences. IEE Proc., Vis. Image Signal Process. 152(4), 387–397 (2005)

    Article  Google Scholar 

  17. Kanade, T., et al.: Eyevision at super bowl XXXV. Web (2001)

    Google Scholar 

  18. Kappei, F., Liedtke, C.-E.: Ein verfahren zur modellierung von 3d-objekten aus fernsehbildfolgen. In: Mustererkennung 1987, 9. DAGM-Symposium, pp. 277–281 (1987)

    Google Scholar 

  19. Koch, R.: Model-based 3-d scene analysis from stereoscopic image sequences. ISPRS J. Photogramm. Remote Sens. 49(5), 23–30 (1994)

    Article  Google Scholar 

  20. Koch, R.. Dynamic 3-d scene analysis through synthesis feedback control. IEEE Trans. Pattern Anal. Mach. Intell. 15(6), 556–568 (1993)

    Article  Google Scholar 

  21. Laurentini, A.: The visual hull concept for silhouette-based image understanding. IEEE Trans. Pattern Anal. Mach. Intell. 16(2), 150–162 (1994)

    Article  Google Scholar 

  22. Levin, A., Lischinski, D., Weiss, Y.: A closed-form solution to natural image matting. IEEE Trans. Pattern Anal. Mach. Intell. 30(2), 228–242 (2007)

    Article  Google Scholar 

  23. Matsuyama, T., Wu, X., Takai, T., Wada, T.: Real-time dynamic 3-d object shape reconstruction and high-fidelity texture mapping for 3-d video. IEEE Trans. Circuits Syst. Video Technol. 14(3), 357–369 (2004)

    Article  Google Scholar 

  24. Matusik, W., Buehler, C., McMillan, L.: Polyhedral visual hulls for real-time rendering. In: Proc. of 12th Eurographics Workshop on Rendering, pp. 116–126 (2001)

    Google Scholar 

  25. Matusik, W., Buehler, C., Raskar, R., Gortler, S.J., McMillan, L.: Image-based visual hulls. In: Akeley, K. (ed.) Siggraph 2000, Computer Graphics Proceedings, pp. 369–374. ACM Press, New York (2000)

    Google Scholar 

  26. Okutomi, M., Kanade, T.: A multiple-baseline stereo. IEEE Trans. Pattern Anal. Mach. Intell. 15(4), 353–363 (1993)

    Article  Google Scholar 

  27. Potmesil, M.: Generating octree models of 3D objects from their silhouettes in a sequence of images. Comput. Vis. Graph. Image Process. 40, 1–29 (1987)

    Article  Google Scholar 

  28. Rander, P., Narayanan, P.J., Kanade, T.: Virtualized reality: Constructing time-varying virtual worlds from real world events. In: IEEE Visualization, pp. 277–284 (1997)

    Google Scholar 

  29. Roble, D., Zafar, N.B.: Don’t trust your eyes: Cutting-edge visual effects. vol. 42, pp. 35–41 (2009)

    Google Scholar 

  30. Rother, C., Kolmogorov, V., Blake, A.: “GrabCut”: Interactive foreground extraction using iterated graph cuts. ACM Trans. Graph. 23(3), 309–314 (2004)

    Article  Google Scholar 

  31. Scharstein, D., Szeliski, R.: A taxonomy and evaluation of dense two-frame stereo correspondence algorithms. Int. J. Comput. Vis. 47(1), 7–42 (2002)

    Article  MATH  Google Scholar 

  32. Seitz, S.M., Curless, B., Diebel, J., Scharstein, D., Szeliski, R.: A comparison and evaluation of multi-view stereo reconstruction algorithms. In: Computer Vision and Pattern Recognition, 2006 IEEE Comput. Soc. Conference on, vol. 1, pp. 519–528 (2006)

    Google Scholar 

  33. Seitz, S.M., Dyer, C.R.: View morphing. In: Proceedings of the 23rd Annual Conference on Computer Graphics and Interactive Techniques, pp. 21–30. ACM, New York (1996)

    Google Scholar 

  34. Shum, H.-Y., Kang, S.B., Chan, S.-C.: Survey of image-based representations and compression techniques. IEEE Trans. Circuits Syst. Video Technol. 13(11), 1020–1037 (2003)

    Article  Google Scholar 

  35. Smith, A.R., Blinn, J.F.: Blue screen matting. In: SIGGRAPH ’96: Proceedings of the 23rd Annual Conference on Computer Graphics and Interactive Techniques, pp. 259–268. ACM, New York (1996)

    Chapter  Google Scholar 

  36. Starck, J., Hilton, A.: Model-based multiple view reconstruction of people. In: Proc. of ICCV, pp. 915–922 (2003)

    Google Scholar 

  37. Starck, J., Hilton, A.: Correspondence labelling for wide-timeframe free-form surface matching. In: Computer Vision, 2007. ICCV 2007. IEEE 11th International Conference on, pp. 1–8. IEEE Comput. Soc., Los Alamitos (2007)

    Chapter  Google Scholar 

  38. Szeliski, R.: Rapid octree construction from image sequences. CVGIP, Image Underst. 58(1), 23–32 (1993)

    Article  Google Scholar 

  39. Franco, J.S., Boyer, E.: Exact polyhedral visual hulls. In: British Machine Vision Conference, pp. 329–338 (2003)

    Google Scholar 

  40. Thomas, G.A.: Real-time camera pose estimation for augmenting sports scenes. In: Proc. of 3rd European Conf. on Visual Media Production (CVMP2006), London, UK, pp. 10–19 (November 2006)

    Google Scholar 

  41. Thomas, G.A., Lau, H.Y.K.: Generation of high quality slow-motion replay using motion compensation. In: Proc. of International Broadcasting Convention (1990)

    Google Scholar 

  42. Tsai, R.: A versatile camera calibration technique for high-accuracy 3D machine vision metrology using off-the-shelf TV cameras and lenses. IEEE J. Robot. Autom. 3(4), 323–344 (1987)

    Article  Google Scholar 

  43. Vedula, S., Baker, S., Rander, P., Collins, R., Kanade, T.: Three-dimensional scene flow. IEEE Trans. Pattern Anal. Mach. Intell. 27(3), 475–480 (2005)

    Article  Google Scholar 

  44. Vlasic, D., Baran, I., Matusik, W., Popović, J.: Articulated mesh animation from multi-view silhouettes. In: ACM SIGGRAPH 2008 papers, pp. 1–9. ACM, New York (2008)

    Chapter  Google Scholar 

  45. Vogiatzis, G., Esteban, C.H., Torr, P.H.S., Cipolla, R.: Multiview stereo via volumetric graph-cuts and occlusion robust photo-consistency. IEEE Trans. Pattern Anal. Mach. Intell. 29(12), 2241–2246 (2007)

    Article  Google Scholar 

  46. Weik, S., Wingbermühle, J., Niem, W.: Automatic creation of flexible antropomorphic models for 3D videoconferencing. In: Computer Graphics International, 1998. Proceedings, pp. 520–527. IEEE Comput. Soc., Los Alamitos (1998)

    Google Scholar 

  47. Würmlin, S., Lamboray, E., Staadt, O.G., Gross, M.H.: 3D video recorder: A system for recording and playing free-viewpoint video. Comput. Graph. Forum 22, 181–193 (2003). Wiley Online Library

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. BBC Research & Development, 56 Wood Lane, London, UK

    Oliver Grau

Authors
  1. Oliver Grau
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Oliver Grau .

Editor information

Editors and Affiliations

  1. Department of Media Technology, Aalborg University, Niels Jernes Vej 14, Aalborg, 9220, Denmark

    Thomas B. Moeslund

  2. Centre for Vision, Speech & Signal Proc., University of Surrey, Guildford, GU2 7XH, Surrey, United Kingdom

    Adrian Hilton

  3. Copenhagen Institute of Technology, Aalborg University, Lautrupvang 2B, Ballerup, 2750, Denmark

    Volker Krüger

  4. Disney Research, Forbes Avenue 615, Pittsburgh, 15213, Pennsylvania, USA

    Leonid Sigal

Rights and permissions

Reprints and permissions

Copyright information

© 2011 Springer-Verlag London Limited

About this chapter

Cite this chapter

Grau, O. (2011). Multi-view 4D Reconstruction of Human Action for Entertainment Applications. In: Moeslund, T., Hilton, A., Krüger, V., Sigal, L. (eds) Visual Analysis of Humans. Springer, London. https://doi.org/10.1007/978-0-85729-997-0_29

Download citation

  • DOI: https://doi.org/10.1007/978-0-85729-997-0_29

  • Publisher Name: Springer, London

  • Print ISBN: 978-0-85729-996-3

  • Online ISBN: 978-0-85729-997-0

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation