Skip to main content

Advertisement

SpringerLink
Log in

Multi-camera tele-immersion system with real-time model driven data compression

A new model-based compression method for massive dynamic point data

  • Original Article
  • Published:
The Visual Computer Aims and scope Submit manuscript

Abstract

Vision-based full-body 3D reconstruction for tele-immersive applications generates large amount of data points, which have to be sent through the network in real time. In this paper, we introduce a skeleton-based compression method using motion estimation where kinematic parameters of the human body are extracted from the point cloud data in each frame. First we address the issues regarding the data capturing and transfer to a remote site for the tele-immersive collaboration. We compare the results of the existing compression methods and the proposed skeleton-based compression technique. We examine the robustness and efficiency of the algorithm through experimental results with our multi-camera tele-immersion system. The proposed skeleton-based method provides high and flexible compression ratios from 50:1 to 5000:1 with reasonable reconstruction quality (peak signal-to-noise ratio from 28 to 31 dB) while preserving real-time (10+ fps) processing.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. Adobe. Quicktime 7.0 h.264 implementation (2006)

  2. Aggarwal, J.K., Cai, Q.: Human motion analysis: a review. Comput. Vis. Image Underst. 73(3), 428–440 (1999)

    Article  Google Scholar 

  3. Arikan, O.: Compression of motion capture databases. ACM Trans. Graph. 25(3), 890–897 (2006)

    Article  MathSciNet  Google Scholar 

  4. Baker, H., Tanguay, D., Sobel, I., Gelb, D., Gross, M., Culbertson, W., Malzenbender, T.: The coliseum immersive teleconferencing system. In: Proceedings of International Workshop on Immersive Telepresence, Juan-les-Pins, France (2002)

  5. Bentley, J.L.: Multidimensional binary search trees used for associative searching. Commun. ACM 18(9), 509–517 (1975)

    Article  MATH  MathSciNet  Google Scholar 

  6. Besl, P.J., McKay, N.D.: A method for registration of 3-D shapes. IEEE Trans. Pattern Anal. Mach. Intell. 14(2), 239–256 (1992)

    Article  Google Scholar 

  7. Cheung, K.M., Baker, S., Kanade, T.: Shape-from-silhouette of articulated objects and its use for human body kinematics estimation and motion capture. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, June 2003

  8. CMU. Graphics Lab Motion Capture Database. Carnegie Mellon University. http://mocap.cs.cmu.edu/

  9. Demirdjian, D., Darrell, T.: 3-D articulated pose tracking for untethered deictic reference. In: ICMI’02: Proceedings of the 4th IEEE International Conference on Multimodal Interfaces, Washington, DC, p. 267. IEEE Comput. Soc., Los Alamitos (2002)

    Chapter  Google Scholar 

  10. Dewaele, G., Devernay, F., Horaud, R.: Hand motion from 3D point trajectories and a smooth surface model. In: ECCV (1), pp. 495–507 (2004)

  11. Gavrila, D.M.: The visual analysis of human movement: a survey. Comput. Vis. Image Underst. 73(1), 82–98 (1999)

    Article  MATH  Google Scholar 

  12. Gross, M., Würmlin, S., Naef, M., Lamboray, E., Spagno, C., Kunz, A., Koller-Meier, E., Svoboda, T., Gool, L.V., Lang, S., Strehlke, K., Moere, A.V., Staadt, O.: Blue-c: a spatially immersive display and 3D video portal for telepresence. ACM Trans. Graph. 22(3), 819–827 (2003)

    Article  Google Scholar 

  13. Gumhold, S., Karni, Z., Isenburg, M., Seidel, H.-P.: Predictive point-cloud compression. In: Siggraph 2005 Sketches (2005)

  14. Hasenfratz, J., Lapierre, M., Sillion, F.: A real-time system for full-body interaction with virtual worlds. In: Proceedings of Eurographics Symposium on Virtual Environments, pp. 147–156. Eurographics Association, Aire-la-Ville (2004)

    Google Scholar 

  15. Holden, M.K.: Virtual environments for motor rehabilitation: review. Cyberpsychol. Behav. 8(3), 187–211 (2005)

    Article  MathSciNet  Google Scholar 

  16. Ibarria, L., Rossignac, J.: Dynapack: space-time compression of the 3D animations of triangle meshes with fixed connectivity. In: SCA’03: Proceedings of the 2003 ACM SIGGRAPH/Eurographics Symposium on Computer Animation, Aire-la-Ville, Switzerland, pp. 126–135. Eurographics Association, Aire-la-Ville (2003)

    Google Scholar 

  17. Isard, M., Blake, A.: Condensation—conditional density propagation for visual tracking. Int. J. Comput. Vis. 29(1), 5–28 (1998)

    Article  Google Scholar 

  18. Jung, S., Bajcsy, R.: A framework for constructing real-time immersive environments for training physical activities. J. Multimed. 1(7), 9–17 (2006)

    Google Scholar 

  19. Kalra, P., Magnenat-Thalman, N., Moccozet, L., Sannier, G., Aubel, A., Thalman, D.: Real-time animation of realistic virtual humans. IEEE Comput. Graph. Appl. 18(25), 42–56 (1998)

    Article  Google Scholar 

  20. Karni, Z., Gotsman, C.: Compression of soft-body animation sequences. Comput. Graph. 28(1), 25–34 (2004)

    Article  Google Scholar 

  21. Keshner, E.A.: Virtual reality and physical rehabilitation: a new toy or a new research and rehabilitation tool? J. Neuroengineering Rehabil. 1(8) (2004)

  22. Keshner, E., Kenyon, R.: Using immersive technology for postural research and rehabilitation. Assist. Technol. 16, 54–62 (2004)

    Google Scholar 

  23. Knoop, R.D.S., Vacek, S.: Sensor fusion for 3D human body tracking with an articulated 3D body model. In: Proceedings of the IEEE International Conference on Robotics and Automation, Walt Disney Resort, Orlando, Florida, 15 May 2006

  24. Kum, S.-U., Mayer-Patel, K.: Real-time multidepth stream compression. ACM Trans. Multimedia Comput. Commun. Appl. 1(2), 128–150 (2005)

    Article  Google Scholar 

  25. Lamboray, E., Würmlin, S., Gross, M.: Real-time streaming of point-based 3D video. In: VR’04: Proceedings of the IEEE Virtual Reality 2004, Washington, DC, p. 91. IEEE Comput. Soc., Los Alamitos (2004)

    Chapter  Google Scholar 

  26. Lanier, J.: Virtually there. Sci. Am. 4, 52–61 (2001)

    Google Scholar 

  27. Lengyel, J.E.: Compression of time-dependent geometry. In: SI3D’99: Proceedings of the 1999 Symposium on Interactive 3D Graphics, New York, NY, pp. 89–95. ACM, New York (1999)

    Chapter  Google Scholar 

  28. Li, L., Zhang, M., Xu, F., Liu, S.: Ert-vr: an immersive virtual reality system for emergency rescue training. Virtual Real. 8(3), 194–197 (2005)

    Article  Google Scholar 

  29. Lien, J.-M., Bajcsy, G.K.R.: Skeleton-based data compression for multi-camera tele-immersion system. In: Proceedings of the International Symposium on Visual Computing (ISVC), pp. 347–354 (2007)

  30. Lopez, E.J.L.: Finite element surface-based stereo 3D reconstruction, April 2006. Poster presentation given at the Trust NSF Site Visit

  31. Marc Alexa, W.M.: Representing animations by principal components. Comput. Graph. Forum 19(3), 411–418 (2000)

    Article  Google Scholar 

  32. Mason, H., Moutahir, M.: Multidisciplinary experiential education in second life: a global approach. In: Second Life Education Workshop, San Francisco, California, pp. 30–34 (2006)

  33. McComas, J., Pivik, J., Laflamme, M.: Current uses of virtual reality for children with disabilities. Virtual Environments in Clinical Psychology and Neuroscience (1998)

  34. Morency, L.-P., Darrell, T.: Stereo tracking using ICP and normal flow constraint. In: Proceedings of International Conference on Pattern Recognition (2002)

  35. Mulligan, J., Daniilidis, K.: Real-time trinocular stereo for tele-immersion. In: Proceedings of 2001 International Conference on Image Processing, Thessaloniki, Greece, pp. 959–962 (2001)

  36. Ochotta, T., Saupe, D.: Compression of point-based 3D models by shape-adaptive wavelet coding of multi-height fields. In: Symposium on Point-Based Graphics, pp. 103–112 (2004)

  37. Patel, K., Bailenson, J.N., Hack-Jung, S., Diankov, R., Bajcsy, R.: The effects of fully immersive virtual reality on the learning of physical tasks. In: Proceedings of the 9th Annual International Workshop on Presence, Ohio, USA, pp. 87–94 (2006)

  38. Piccardi, M.: Background subtraction techniques: a review. In: Proceedings of IEEE International Conference on Systems, Man and Cybernetics, Hague, Netherlands, pp. 3099–3104 (2004)

  39. Point Grey Research Inc, Vancouver, Canada

  40. Robb, R.: Virtual reality in medicine: A personal perspective. J. Vis. 5(4), 317–326 (2002)

    Google Scholar 

  41. Rusinkiewicz, S., Levoy, M.: Efficient variants of the ICP algorithm. In: Proceedings of the Third International Conference on 3-D Digital Imaging and Modeling (3DIM), pp. 145–152 (2001)

  42. Sattler, M., Sarlette, R., Klein, R.: Simple and efficient compression of animation sequences. In: SCA’05: Proceedings of the 2005 ACM SIGGRAPH/Eurographics Symposium on Computer Animation, New York, NY, pp. 209–217. ACM, New York (2005)

    Chapter  Google Scholar 

  43. Simon, D., Hebert, M., Kanade, T.: Real-time 3-D pose estimation using a high-speed range sensor. In: Proceedings of IEEE International Conference on Robotics and Automation (ICRA’94), vol. 3, pp. 2235–2241 (1994)

  44. Tsai, R.: A versatile camera calibration technique for high-accuracy 3D machine vision metrology using off-the-shelf TV cameras and lenses. IEEE J. Robot. Autom. RA3(4), 323–344 (1987)

    Article  Google Scholar 

  45. William Pennebaker, J.M.: JPEG: Still Image Data Compression Standard. Springer, Berlin (1992)

    Google Scholar 

  46. Würmlin, S., Lamboray, E., Gross, M.: 3D video fragments: dynamic point samples for real-time free-viewpoint video. Comput. Graph. 28, 3–14 (2004)

    Article  Google Scholar 

  47. Yang, Y., Wang, X., Chen, J.X.: Rendering avatars in virtual reality: integrating a 3D model with 2D images. Comput. Sci. Eng. 4(1), 86–91 (2002)

    Article  MathSciNet  Google Scholar 

  48. Yang, Z., Cui, Y., Anwar, Z., Bocchino, R., Kiyanclar, N., Nahrstedt, K., Campbell, R.H., Yurcik, W.: Real-time 3D video compression for tele-immersive environments. In: Proc. of SPIE/ACM Multimedia Computing and Networking (MMCN’06), San Jose, CA (2006)

  49. Zhang, J., Owen, C.B.: Octree-based animated geometry compression. In: DCC’04: Proceedings of the Conference on Data Compression, Washington, DC, p. 508. IEEE Comput. Soc., Los Alamitos (2004)

    Google Scholar 

  50. Zhang, D., Nomura, Y., Fujii, S.: Error analysis and optimization of camera calibration. In: Proceedings of IEEE/RSJ International Workshop on Intelligent Robots and Systems (IROS’91), Osaka, Japan, pp. 292–296 (1991)

  51. Zhao, W., Nandhakumar, N.: Effects of camera alignment errors on stereoscopic depth estimates. Pattern Recogn. 29(12), 2115–2126 (1996)

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. George Mason University, Fairfax, VA, USA

    Jyh-Ming Lien

  2. University of California, Berkeley, CA, USA

    Gregorij Kurillo & Ruzena Bajcsy

Authors
  1. Jyh-Ming Lien
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Gregorij Kurillo
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Ruzena Bajcsy
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Jyh-Ming Lien.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Lien, JM., Kurillo, G. & Bajcsy, R. Multi-camera tele-immersion system with real-time model driven data compression. Vis Comput 26, 3–15 (2010). https://doi.org/10.1007/s00371-009-0367-8

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00371-009-0367-8

Keywords

Search

Navigation