Skip to main content
Springer Nature Link
Log in

Low-bandwidth 3D visual telepresence system

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

We present a methodology to develop a low-cost, low-bandwidth visual telepresence system using commodity depth sensors. To obtain a precise representation of the participants, we fuse together multiple views extracted using a deep background subtraction method. We build a proof-of-concept display composed of a video projector and a quadrangular pyramid made of acrylic, to demonstrate the visualization of a remote person without the need for head-mounted displays. Our system represents an attempt to democratize high-fidelity 3D telepresence using off-the-shelf components.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8

Similar content being viewed by others

Notes

  1. http://github.com/jrterven/backsub

References

  1. Antonio S, Herrera R, Enriquez E (2013) Projection’s panel of models for touch screen. Int J Innovative Res Comput Commun Eng 1(9):2057–2064

    Google Scholar 

  2. Babaee M, Dinh DT, Rigoll G (2018) A deep convolutional neural network for video sequence background subtraction. Pattern Recogn 76:635–649

    Article  Google Scholar 

  3. Badrinarayanan V, Kendall A, Cipolla R (2015) Segnet: a deep convolutional encoder-decoder architecture for image segmentation. arXiv:1511.00561

  4. Beck S, Kunert A, Kulik A, Froehlich B (2013) Immersive group-to-group telepresence. IEEE Trans Vis Comput Graph 19(4):616–625

    Article  Google Scholar 

  5. Bimber O, Raskar R (2005) Spatial augmented reality: merging real and virtual worlds. AK Peters/ CRC Press

  6. Blanche PA, Bablumian A, Voorakaranam R, Christenson C, Lin W, Gu T, Flores D, Wang P, Hsieh WY, Kathaperumal M et al (2010) Holographic three-dimensional telepresence using large-area photorefractive polymer. Nature 468(7320):80

    Article  Google Scholar 

  7. Brown DC (1971) Lens distortion for close-range photogrammetry. Photogramm Eng 37(8):855–866

    Google Scholar 

  8. Córdova-Esparza DM, Terven JR, Jiménez-Hernández H, Herrera-Navarro AM (2017) A multiple camera calibration and point cloud fusion tool for Kinect V2. Sci Comput Program 143:1–8

    Article  Google Scholar 

  9. Dalvi AA, Siddavatam I, Dandekar NG, Patil AV (2015) 3D holographic projections using prism and hand gesture recognition. In: Proceedings of the 2015 international conference on advanced research in computer science engineering & technology (ICARCSET 2015). ACM, p 18

  10. Dreshaj E (2015) Holosuite: an exploration into interactive holographic telepresence (Doctoral dissertation Massachusetts Institute of Technology)

  11. Duane CB (1971) Close-range camera calibration. Photogramm Eng 37(8):855–866

    Google Scholar 

  12. Eigen D, Fergus R (2015) Predicting depth, surface normals and semantic labels with a common multi-scale convolutional architecture. In: Proceedings of the IEEE international conference on computer vision, pp 2650–2658

  13. Gohane MST, Longadge MRN (2014) 3D holograph projection-future of visual communication

  14. Goyette N, Jodoin PM, Porikli F, Konrad J, Ishwar P (2012) Changedetection. net: a new change detection benchmark dataset. In: 2012 IEEE computer society conference on computer vision and pattern recognition workshops. IEEE, pp 1–8

  15. Hofmann M, Tiefenbacher P, Rigoll G (2012) Background segmentation with feedback: the pixel-based adaptive segmenter. In: 2012 IEEE computer society conference on computer vision and pattern recognition workshops (CVPRW). IEEE, pp 38–43

  16. Lee H, Ha G, Lee S, Cha J, Kim S (2016) A hologram based tele-existence platform for emotional exchange among a group of users in both real and virtual environments. In: Proceedings of the 22nd ACM conference on virtual reality software and technology. ACM, pp 295–296

  17. Lee H, Ha G, Lee S, Kim S (2017) A mixed reality tele-presence platform to exchange emotion and sensory information based on MPEG-V standard. In: 2017 IEEE Virtual Reality (VR). IEEE, pp 349–350

  18. Levenberg K (1944) A method for the solution of certain non-linear problems in least squares. Q Appl Math 2(2):164–168

    Article  MathSciNet  MATH  Google Scholar 

  19. Lu X, Shen J, Perugini S, Yang J (2015) An immersive telepresence system using rgb-d sensors and head mounted display. In: 2015 IEEE international symposium on multimedia (ISM). IEEE, pp 453–458

  20. Maimone A, Fuchs H (2011) Encumbrance-free telepresence system with real-time 3D capture and display using commodity depth cameras. In: 2011 10th IEEE international symposium on mixed and augmented reality (ISMAR). IEEE, pp 137–146

  21. Maimone A, Fuchs H (2012) Real-time volumetric 3D capture of room-sized scenes for telepresence. In: 3DTV-Conference: the true vision-capture, transmission and display of 3D video (3DTV-CON). IEEE, pp 1–4

  22. Maimone A, Yang X, Dierk N, State A, Dou M, Fuchs H (2013) General-purpose telepresence with head-worn optical see-through displays and projector-based lighting. In: 2013 IEEE virtual reality (VR). IEEE, pp 23–26

  23. Marquardt DW (1963) An algorithm for least-squares estimation of nonlinear parameters. J Soc Ind Appl Math 11(2):431–441

    Article  MathSciNet  MATH  Google Scholar 

  24. Mekuria R, Cesar P (2016) MP3DG-PCC, open source software framework for implementation and evaluation of point cloud compression. In: Proceedings of the 24th ACM international conference on multimedia. ACM, pp 1222–1226

  25. Noor AK, Aras R (2015) Potential of multimodal and multiuser interaction with virtual holography. Adv Eng Softw 81:1–6

    Article  Google Scholar 

  26. Orts-Escolano S, Rhemann C, Fanello S, Chang W, Kowdle A, Degtyarev Y, Kim D, Davidson PL, Khamis S, Dou M et al (2016) Holoportation: virtual 3d teleportation in real-time. In: Proceedings of the 29th annual symposium on user interface software and technology. ACM, pp 741–754

  27. Pejsa T, Kantor J, Benko H, Ofek E, Wilson A (2016) Room2room: enabling life-size telepresence in a projected augmented reality environment. In: Proceedings of the 19th ACM conference on computer-supported cooperative work & social computing. ACM, pp 1716–1725

  28. Prince SJ (2012) Computer vision: models, learning, and inference. Cambridge University Press, United Kingdom

    Book  MATH  Google Scholar 

  29. Sajid H, Cheung SCS (2015) Background subtraction for static & moving camera. In: 2015 IEEE international conference on image processing (ICIP). IEEE, pp 4530–4534

  30. Sakkos D, Liu H, Han J, Shao L (2017) End-to-end video background subtraction with 3d convolutional neural networks. Multimed Tools Appl, pp 1–19

  31. Sedky M, Chibelushi CC, MONIRI M (2010) Image processing: object segmentation using full-spectrum matching of albedo derived from colour images

  32. Schwarz S, Preda M, Baroncini V, Budagavi M, Cesar P, Chou PA, Llach J (2018) Emerging MPEG standards for point cloud compression. IEEE J Emerging Sel Top Circuits Syst 9(1):133–148

    Article  Google Scholar 

  33. St-Charles PL, Bilodeau GA, Bergevin R (2015) A self-adjusting approach to change detection based on background word consensus. In: 2015 IEEE winter conference on applications of computer vision (WACV). IEEE, pp 990–997

  34. St-Charles PL, Bilodeau GA, Bergevin R (2015) Subsense: a universal change detection method with local adaptive sensitivity. IEEE Trans Image Process 24(1):359–373

    Article  MathSciNet  MATH  Google Scholar 

  35. Statista: global business travel spending in 2015, 2016 and 2020 (in trillion U.S. dollars) (2017). https://www.statista.com/statistics/612244/global-business-travel-spending/ [Online; accessed 1-April-2018]

  36. Stauffer C, Grimson WEL (1999) Adaptive background mixture models for real-time tracking. In: CVPR. IEEE, p 2246

  37. Terven JR, Córdova-Esparza DM (2016) Kin2. A Kinect 2 toolbox for MATLAB. Sci Comput Program 130:97–106

    Article  Google Scholar 

  38. Tiro D, Poturiović A, Buzadjija N (2015) The possibility of the hologram pyramid applying in the rapid prototyping. In: 2015 4th mediterranean conference on embedded computing (MECO). IEEE, pp 25–30

  39. Towles H, Chen WC, Yang R, Kum SU, Kelshikar HFN, Mulligan J, Daniilidis K, Fuchs H, Hill CC, Mulligan NKJ et al (2002) 3D tele-collaboration over internet2. International Workshop on Immersive Telepresence, Juan Les Pins

  40. Varadarajan S, Miller P, Zhou H (2015) Region-based mixture of gaussians modelling for foreground detection in dynamic scenes. Pattern Recogn 48(11):3488–3503

    Article  MATH  Google Scholar 

  41. Wikimedia Commons. File:ABC Clarke predicts internet and PC.ogv — Wikimedia Commons{,} the free media repository (2018). https://commons.wikimedia.org/w/index.php?title=File:ABC_Clarke_predicts_internet_and_PC.ogv&oldid=292577968 [Online; accessed 1-April-2018]

  42. Yoo H, Kim H (2014) On study of the volumetric display techniques. In: Interactive media arts proceedings

Download references

Acknowledgements

This work was supported by CONACYT through postdoctoral support number 291113. We also want to thank CIDESI for providing the facilities and assistance during the development of this project.

Author information

Authors and Affiliations

  1. UAQ, Universidad Autónoma de Querétaro, Facultad de Informática, Av. de las Ciencias S/N, Campus Juriquilla , Querétaro, C.P. 76230, México

    Diana-Margarita Córdova-Esparza & Ana Herrera-Navarro

  2. AiFi Inc., 2364 Walsh Av., Santa Clara, CA95051, USA

    Juan R. Terven

  3. CIDESI, Centro de Ingeniería y Desarrollo Industrial, Av. Playa Pie de la Cuesta 702, Desarrollo San Pablo, Querétaro, C.P. 76130, México

    Hugo Jiménez-Hernández, Alberto Vázquez-Cervantes & Juan-M. García-Huerta

Authors
  1. Diana-Margarita Córdova-Esparza
    View author publications

    You can also search for this author inPubMed Google Scholar

  2. Juan R. Terven
    View author publications

    You can also search for this author inPubMed Google Scholar

  3. Hugo Jiménez-Hernández
    View author publications

    You can also search for this author inPubMed Google Scholar

  4. Ana Herrera-Navarro
    View author publications

    You can also search for this author inPubMed Google Scholar

  5. Alberto Vázquez-Cervantes
    View author publications

    You can also search for this author inPubMed Google Scholar

  6. Juan-M. García-Huerta
    View author publications

    You can also search for this author inPubMed Google Scholar

Corresponding author

Correspondence to Diana-Margarita Córdova-Esparza.

Ethics declarations

Conflict of interests

The authors declare that there is no conflict of interest.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Córdova-Esparza, DM., Terven, J.R., Jiménez-Hernández, H. et al. Low-bandwidth 3D visual telepresence system. Multimed Tools Appl 78, 21273–21290 (2019). https://doi.org/10.1007/s11042-019-7464-0

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-019-7464-0

Keywords