Skip to main content
Log in

New distributed virtual TV set architecture for a synergistic operation of sensors and improved interaction between real and virtual worlds

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

A virtual TV set is a studio that is able to combine recorded actors and objects with computer generated virtual environments in real time. In order to achieve this combination seamlessly, in an ideal configuration, several elements such as cameras, objects and people should be tracked so that all their actions on the stage have a corresponding effect in the virtual world. However, in the actual professional virtual TV sets, the tracking possibilities are quite limited because of the hardware and software architecture used, which has not had a major evolution since the first prototypes presented in the nineties. This traditional architecture uses to be rigid, including just one monolithic tracking system and low levels of interactivity. In this paper, a new distributed, flexible and scalable hardware and software architecture that allows the inclusion of multiple kinds of devices in parallel is introduced. It breaks with the traditional structure of the virtual TV sets, opening the technology to an easier inclusion of new devices without the need of updating the proprietary software of the set, thus facilitating its future evolution. The design, implementation and test of this architecture, through the adaptation of a traditional virtual TV set, is presented. The tests are developed through the inclusion of modern devices (in our case Optitrack infrared cameras, Microsoft Kinect V2 and Leap Motion) that, through a synergistic operation, allow the system to solve some traditional drawbacks of this technology such as free and multiple object and camera tracking, presenter natural interaction and automatic distance keying.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11

Similar content being viewed by others

References

  1. Blonde L, Buck M, Galli R, Niem W, Paker Y, Schmidt W, Thomas G (Summer 1996) A virtual studio for live broadcasting: The Mona Lisa project. IEEE Multimed 3(2):18–29

    Article  Google Scholar 

  2. Brainstorm | Real-time 3D graphics and virtual set solutions (2016) [Online]. Available: http://www.brainstorm.es/. [Accessed: 10- May- 2016]

  3. Eckerson W (1995) Three Tier Client/Server Architecture: Achieving Scalability, Performance, and Efficiency in Client Server Applications. Open Inform Syst 10(1)

  4. Fellous A (1993) "Stv-synthetic tv: from laboratory prototype to production tools." in Virtual worlds and multimedia, John Wiley & Sons, Inc., Oct. , pp. 127–133

  5. Freund Y, Schapire RE (1999) A Short Introduction to Boosting. J Jap Soc Artif Intell 14(5):771–780

    Google Scholar 

  6. Gang Z, Tingfu M, Chen D, Lily Q, Gao M. (2008) "Research on low-cost virtual studio system for education resource authoring". IT in Medicine and Education, 2008. ITME 2008. IEEE International Symposium on, Xiamen pp 198-201

  7. Gibbs S, Arapis C, Breiteneder C, Lalioti V, Mostafawy S, Speier J (1998) Virtual studios: an overview. IEEE MultiMed 5(1):18–35

    Article  Google Scholar 

  8. A. N. Goldsmlith (1937) Color micro-facsimile system", U S Patent No 2,073,370

  9. H. Gonzalez-Jorge, B. Riveiro, E. Vazquez-Fernandez, J. Martínez-Sánchez, P. Arias, "Metrological evaluation of microsoft kinect and asus xtion sensors". Measurement, vol. 46, no. 6, pp. 1800–1806, Nov. 2013

  10. Guna J, Jakus G, Pogacnik M, Tomazic S, Sodnik J (Feb. 2014) An Analysis of the Precision and Reliability of the Leap Motion Sensor and Its Suitability for Static and Dynamic Tracking. Sensors 14:3702–3720

    Article  Google Scholar 

  11. Hayashi M (Jan-Mar 1998) Image compositing based on virtual cameras. IEEE MultiMedia 5(1):36–48

    Article  Google Scholar 

  12. "HD/SD Chroma Keyer | MBP-100CK | Products | FOR-A" (2016) [Online]. Available: http://www.for-a.com/products/mbp100ck/mbp100ck.html. [Accessed: 20- Apr.- 2016]

  13. Kennedy RC, Gaskins FJ (1958) Electronic Composites in Modern Television. Proc IRE 46(11):1798–1807

    Article  Google Scholar 

  14. Khoshelham K, Elberink SO (2012) Accuracy and resolution of kinect depth data for indoor mapping applications. Sensors 12:1437–1454

    Article  Google Scholar 

  15. Lalioti V, Woolard A (2003) Mixed Reality productions of the future. IBC 2003 Conference, International Broadcasting Convention

  16. Lange R, Seitz P (2001) Solid-state time-of-flight range camera. IEEE J Quantum Electron 37(3):390–397

    Article  Google Scholar 

  17. Leap Motion | Mac & PC Motion Controller for Games, Design, Virtual Reality & More (2016) [Online]. Available: https://www.leapmotion.com/. [Accessed: 20- Apr.- 2016]

  18. Méndez R, Flores J, Castelló E, Arenas R. 2016. Preliminary evaluation of the Kinect V2 sensor for its use in virtual TV sets with natural interaction. In Proceedings of the XVII International Conference on Human Computer Interaction (Interacción '16), no 14, pp. 1–2

  19. Motion Capture Systems - Optitrack (2016) [Online]. Available: http://www.optitrack.com/. [Accessed: 20- Apr.- 2016]

  20. Multimedia | XtionPRO LIVE | ASUS (2016) [Online]. Available: https://www.asus.com/3D-Sensor/Xtion_PRO_LIVE/. [Accessed: 20- Apr.- 2016]

  21. Myo Gesture Control Armband - Wearable Technology by Thalmic Labs (2016) [Online]. Available: https://www.myo.com/. [Accessed 20- Apr.- 2016]

  22. Ring Zero (2015) [Online]. Available: http://ringzero.logbar.jp/. [Accessed: 20- Apr.- 2016]

  23. Shimoda S, Hayashi M, Kanatsugu Y (1989) New chromakey imaging technique with Hi-vision background. IEEE Trans Broadcast 35(4):357–361

    Article  Google Scholar 

  24. Suma E, Lange B, Rizzo AS, Krum DM, Bolas M (2011) Faast: The flexible action and articulated skeleton toolkit. Virtual Reality Conference (VR), 247–248

  25. Taylor RM II, Hudson TC, Seeger A, Weber H, Juliano J, Helser AT (2001) VRPN: a device-independent, network-transparent VR peripheral system. Proc ACM Symp Virtual Real Software Technol, 55–61

  26. Thomas G (2006) Mixed reality techniques for TV and their application for on-set and pre-visualization in film production. International Workshop on Mixed Reality Technology for Filmmaking, 31–36

  27. Thomas GA, Jin J, Niblett T, Urquhart C (1997) A versatile camera position measurement system for virtual reality TV production. Broadcasting Convention, 1997. International, Amsterdam, pp 284–289

    Google Scholar 

  28. Weichert F, Bachmann D, Rudak B, Fisseler D (2013) Analysis of the accuracy and robustness of the leap motion controller. Sensors 13(5):6380–6393

    Article  Google Scholar 

  29. Wojdala A (1998) Challenges of virtual set technology. IEEE MultiMedia 5(1):50–57

    Article  Google Scholar 

  30. Q. Xiao, G. Zhao, R. Li. (2008) "Three–Dimensional Low Cost Virtual Studio System," Knowledge Acquisition and Modeling, 2008. KAM '08. International Symposium on, Wuhan. pp. 741–744

  31. Xirouhakis Y, Drosopoulos A, Delopoulos A (2001) Efficient optical camera tracking in virtual sets. IEEE Trans Image Process 10(4):609–622

    Article  MATH  Google Scholar 

  32. Yang L, Zhang L, Dong H, Alelaiwi A, El Saddik A (2015) Evaluating and Improving the Depth Accuracy of Kinect for Windows v2. IEEE Sensors J 15(8):4275–4285

    Article  Google Scholar 

  33. Zhang Z (2012) Microsoft kinect sensor and its effect. IEEE MultiMedia 19(2):4–10

    Article  Google Scholar 

Download references

Acknowledgements

Brainstorm Multimedia for their continuous support during the development of this project. Manuel Fidalgo for his help and advice. This work has received financial support from the Consellería de Cultura, Educación e Ordenación Universitaria (accreditation 2016-2019, ED431G/08) and the European Regional Development Fund (ERDF).

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Roi Méndez.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Méndez, R., Flores, J., Castelló, E. et al. New distributed virtual TV set architecture for a synergistic operation of sensors and improved interaction between real and virtual worlds. Multimed Tools Appl 77, 18999–19025 (2018). https://doi.org/10.1007/s11042-017-5353-y

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-017-5353-y

Keywords

Navigation