Abstract
A virtual TV set is a studio that is able to combine recorded actors and objects with computer generated virtual environments in real time. In order to achieve this combination seamlessly, in an ideal configuration, several elements such as cameras, objects and people should be tracked so that all their actions on the stage have a corresponding effect in the virtual world. However, in the actual professional virtual TV sets, the tracking possibilities are quite limited because of the hardware and software architecture used, which has not had a major evolution since the first prototypes presented in the nineties. This traditional architecture uses to be rigid, including just one monolithic tracking system and low levels of interactivity. In this paper, a new distributed, flexible and scalable hardware and software architecture that allows the inclusion of multiple kinds of devices in parallel is introduced. It breaks with the traditional structure of the virtual TV sets, opening the technology to an easier inclusion of new devices without the need of updating the proprietary software of the set, thus facilitating its future evolution. The design, implementation and test of this architecture, through the adaptation of a traditional virtual TV set, is presented. The tests are developed through the inclusion of modern devices (in our case Optitrack infrared cameras, Microsoft Kinect V2 and Leap Motion) that, through a synergistic operation, allow the system to solve some traditional drawbacks of this technology such as free and multiple object and camera tracking, presenter natural interaction and automatic distance keying.
Similar content being viewed by others
References
Blonde L, Buck M, Galli R, Niem W, Paker Y, Schmidt W, Thomas G (Summer 1996) A virtual studio for live broadcasting: The Mona Lisa project. IEEE Multimed 3(2):18–29
Brainstorm | Real-time 3D graphics and virtual set solutions (2016) [Online]. Available: http://www.brainstorm.es/. [Accessed: 10- May- 2016]
Eckerson W (1995) Three Tier Client/Server Architecture: Achieving Scalability, Performance, and Efficiency in Client Server Applications. Open Inform Syst 10(1)
Fellous A (1993) "Stv-synthetic tv: from laboratory prototype to production tools." in Virtual worlds and multimedia, John Wiley & Sons, Inc., Oct. , pp. 127–133
Freund Y, Schapire RE (1999) A Short Introduction to Boosting. J Jap Soc Artif Intell 14(5):771–780
Gang Z, Tingfu M, Chen D, Lily Q, Gao M. (2008) "Research on low-cost virtual studio system for education resource authoring". IT in Medicine and Education, 2008. ITME 2008. IEEE International Symposium on, Xiamen pp 198-201
Gibbs S, Arapis C, Breiteneder C, Lalioti V, Mostafawy S, Speier J (1998) Virtual studios: an overview. IEEE MultiMed 5(1):18–35
A. N. Goldsmlith (1937) Color micro-facsimile system", U S Patent No 2,073,370
H. Gonzalez-Jorge, B. Riveiro, E. Vazquez-Fernandez, J. Martínez-Sánchez, P. Arias, "Metrological evaluation of microsoft kinect and asus xtion sensors". Measurement, vol. 46, no. 6, pp. 1800–1806, Nov. 2013
Guna J, Jakus G, Pogacnik M, Tomazic S, Sodnik J (Feb. 2014) An Analysis of the Precision and Reliability of the Leap Motion Sensor and Its Suitability for Static and Dynamic Tracking. Sensors 14:3702–3720
Hayashi M (Jan-Mar 1998) Image compositing based on virtual cameras. IEEE MultiMedia 5(1):36–48
"HD/SD Chroma Keyer | MBP-100CK | Products | FOR-A" (2016) [Online]. Available: http://www.for-a.com/products/mbp100ck/mbp100ck.html. [Accessed: 20- Apr.- 2016]
Kennedy RC, Gaskins FJ (1958) Electronic Composites in Modern Television. Proc IRE 46(11):1798–1807
Khoshelham K, Elberink SO (2012) Accuracy and resolution of kinect depth data for indoor mapping applications. Sensors 12:1437–1454
Lalioti V, Woolard A (2003) Mixed Reality productions of the future. IBC 2003 Conference, International Broadcasting Convention
Lange R, Seitz P (2001) Solid-state time-of-flight range camera. IEEE J Quantum Electron 37(3):390–397
Leap Motion | Mac & PC Motion Controller for Games, Design, Virtual Reality & More (2016) [Online]. Available: https://www.leapmotion.com/. [Accessed: 20- Apr.- 2016]
Méndez R, Flores J, Castelló E, Arenas R. 2016. Preliminary evaluation of the Kinect V2 sensor for its use in virtual TV sets with natural interaction. In Proceedings of the XVII International Conference on Human Computer Interaction (Interacción '16), no 14, pp. 1–2
Motion Capture Systems - Optitrack (2016) [Online]. Available: http://www.optitrack.com/. [Accessed: 20- Apr.- 2016]
Multimedia | XtionPRO LIVE | ASUS (2016) [Online]. Available: https://www.asus.com/3D-Sensor/Xtion_PRO_LIVE/. [Accessed: 20- Apr.- 2016]
Myo Gesture Control Armband - Wearable Technology by Thalmic Labs (2016) [Online]. Available: https://www.myo.com/. [Accessed 20- Apr.- 2016]
Ring Zero (2015) [Online]. Available: http://ringzero.logbar.jp/. [Accessed: 20- Apr.- 2016]
Shimoda S, Hayashi M, Kanatsugu Y (1989) New chromakey imaging technique with Hi-vision background. IEEE Trans Broadcast 35(4):357–361
Suma E, Lange B, Rizzo AS, Krum DM, Bolas M (2011) Faast: The flexible action and articulated skeleton toolkit. Virtual Reality Conference (VR), 247–248
Taylor RM II, Hudson TC, Seeger A, Weber H, Juliano J, Helser AT (2001) VRPN: a device-independent, network-transparent VR peripheral system. Proc ACM Symp Virtual Real Software Technol, 55–61
Thomas G (2006) Mixed reality techniques for TV and their application for on-set and pre-visualization in film production. International Workshop on Mixed Reality Technology for Filmmaking, 31–36
Thomas GA, Jin J, Niblett T, Urquhart C (1997) A versatile camera position measurement system for virtual reality TV production. Broadcasting Convention, 1997. International, Amsterdam, pp 284–289
Weichert F, Bachmann D, Rudak B, Fisseler D (2013) Analysis of the accuracy and robustness of the leap motion controller. Sensors 13(5):6380–6393
Wojdala A (1998) Challenges of virtual set technology. IEEE MultiMedia 5(1):50–57
Q. Xiao, G. Zhao, R. Li. (2008) "Three–Dimensional Low Cost Virtual Studio System," Knowledge Acquisition and Modeling, 2008. KAM '08. International Symposium on, Wuhan. pp. 741–744
Xirouhakis Y, Drosopoulos A, Delopoulos A (2001) Efficient optical camera tracking in virtual sets. IEEE Trans Image Process 10(4):609–622
Yang L, Zhang L, Dong H, Alelaiwi A, El Saddik A (2015) Evaluating and Improving the Depth Accuracy of Kinect for Windows v2. IEEE Sensors J 15(8):4275–4285
Zhang Z (2012) Microsoft kinect sensor and its effect. IEEE MultiMedia 19(2):4–10
Acknowledgements
Brainstorm Multimedia for their continuous support during the development of this project. Manuel Fidalgo for his help and advice. This work has received financial support from the Consellería de Cultura, Educación e Ordenación Universitaria (accreditation 2016-2019, ED431G/08) and the European Regional Development Fund (ERDF).
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Méndez, R., Flores, J., Castelló, E. et al. New distributed virtual TV set architecture for a synergistic operation of sensors and improved interaction between real and virtual worlds. Multimed Tools Appl 77, 18999–19025 (2018). https://doi.org/10.1007/s11042-017-5353-y
Received:
Revised:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11042-017-5353-y