Abstract
Natural user interfaces (NUIs) provide human computer interaction (HCI) with natural and intuitive operation interfaces, such as using human gestures and voice. We have developed a real-time NUI engine architecture using a web camera as a means of implementing NUI applications. The system captures video via the web camera, implements real-time image processing using graphic processing unit (GPU) programming. This paper describes the architecture of the engine and the real-virtual environment interaction methods, such as foreground segmentation and hand gesture recognition. These methods are implemented using GPU programming in order to realize real-time image processing for HCI. To verify the efficacy of our proposed NUI engine, we utilized it in the development and implementation of several mixed reality games and touch-less operation applications, using the developed NUI engine and the DirectX SDK. Our results confirm that the methods implemented by the engine operate in real time and the interactive operations are intuitive.
Similar content being viewed by others
References
Ali S, Shah M (2010) Human action recognition in videos using kinematic features and multiple instance learning. IEEE Trans Pattern Anal Mach Intell 32(2):288–303
Betke M, Gips J, Fleming P (2002) The camera mouse: visual tracking of body features to provide computer access for people with severe disabilities. IEEE Trans Neural Syst Rehabil Eng 10(1):1–10
Biswas KK, Basu SK (2011) Gesture recognition using microsoft kinect. Proceedings of the 5th international conference on automation, robotics and applications, pp 100–103
Dickinson P, Hunter A, Appiah K (2009) A spatially distributed model for foreground segmentation. Image Vis Comput 27:1326–1335
Fan YC, Chen CL, Huang SS (2013) Natural user interface for interactive television. 2013 I.E. 17th international symposium on consumer electronics, pp 189–190
Goswami K, Hong GS, Kim BG (2013) A novel mesh-based moving object detection technique in video sequence. J Convergence 4(3):20–24
Ho YS (2013) Challenging technical issues of 3D video processing. J Convergence 4(1):1–6
Homer BD, Kinzer CK, Plass JL et al (2014) Moved to learn: the effects of interactivity in a Kinect-based literacy game for beginning readers. Comput Educ 74:37–49
Hussain A, Abbasi A, Afzulpurkar N (2012) Detecting & interpreting self-manipulating hand movements for student’s affect prediction. Human-Centric Comput Inf Sci 2(14):1–18
Kenneth S, Christian S, Bruce T et al (2013) The molecular control toolkit: controlling 3D molecular graphics via gesture and voice. 2013 I.E. symposium on biological data visualization, pp 49–56
Kim JO, Park C, Jeong JS et al (2012) A gesture based camera controlling method in the 3D virtual space. J Smart Home 6(4):117–126
Lai K, Konrad J, Ishwar P (2012) A gesture-driven computer interface using kinect. IEEE Southwest symposium on image analysis and interpretation, pp 185–188
Ling Q, Yan J, Li F et al (2014) A background modeling and foreground segmentation approach based on the feedback of moving objects in traffic surveillance systems. Neurocomputing 133:32–45
Manh HT, Lee G (2013) Small object segmentation based on visual saliency in natural images. J Inf Process Syst 9(4):592–601
Ren Z, Yuan J, Zhang Z (2013) Robust part-based hand gesture recognition using Kinect sensor. IEEE Trans Multimed 15(5):1110–1120
Shotton J, Fitzgibbon A, Cook M et al (2011) Real-time human pose recognition in parts from single depth images. IEEE conf. computer vision and pattern recognition, pp 1297–1304
Song W, Cho K, Um K et al (2012) Intuitive terrain reconstruction using height observation-based ground segmentation and 3D object boundary estimation. Sensors 12(12):17186–17207
Vanus, Kucera P, Martinek R et al (2014) Development and testing of a visualization application software, implemented with wireless control system in smart home care. Human-Centric Comput Inf Sci 4(8)
Wang Y, Yang C, Wu X et al (2014) Kinect based dynamic hand gesture recognition algorithm research. 2012 4th international conference on intelligent human-machine systems and cybernetics, pp 274–279
Wilson AD, Benko H (2010) Combining multiple depth cameras and projectors for interactions on, above, and between surfaces. Proceedings of the 23rd annual ACM symposium on User interface software and technology, pp 273–282
Yeh CH, Lin CY, Muchtar K et al (2014) Real-time background modeling based on a multi-level texture description. Inf Sci 269:106–127
Acknowledgments
This research was supported by the MSIP(Ministry of Science, ICT and Future Planning), Korea, under the ITRC(Information Technology Research Center) support program (IITP-2015-H8501-15-1014) supervised by the IITP(Institute for Information & communications Technology Promotion).
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Song, W., Cai, X., Xi, Y. et al. Real-time single camera natural user interface engine development. Multimed Tools Appl 76, 11159–11175 (2017). https://doi.org/10.1007/s11042-015-2986-6
Received:
Revised:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11042-015-2986-6