Skip to main content
SpringerLink
Log in

Real-time single camera natural user interface engine development

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

Natural user interfaces (NUIs) provide human computer interaction (HCI) with natural and intuitive operation interfaces, such as using human gestures and voice. We have developed a real-time NUI engine architecture using a web camera as a means of implementing NUI applications. The system captures video via the web camera, implements real-time image processing using graphic processing unit (GPU) programming. This paper describes the architecture of the engine and the real-virtual environment interaction methods, such as foreground segmentation and hand gesture recognition. These methods are implemented using GPU programming in order to realize real-time image processing for HCI. To verify the efficacy of our proposed NUI engine, we utilized it in the development and implementation of several mixed reality games and touch-less operation applications, using the developed NUI engine and the DirectX SDK. Our results confirm that the methods implemented by the engine operate in real time and the interactive operations are intuitive.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12

Similar content being viewed by others

References

  1. Ali S, Shah M (2010) Human action recognition in videos using kinematic features and multiple instance learning. IEEE Trans Pattern Anal Mach Intell 32(2):288–303

    Article  Google Scholar 

  2. Betke M, Gips J, Fleming P (2002) The camera mouse: visual tracking of body features to provide computer access for people with severe disabilities. IEEE Trans Neural Syst Rehabil Eng 10(1):1–10

    Article  Google Scholar 

  3. Biswas KK, Basu SK (2011) Gesture recognition using microsoft kinect. Proceedings of the 5th international conference on automation, robotics and applications, pp 100–103

  4. Dickinson P, Hunter A, Appiah K (2009) A spatially distributed model for foreground segmentation. Image Vis Comput 27:1326–1335

    Article  Google Scholar 

  5. Fan YC, Chen CL, Huang SS (2013) Natural user interface for interactive television. 2013 I.E. 17th international symposium on consumer electronics, pp 189–190

  6. Goswami K, Hong GS, Kim BG (2013) A novel mesh-based moving object detection technique in video sequence. J Convergence 4(3):20–24

    Google Scholar 

  7. Ho YS (2013) Challenging technical issues of 3D video processing. J Convergence 4(1):1–6

    Article  Google Scholar 

  8. Homer BD, Kinzer CK, Plass JL et al (2014) Moved to learn: the effects of interactivity in a Kinect-based literacy game for beginning readers. Comput Educ 74:37–49

    Article  Google Scholar 

  9. Hussain A, Abbasi A, Afzulpurkar N (2012) Detecting & interpreting self-manipulating hand movements for student’s affect prediction. Human-Centric Comput Inf Sci 2(14):1–18

    Google Scholar 

  10. Kenneth S, Christian S, Bruce T et al (2013) The molecular control toolkit: controlling 3D molecular graphics via gesture and voice. 2013 I.E. symposium on biological data visualization, pp 49–56

  11. Kim JO, Park C, Jeong JS et al (2012) A gesture based camera controlling method in the 3D virtual space. J Smart Home 6(4):117–126

    Google Scholar 

  12. Lai K, Konrad J, Ishwar P (2012) A gesture-driven computer interface using kinect. IEEE Southwest symposium on image analysis and interpretation, pp 185–188

  13. Ling Q, Yan J, Li F et al (2014) A background modeling and foreground segmentation approach based on the feedback of moving objects in traffic surveillance systems. Neurocomputing 133:32–45

    Article  Google Scholar 

  14. Manh HT, Lee G (2013) Small object segmentation based on visual saliency in natural images. J Inf Process Syst 9(4):592–601

    Article  Google Scholar 

  15. Ren Z, Yuan J, Zhang Z (2013) Robust part-based hand gesture recognition using Kinect sensor. IEEE Trans Multimed 15(5):1110–1120

    Article  Google Scholar 

  16. Shotton J, Fitzgibbon A, Cook M et al (2011) Real-time human pose recognition in parts from single depth images. IEEE conf. computer vision and pattern recognition, pp 1297–1304

  17. Song W, Cho K, Um K et al (2012) Intuitive terrain reconstruction using height observation-based ground segmentation and 3D object boundary estimation. Sensors 12(12):17186–17207

    Article  Google Scholar 

  18. Vanus, Kucera P, Martinek R et al (2014) Development and testing of a visualization application software, implemented with wireless control system in smart home care. Human-Centric Comput Inf Sci 4(8)

  19. Wang Y, Yang C, Wu X et al (2014) Kinect based dynamic hand gesture recognition algorithm research. 2012 4th international conference on intelligent human-machine systems and cybernetics, pp 274–279

  20. Wilson AD, Benko H (2010) Combining multiple depth cameras and projectors for interactions on, above, and between surfaces. Proceedings of the 23rd annual ACM symposium on User interface software and technology, pp 273–282

  21. Yeh CH, Lin CY, Muchtar K et al (2014) Real-time background modeling based on a multi-level texture description. Inf Sci 269:106–127

    Article  MathSciNet  Google Scholar 

Download references

Acknowledgments

This research was supported by the MSIP(Ministry of Science, ICT and Future Planning), Korea, under the ITRC(Information Technology Research Center) support program (IITP-2015-H8501-15-1014) supervised by the IITP(Institute for Information & communications Technology Promotion).

Author information

Authors and Affiliations

  1. College of Computer, North China University of Technology, No.5 Jinyuanzhuang Road, Shijingshan District, Beijing, 100-144, People’s Republic of China

    Wei Song & Xingquan Cai

  2. Department of Multimedia Engineering, Dongguk University-Seoul, 26 Pildong 3 Ga, Jung-gu, Seoul, 100-715, Korea

    Yulong Xi, Seoungjae Cho & Kyungeun Cho

Authors
  1. Wei Song
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Xingquan Cai
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Yulong Xi
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Seoungjae Cho
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Kyungeun Cho
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Kyungeun Cho.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Song, W., Cai, X., Xi, Y. et al. Real-time single camera natural user interface engine development. Multimed Tools Appl 76, 11159–11175 (2017). https://doi.org/10.1007/s11042-015-2986-6

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-015-2986-6

Keywords

Search

Navigation