Skip to main content
SpringerLink
Log in

Multi-modal user interface combining eye tracking and hand gesture recognition

  • Original Paper
  • Published:
Journal on Multimodal User Interfaces Aims and scope Submit manuscript

Abstract

Many studies on eye tracking have been conducted in diverse research areas. Nevertheless, eye tracking continues to be limited by low accuracy and a severe vibration problem due to pupil tremors. Furthermore, because almost all selection interactions, such as click events, use a dwell-time or eye-blinking method, eye tracking presents issues for both time consumption and involuntary blinking. In this paper, we therefore propose a multi-modal interaction method using a combination of eye tracking and hand gesture recognition with the commercial hand gesture controller. This method performs global and intuitive navigation using eye tracking, and local and detailed navigation using hand gesture controller. It supports intuitive hand gestures for mouse-button clicking. Experimental results indicate that the targeting time for small points is significantly improved using the proposed method. Especially, the proposed method has advantages in large display with high spatial resolution environment. Also, the proposed clicking interaction and modality switching concept showed accurate recognition rate and positive training effect, respectively.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16
Fig. 17

Similar content being viewed by others

References

  1. MacKenzie IS, Zhang X (2008) Eye typing using word and letter prediction and a fixation algorithm. In: 2008 symposium on eye tracking research and applications, pp 55–58

  2. Zhu Z, Ji Q (2004) Eye and gaze tracking for interactive graphic display. Mach Vis Appl 15(July):139–148

    Google Scholar 

  3. Cutrell E, Guan Z (2007) What are you looking for? An eye-tracking study of information usage in web Search. SIGCHI conference on human factors in computing systems, pp 407–416

  4. Pan B, Hembrooke HA, Gay GK, Granka LA, Feusner MK, Newman JK (2004) Determinants of web page viewing behavior: an eye-tracking study. In: 2004 symposium on eye tracking research and applications, pp 147–154

  5. Lee EC, Park KR (2007) A study on eye gaze estimation method based on cornea model of human eye. Comput Vis/Comput Gr Collab Tech 4418(March):307–317

    Google Scholar 

  6. Suh KH, Kim Y, Kim Y, Ko D, Lee EC (2015) Monocular eye tracking system using webcam and zoom lens. Adv Multimed Ubiquitous Eng 352(June):135–141

    Google Scholar 

  7. Grauman K, Betke M, Gips J, Bradski GR (2001) Communication via eye blinks: detection and duration analysis in real time. In: 2001 IEEE computer society conference on computer vision and pattern recognition, vol 1, pp I-1010-I-1017

  8. Grauman K, Betke M, Lombardi J, Gips J, Bradski GR (2003) Communication via eye blinks and eyebrow raises: video-based human-computer interfaces. Univers Access Inf Soc 2:2–4

    Article  Google Scholar 

  9. Yamamoto Y, Yoda I, Sakaue K (2004) Arm-pointing gesture interface using surrounded stereo cameras system. In: 17th international conference on pattern recognition, vol 4, pp 965–970

  10. Jing P, Yepeng G (2013) Human-computer Interaction using pointing gesture based on an adaptive virtual touch screen. J Signal Process Pattern Recognit 6(4):81–92

    Google Scholar 

  11. Yepeng G, Mingen Z (2008) Real-time 3D pointing gesture recognition for natural HCI. In: 7th World Congress on intelligent control and automation, pp 2433–2436

  12. Carbini S, Viallet JE, Bernier O (2004) Pointing gesture visual recognition for large display. In: International workshop on visual observation of deictic gestures, pp 27–32

  13. Park H, Choi J, Park J, Moon K (2013) A study on hand region detection for kinect-based hand shape recognition. Korean Soc Broadcast Eng 18(3):393–400

    Article  Google Scholar 

  14. Choi J, Park H, Park J (2011) Hand shape recognition using distance transform and shape decomposition. In: 18th IEEE international conference on image processing (ICIP), pp 3605–3608

  15. Oikonomidis I, Kyriazis N, Argyros A (2010) Markerless and efficient 26-DOF hand pose recovery. In: 10th Asian conference on computer vision, pp 744–757

  16. Cao C, Sun Y, Li R, Chen L (2011) Hand posture recognition via joint feature sparse representation. Optical Eng 50(12):127210

    Article  Google Scholar 

  17. Shin G, Chun J (2007) Vision-based multimodal human computer interface based on parallel tracking of eye and hand motion. In: 2007 international conference on convergence information technology, pp 2443–2448

  18. Bachmann D, Weichert F, Rinkenauer G (2014) Evaluation of the leap motion controller as a new contact-free pointing device. Sensors 15:214–233

    Article  Google Scholar 

  19. http://sourceforge.net/projects/opencvlibrary. Accessed 20 Sept 2015

  20. http://ashlandtech.org/2014/04/02/product-comparison-kinect-and-leap-motion/. Accessed 30 Sept 2015

  21. Guna J, Jakus G, Pogacnik M, Tomazic S, Sodnik J (2014) An analysis of the precision and reliability of the leap motion sensor and its suitability for static and dynamic tracking. Sensors 14:3702–3720

    Article  Google Scholar 

  22. https://developer.leapmotion.com/documentation/cpp/devguide/Leap_Coordinate_Mapping.html. Accessed 30 Sept 2015

  23. http://blog.leapmotion.com/getting-started-leap-motion-sdk/. Accessed 30 Sept 2015

  24. SetCursorPos Function (Windows), http://msdn.microsoft.com/en-us/library/windows/desktop/ms648394(v=vs.85).aspx. Accessed 30 Sept 2015

Download references

Acknowledgements

This work was supported by the ICT R&D program of MSIP/IITP. [R0126-15-1045, The development of technology for social life logging based on analysis of social emotion and intelligence of convergence content].

Author information

Authors and Affiliations

  1. Department of Computer Science, Sangmyung University, 7 Hongji-dong, Jongno-gu, Seoul, 110-743, Republic of Korea

    Hansol Kim, Kun Ha Suh & Eui Chul Lee

Authors
  1. Hansol Kim
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Kun Ha Suh
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Eui Chul Lee
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Eui Chul Lee.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Kim, H., Suh, K.H. & Lee, E.C. Multi-modal user interface combining eye tracking and hand gesture recognition. J Multimodal User Interfaces 11, 241–250 (2017). https://doi.org/10.1007/s12193-017-0242-2

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s12193-017-0242-2

Keywords

Search

Navigation