Skip to main content
Springer Nature Link
Log in

Development and Evaluation of a Mouse Emulator Using Multi-modal Real-Time Head Tracking Systems with Facial Gesture Recognition as a Switching Mechanism

  • Conference paper
  • First Online:
Pattern Recognition. ICPR International Workshops and Challenges (ICPR 2021)

Part of the book series: Lecture Notes in Computer Science ((LNIP,volume 12662))

Included in the following conference series:

  • 2422 Accesses

Abstract

The objective of this study is to evaluate and compare the performance of a set of low-cost multi-modal head tracking systems incorporating facial gestures as a switching mechanism. The proposed systems are aimed to enable severely disabled patients to access a computer. In this paper, we are comparing RGB (2D) and RGB-D (3D) sensors for both head tracking and facial gesture recognition. System evaluations and usability assessment were carried out on 21 healthy individuals. Two types of head tracking systems were compared - a web camera-based and another using the Kinect sensor. The two facial switching mechanisms were eye blink and eyebrows movement. Fitts’ Test is used to evaluate the proposed systems. Movement Time (MT) was used to rank the performance of the proposed systems. The Kinect-Eyebrows system had the lowest MT, followed by the Kinect-Blink, Webcam-Blink and Webcam-Eyebrows systems. The 3D Kinect systems performed better than the 2D Vision systems for both gestures. Both Kinect systems have the lowest MT and best performance, thus showing the advantage of using depth.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

References

  1. Kinect for Windows. http://www.microsoft.com/en-us/kinectforwindows/

  2. SmartNav. http://www.naturalpoint.com/smartnav/

  3. Tobii Technology. http://www.tobii.com/

  4. Abascal, J., Nicolle, C.: Moving towards inclusive design guidelines for socially and ethically aware HCI. Interact. Comput. 17(5), 484–505 (2005). https://doi.org/10.1016/j.intcom.2005.03.002. http://iwc.oxfordjournals.org/cgi/doi/10.1016/j.intcom.2005.03.002

    Article  Google Scholar 

  5. Accot, J., Zhai, S.: Beyond Fitts’ law: models for trajectory-based HCI tasks. In: Proceedings of the ACM SIGCHI Conference on Human Factors in Computing Systems, pp. 295–301 (1997). https://doi.org/10.1145/258549.258760. http://dl.acm.org/citation.cfm?id=258760

  6. Betke, M., Gips, J., Fleming, P.: The camera mouse: visual tracking of body features to provide computer access for people with severe disabilities. IEEE Trans. Neural Syst. Rehabil. Eng. 10(1), 1–10 (2002). A Publication of the IEEE Engineering in Medicine and Biology Society

    Article  Google Scholar 

  7. Betke, M.: Intelligent interfaces to empower people with disabilities. In: Nakashima, H., Aghajan, H., Augusto, J.C. (eds.) Handbook of Ambient Intelligence and Smart Environments, pp. 409–432. Springer, Boston (2010). https://doi.org/10.1007/978-0-387-93808-0_15

    Chapter  Google Scholar 

  8. Bradski, G.: The OpenCV library. Dr. Dobb’s J. Softw. Tools 25, 120–125 (2000)

    Google Scholar 

  9. Cloud, R., Betke, M., Gips, J.: Experiments with a camera-based human-computer interface system. In: Proceedings of the 7th ERCIM Workshop “User Interfaces for All,” UI4ALL 2002, pp. 103–110 (2002). http://cstest.bc.edu/~gips/UI4ALL-2002.pdf www.cs.bu.edu/faculty/betke/papers/Cloud-Betke-Gips-UI4ALL-2002.pdf

  10. Cox, D.R., Snell, E.J.: Analysis of Binary Data, vol. 32. CRC Press, Boca Raton (1989)

    MATH  Google Scholar 

  11. del Val, L., Jiménez, M.I., Alonso, A., de la Rosa, R., Izquierdo, A., Carrera, A.: Assistance system for disabled people: a robot controlled by blinking and wireless link. In: Lytras, M.D., Ordonez De Pablos, P., Ziderman, A., Roulstone, A., Maurer, H., Imber, J.B. (eds.) WSKS 2010. CCIS, vol. 111, pp. 383–388. Springer, Heidelberg (2010). https://doi.org/10.1007/978-3-642-16318-0_45

    Chapter  Google Scholar 

  12. Douglas, S.A., Kirkpatrick, A.E., MacKenzie, I.S.: Testing pointing device performance and user assessment with the ISO 9241, part 9 standard. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, CHI 1999, pp. 215–222. ACM, New York (1999). https://doi.org/10.1145/302979.303042

  13. Fitts, P.M., Radford, B.: Information capacity of discrete motor responses under different cognitive sets. J. Exp. Psychol. 71, 475–482 (1966)

    Article  Google Scholar 

  14. Fitts, P.: The information capacity of the human motor system in controlling the amplitude of movement. J. Exp. Psychol. 47(6) (1954). http://psycnet.apa.org/journals/xge/47/6/381/

  15. Grauman, K., Betke, M., Lombardi, J., Gips, J., Bradski, G.: Communication via eye blinks and eyebrow raises: video-based human-computer interfaces. Univ. Access Inf. Soc. 2(4), 359–373 (2003). https://doi.org/10.1007/s10209-003-0062-x

    Article  Google Scholar 

  16. Guness, S.P., Deravi, F., Sirlantzis, K., Pepper, M.G., Sakel, M.: Evaluation of vision-based head-trackers for assistive devices. In: Proceedings of the Annual International Conference of the IEEE Engineering in Medicine and Biology Society, EMBS, pp. 4804–4807 (2012). https://doi.org/10.1109/EMBC.2012.6347068. http://ieeexplore.ieee.org/xpls/abs_all.jsp?arnumber=6347068

  17. Gustavsson, A., et al.: Cost of disorders of the brain in Europe 2010. Eur. Neuropsychopharmacol. J. Eur. Coll. Neuropsychopharmacol. 21(10), 718–79 (2011). https://doi.org/10.1016/j.euroneuro.2011.08.008. http://www.ncbi.nlm.nih.gov/pubmed/21924589

  18. ISO TC 159/SC 4: ISO 9241–9:2000, ergonomic requirements for office work with visual display terminals (VDTs) - part 9: requirements for non-keyboard input devices. International Organization for Standardization (2002)

    Google Scholar 

  19. Karson, C.N.: Spontaneous eye-blink rates and dopaminergic systems. Brain: J. Neurol. 106(Pt 3), 643–53 (1983). http://www.ncbi.nlm.nih.gov/pubmed/6640274

  20. Kass, M., Witkin, A., Terzopoulos, D.: Snakes: active contour models. Int. J. Comput. Vis. 1(4), 321–331 (1988). https://doi.org/10.1007/BF00133570

    Article  MATH  Google Scholar 

  21. King, D.E.: Dlib-ml: a machine learning toolkit. J. Mach. Learn. Res. (JMLR) 10, 1755–1758 (2009). http://dl.acm.org/citation.cfm?id=1577069.1755843

    Google Scholar 

  22. Krolak, A., Strumillo, P.: Vision-based eye blink monitoring system for human-computer interfacing. In: 2008 Conference on Human System Interactions. Institute of Electrical & Electronics Engineers (IEEE), May 2008. https://doi.org/10.1109/hsi.2008.4581580

  23. MacKenzie, I.S.: Fitts’ law as a research and design tool in human-computer interaction. Hum. -Comput. Interact. 7(1), 91–139 (1992). https://doi.org/10.1207/s15327051hci0701_3

    Article  Google Scholar 

  24. MacKenzie, I.S., Buxton, W.: A tool for the rapid evaluation of input devices using Fitts’ law models. ACM SIGCHI Bull. 25(3), 58–63 (1993). https://doi.org/10.1145/155786.155801. http://portal.acm.org/citation.cfm?doid=155786.155801

  25. Malik, K., Smolka, B.: Eye blink detection using local binary patterns. In: 2014 International Conference on Multimedia Computing and Systems (ICMCS), pp. 385–390, April 2014. https://doi.org/10.1109/ICMCS.2014.6911268

  26. Missimer, E., Betke, M.: Blink and wink detection for mouse pointer control. ACM Press, New York (2010). https://doi.org/10.1145/1839294.1839322. http://portal.acm.org/citation.cfm?doid=1839294.1839322

  27. Ojala, T., Pietikäinen, M., Harwood, D.: A comparative study of texture measures with classification based on featured distributions. Pattern Recogn. 29(1), 51–59 (1996). https://doi.org/10.1016/0031-3203(95)00067-4

    Article  Google Scholar 

  28. Pan, G., Sun, L., Wu, Z., Lao, S.: Eyeblink-based anti-spoofing in face recognition from a generic webcamera. In: 2007 IEEE 11th International Conference on Computer Vision. Institute of Electrical & Electronics Engineers (IEEE) (2007). https://doi.org/10.1109/iccv.2007.4409068

  29. Pistori, H.: Computer vision and digital inclusion of persons with special needs: overview and state of art. In: Computational Modelling of Objects Represented in Images (2018). https://www.taylorfrancis.com/books/e/9781351377133/chapters/10.1201%2F9781315106465-6

  30. Royal College of Physicians: Medical rehabilitation in 2011 and beyond. Report of a working party, November 2010, Royal College of Physicians & British Society of Rehabilitation Medicine. RCP, London (2010)

    Google Scholar 

  31. Tuisku, O., Surakka, V., Vanhala, T., Rantanen, V., Lekkala, J.: Wireless face interface: using voluntary gaze direction and facial muscle activations for human-computer interaction. Interact. Comput. 24(1), 1–9 (2012). https://doi.org/10.1016/j.intcom.2011.10.002. http://iwc.oxfordjournals.org/cgi/doi/10.1016/j.intcom.2011.10.002

  32. Viola, P., Jones, M.J.: Robust real-time face detection. Int. J. Comput. Vis. 57(2), 137–154 (2004). https://doi.org/10.1023/B:VISI.0000013087.49260.fb

    Article  Google Scholar 

  33. Viola, P., Jones, M.: Robust real-time object detection. Int. J. Comput. Vis. 57, 137–154 (2001). https://doi.org/10.1023/B:VISI.0000013087.49260.fb

    Article  Google Scholar 

  34. Ward, D.J., Blackwell, A.F., MacKay, D.J.C.: Dasher–a data entry interface using continuous gestures and language models, pp. 129–137. ACM Press (2000). https://doi.org/10.1145/354401.354427. http://portal.acm.org/citation.cfm?doid=354401.354427

  35. Wilson, G.F.: An analysis of mental workload in pilots during flight using multiple psychophysiological measures. Int. J. Aviat. Psychol. 12(1), 3–18 (2002)

    Article  Google Scholar 

  36. World Health Organization: World report on disability. Technical report, WHO, Geneva (2011). http://whqlibdoc.who.int/publications/2011/9789240685215_eng.pdf

  37. Yunqi, L., Meiling, Y., Xiaobing, S., Xiuxia, L., Jiangfan, O.: Recognition of eye states in real time video, pp. 554–559, January 2009. https://doi.org/10.1109/ICCET.2009.105. http://ieeexplore.ieee.org/lpdocs/epic03/wrapper.htm?arnumber=4769528

  38. Zhang, X., Kulkarni, H., Morris, M.R.: Smartphone-based gaze gesture communication for people with motor disabilities. In: Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems, CHI 2017, pp. 2878–2889. ACM, New York (2017). https://doi.org/10.1145/3025453.3025790

Download references

Author information

Authors and Affiliations

  1. net.connection Ltd., Moka, Mauritius

    Shivanand P. Guness

  2. School of Digital Arts and Engineering, University of Kent, Canterbury, Kent, UK

    Farzin Deravi, Konstantinos Sirlantzis & Matthew G. Pepper

  3. East Kent University Hospitals Trust, Canterbury, Kent, UK

    Matthew G. Pepper & Mohammed Sakel

Authors
  1. Shivanand P. Guness
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Farzin Deravi
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Konstantinos Sirlantzis
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Matthew G. Pepper
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Mohammed Sakel
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Shivanand P. Guness .

Editor information

Editors and Affiliations

  1. Dipartimento di Ingegneria dell’Informazione, University of Firenze, Firenze, Italy

    Alberto Del Bimbo

  2. Dipartimento di Ingegneria “Enzo Ferrari”, Università di Modena e Reggio Emilia, Modena, Italy

    Rita Cucchiara

  3. Department of Computer Science, Boston University, Boston, MA, USA

    Stan Sclaroff

  4. Dipartimento di Matematica e Informatica, University of Catania, Catania, Italy

    Giovanni Maria Farinella

  5. Cloud & AI, JD.COM, Beijing, China

    Tao Mei

  6. Dipartimento di Ingegneria dell’Informazione, Universita di Firenze, Firenze, Italy

    Marco Bertini

  7. Computational Sciences Department, National Institute of Astrophysics, Optics and Electronics (INAOE), Tonantzintla, Puebla, Mexico

    Hugo Jair Escalante

  8. Dipartimento di Ingegneria “Enzo Ferrari”, Università di Modena e Reggio Emilia, Modena, Italy

    Roberto Vezzani

Rights and permissions

Reprints and permissions

Copyright information

© 2021 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Guness, S.P., Deravi, F., Sirlantzis, K., Pepper, M.G., Sakel, M. (2021). Development and Evaluation of a Mouse Emulator Using Multi-modal Real-Time Head Tracking Systems with Facial Gesture Recognition as a Switching Mechanism. In: Del Bimbo, A., et al. Pattern Recognition. ICPR International Workshops and Challenges. ICPR 2021. Lecture Notes in Computer Science(), vol 12662. Springer, Cham. https://doi.org/10.1007/978-3-030-68790-8_18

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-68790-8_18

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-68789-2

  • Online ISBN: 978-3-030-68790-8

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Societies and partnerships