Skip to main content
SpringerLink
Log in

Robust Motion Recognition Using Gesture Phase Annotation

  • Conference paper
  • First Online:
Digital Human Modeling and Applications in Health, Safety, Ergonomics and Risk Management (HCII 2023)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 14028))

Included in the following conference series:

Abstract

Robust gesture recognition is key to multimodal language understanding as well as human-computer interaction. While vision-based approaches to gesture recognition rightly focus on detecting hand poses in a single frame of video, there is less focus on recognizing the distinct “phases” of gesture as used in real interaction between humans or between humans and computers. Following the semantics of gesture originally outlined by Kendon, and elaborated by many such as McNeill and Lascarides and Stone, we propose a method to automatically detect the preparatory, “stroke,” and recovery phases of semantic gestures. This method can be used to mitigate errors in automatic motion recognition, such as when the hand pose of a gesture is formed before semantic content is intended to be communicated and in semi-automatically creating or augmenting large gesture-speech alignment corpora.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 84.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 109.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    A similar formulation that models deictic precision decreasing with distance is adopted by van der Sluis and Krahmer [25].

References

  1. Allena, C.D., De Leon, R.C., Wong, Y.H.: Easy hand gesture control of a ROS-car using google mediapipe for surveillance use. In: Fui-Hoon Nah, F., Siau, K. (eds.) HCII 2022. LNCS, vol. 13327, pp. 247–260. Springer, Cham (2022). https://doi.org/10.1007/978-3-031-05544-7_19

    Chapter  Google Scholar 

  2. Bradford, M., Khebour, I., Blanchard, N., Krishnaswamy, N.: Automatic detection of collaborative states in small groups using multimodal features. In: International Conference on Artificial Intelligence in Education (under review)

    Google Scholar 

  3. Breiman, L.: Bagging predictors. Mach. Learn. 24, 123–140 (1996)

    Article  MATH  Google Scholar 

  4. Bugarin, C.A.Q., Lopez, J.M.M., Pineda, S.G.M., Sambrano, M.F.C., Loresco, P.J.M.: Machine vision-based fall detection system using mediapipe pose with IoT monitoring and alarm, pp. 269–274 (2022). https://doi.org/10.1109/R10-HTC54060.2022.9929527

  5. Gebre, B.G., Wittenburg, P., Lenkiewicz, P.: Towards automatic gesture stroke detection. In: LREC 2012: 8th International Conference on Language Resources and Evaluation, pp. 231–235. European Language Resources Association (2012)

    Google Scholar 

  6. Indriani, M.H., Agoes, A.S.: Applying hand gesture recognition for user guide application using mediapipe. In: 2nd International Seminar of Science and Applied Technology (ISSAT 2021), pp. 101–108 (2021)

    Google Scholar 

  7. Kandoi, C., Jung, C., Mannan, S., VanderHoeven, H., Meisman, Q., Krishnaswamy, N., Blanchard, N.: Intentional microgesture recognition for extended human-computer interaction. In: HCII 2023. LNCS. Springer, Cham (2023)

    Google Scholar 

  8. Kendon, A., et al.: Gesticulation and speech: two aspects of the process of utterance. In: The Relationship of Verbal and Nonverbal Communication, vol. 25, no. 1980, pp. 207–227 (1980)

    Google Scholar 

  9. Kranstedt, A., Lücking, A., Pfeiffer, T., Rieser, H., Wachsmuth, I.: Deixis: how to determine demonstrated objects using a pointing cone. In: Gibet, S., Courty, N., Kamp, J.-F. (eds.) GW 2005. LNCS (LNAI), vol. 3881, pp. 300–311. Springer, Heidelberg (2006). https://doi.org/10.1007/11678816_34

    Chapter  Google Scholar 

  10. Kranstedt, A., Wachsmuth, I.: Incremental generation of multimodal deixis referring to objects. In: Proceedings of the Tenth European Workshop on Natural Language Generation (ENLG 2005) (2005)

    Google Scholar 

  11. Krishnaswamy, N., Alalyani, N.: Embodied multimodal agents to bridge the understanding gap. In: Proceedings of the First Workshop on Bridging Human-Computer Interaction and Natural Language Processing, pp. 41–46 (2021)

    Google Scholar 

  12. Krishnaswamy, N., et al.: Diana’s world: a situated multimodal interactive agent. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 34, pp. 13618–13619 (2020)

    Google Scholar 

  13. Krishnaswamy, N., et al.: Communicating and acting: understanding gesture in simulation semantics. In: IWCS 2017–12th International Conference on Computational Semantics-Short papers (2017)

    Google Scholar 

  14. Lascarides, A., Stone, M.: Formal semantics for iconic gesture. Universität Potsdam (2006)

    Google Scholar 

  15. Lascarides, A., Stone, M.: A formal semantic analysis of gesture. J. Semant. 26(4), 393–449 (2009)

    Article  Google Scholar 

  16. Lücking, A., Bergman, K., Hahn, F., Kopp, S., Rieser, H.: Data-based analysis of speech and gesture: the bielefeld speech and gesture alignment corpus (SaGA) and its applications. J. Multimodal User Interfaces 7, 5–18 (2013)

    Article  Google Scholar 

  17. Lücking, A., Bergmann, K., Hahn, F., Kopp, S., Rieser, H.: The bielefeld speech and gesture alignment corpus (SaGA). In: LREC 2010 Workshop: Multimodal Corpora-Advances in Capturing, Coding and Analyzing Multimodality (2010)

    Google Scholar 

  18. Lücking, A., Pfeiffer, T., Rieser, H.: Pointing and reference reconsidered. J. Pragmat. 77, 56–79 (2015)

    Article  Google Scholar 

  19. McNeill, D.: Hand and mind. In: Advances in Visual Semiotics, p. 351 (1992)

    Google Scholar 

  20. McNeill, D.: Language and Gesture, vol. 2. Cambridge University Press, Cambridge (2000)

    Book  Google Scholar 

  21. McNeill, D.: Gesture and thought. In: Gesture and Thought. University of Chicago Press (2008)

    Google Scholar 

  22. Narayana, P., Beveridge, R., Draper, B.A.: Gesture recognition: focus on the hands. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 5235–5244 (2018)

    Google Scholar 

  23. Roygaga, C., et al.: APE-V: Athlete Performance Evaluation using Video, pp. 691–700 (2022)

    Google Scholar 

  24. Singh, A.K., Kumbhare, V.A., Arthi, K.: Real-Time Human Pose Detection and Recognition Using MediaPipe, pp. 145–154 (2022). https://doi.org/10.1007/978-981-16-7088-6_12

  25. van der Sluis, I., Krahmer, E.: Generating multimodal references. Discourse Process. 44(3), 145–174 (2007)

    Article  Google Scholar 

  26. Wolf, K., Naumann, A., Rohs, M., Müller, J.: A taxonomy of microinteractions: defining microgestures based on ergonomic and scenario-dependent requirements. In: Campos, P., Graham, N., Jorge, J., Nunes, N., Palanque, P., Winckler, M. (eds.) INTERACT 2011. LNCS, vol. 6946, pp. 559–575. Springer, Heidelberg (2011). https://doi.org/10.1007/978-3-642-23774-4_45

    Chapter  Google Scholar 

  27. Zhang, F., et al.: Mediapipe hands: on-device real-time hand tracking. arXiv preprint arXiv:2006.10214 (2020)

Download references

Acknowledgements

This work was partially supported by the National Science Foundation under awards CNS 2016714 and DRL 1559731 to Colorado State University. The views expressed are those of the authors and do not reflect the official policy or position of the U.S. Government. All errors and mistakes are, of course, the responsibilities of the authors.

Author information

Authors and Affiliations

  1. Colorado State University, Fort Collins, CO, 80523, USA

    Hannah VanderHoeven, Nathaniel Blanchard & Nikhil Krishnaswamy

Authors
  1. Hannah VanderHoeven
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Nathaniel Blanchard
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Nikhil Krishnaswamy
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Hannah VanderHoeven .

Editor information

Editors and Affiliations

  1. Purdue University, West Lafayette, IN, USA

    Vincent G. Duffy

Rights and permissions

Reprints and permissions

Copyright information

© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

VanderHoeven, H., Blanchard, N., Krishnaswamy, N. (2023). Robust Motion Recognition Using Gesture Phase Annotation. In: Duffy, V.G. (eds) Digital Human Modeling and Applications in Health, Safety, Ergonomics and Risk Management. HCII 2023. Lecture Notes in Computer Science, vol 14028. Springer, Cham. https://doi.org/10.1007/978-3-031-35741-1_42

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-35741-1_42

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-35740-4

  • Online ISBN: 978-3-031-35741-1

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation