Skip to main content
Springer Nature Link
Log in

Universal Hand Gesture Interaction Vocabulary for Cross-Cultural Users: Challenges and Approaches

  • Conference paper
  • First Online:
HCI International 2024 Posters (HCII 2024)

Part of the book series: Communications in Computer and Information Science ((CCIS,volume 2114))

Included in the following conference series:

  • 796 Accesses

Abstract

This paper highlights the complexity of creating a universal, cross-cultural Hand Gesture Recognition (HGR) vocabulary for global products and systems where worldwide users interact by sharing the same system or space. We revisit the concept of an idealistic universal HGR vocabulary for cross-cultural users and systems that suit all users and present potential development challenges by reviewing the hand gesture taxonomy, lexical, vocabulary, and the current design methods. The analysis emphasizes the importance of creating a cohesive movement towards standardizing HGR vocabulary for innovative products and services that accommodate cultural diversity and converge for a universal agreement at some point.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

References

  1. Ardito, C., Costabile, M.F., Jetter, H.C.: Gestures that people can understand and use. J. Visual Lang. Comput. 25(5), 572–576 (2014). https://doi.org/10.1016/j.jvlc.2014.07.002

    Article  Google Scholar 

  2. Bailey, S.K.T., Johnson, C.I.: A human-centered approach to designing gestures for natural user interfaces. In: Proceedings of the 22nd International Conference on Human-Computer Interaction, pp. 3–18 (2020). https://doi.org/10.1007/978-3-030-49062-1_1

  3. Brito, I.V., Freire, E.O., Carvalho, E.A.N., Molina, L.: Analysis of cross-cultural effect on gesture-based human-robot interaction. Int. J. Mech. Eng. Robot. Res. 8(6), 852–859 (2019). https://doi.org/10.18178/ijmerr.8.6.852-859

  4. Brudy, F., et al.: Cross-device taxonomy: survey, opportunities and challenges of interactions spanning across multiple devices. In: Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems, pp. 1–28 (2019). https://doi.org/10.1145/3290605.3300792

  5. Cao, X., Hsu, Y., Wu, W.: Cross-cultural design: a set of design heuristics for concept generation of sustainable packagings. In: Proceedings of the 23rd International Conference on Human-Computer Interaction, pp. 197–209 (2021). https://doi.org/10.1007/978-3-030-77074-7_16

  6. Carfì, A., Mastrogiovanni, F.: Gesture-based human-machine interaction: taxonomy, problem definition, and analysis. IEEE Trans. Cybern. 53(1), 497–513 (2023). https://doi.org/10.1109/TCYB.2021.3129119

    Article  Google Scholar 

  7. Chang, X., et al.: It must be gesturing towards me: gesture-based interaction between autonomous vehicles and pedestrians, pp. 1–25 (2024). https://doi.org/10.48550/arXiv.2402.14455

  8. Chaudhary, A., Raheja, J.L., Das, K., Raheja, S.: Intelligent approaches to interact with machines using hand gesture recognition in natural way: a survey. Int. J. Comput. Sci. Eng. Surv. 2(1), 122–133 (2011). https://doi.org/10.5121/ijcses.2011.2109

    Article  Google Scholar 

  9. Deshpande, K., Mashalkar, V., Mhaisekar, K., Naikwadi, A., Ghotkar, A.: Study and survey on gesture recognition systems. In: Proceedings of the 2023 7th International Conference On Computing, Communication, Control and Automation, pp. 1–6 (2023). https://doi.org/10.1109/iccubea58933.2023.10392214

  10. Gheran, B.F., Vanderdonckt, J., Vatavu, R.D.: Gestures for smart rings: empirical results, insights, and design implications. In: Proceedings of the 2018 Designing Interactive Systems Conference, pp. 623–635 (2018). https://doi.org/10.1145/3196709.3196741

  11. Gheran, B.F., Vatavu, R.D., Vanderdonckt, J.: New insights into user-defined smart ring gestures with implications for gesture elicitation studies. In: Proceedings of the 2023 Annual ACM Conference Extended Abstracts on Human Factors in Computing Systems, pp. 1–8 (2023). https://doi.org/10.1145/3544549.3585590

  12. Good, M.D., Whiteside, J.A., Wixon, D.R., Jones, S.J.: Building a user-derived interface. Commun. ACM 27(10), 1032–1043 (1984). https://doi.org/10.1145/358274.358284

    Article  Google Scholar 

  13. Gope, D.C.: Hand gesture interaction with human-computer. Global J. Comput. Sci. Technol. 11(23), 3–12 (2011). https://computerresearch.org/index.php/computer/article/view/414

  14. Hasler, B.S., Salomon, O., Tuchman, P., Lev-Tov, A., Friedman, D.: Real-time gesture translation in intercultural communication. AI Soc. 32(1), 25–35 (2014). https://doi.org/10.1007/s00146-014-0573-4

    Article  Google Scholar 

  15. Hitz, M., Königstorfer, E., Peshkova, E.: Exploring cognitive load of single and mixed mental models gesture sets for UAV navigation. In: Proceedings of the 2019 Annual ACM Conference Extended Abstracts on Human Factors in Computing Systems, pp. 1–8 (2019). https://api.semanticscholar.org/CorpusID:198332850

  16. Hofstede, G.: The business of international business is culture. Int. Bus. Rev. 3(1), 1–14 (1994). https://doi.org/10.1016/0969-5931(94)90011-6

    Article  Google Scholar 

  17. Hosseini, M., Ihmels, T., Chen, Z., Koelle, M., Müller, H., Boll, S.: Towards a consensus gesture set: a survey of mid-air gestures in HCI for maximized agreement across domains. In: Proceedings of the 2023 CHI Conference on Human Factors in Computing System, pp. 1–24 (2023). https://doi.org/10.1145/3544548.3581420

  18. Ingram, A., Wang, X., Ribarsky, W.: Towards the establishment of a framework for intuitive multi-touch interaction design. In: Proceedings of the 2012 International Working Conference on Advanced Visual Interfaces, pp. 66–73 (2012). https://doi.org/10.1145/2254556.2254571

  19. International Organization for Standardization: ISO/TS 9241-430:2021 – Ergonomics of Human-System Interaction, Part 430: Recommendations for the Design of Non-Touch Gestural Input for the Reduction of Biomechanical Stress (2021). https://www.iso.org/standard/80270.html

  20. Karam, M., Schraefel, M.C.: A Taxonomy of Gestures in Human Computer Interactions. Resreport, University of Southampton (Soton), United Kingdom (2005)

    Google Scholar 

  21. Kendon, A.: Gesture: Visible Action as Utterance. Cambridge University Press (2004). https://doi.org/10.1017/CBO9780511807572

  22. Koh, J.I., Cherian, J., Taele, P., Hammond, T.: Developing a hand gesture recognition system for mapping symbolic hand gestures to analogous emojis in computer-mediated communication. ACM Trans. Interact. Intell. Syst. 9(1), 1–35 (2019). https://doi.org/10.1145/3297277

    Article  Google Scholar 

  23. Lindenberg, R., Uhlig, M., Scherfeld, D., Schlaug, G., Seitz, R.J.: Communication with emblematic gestures: shared and distinct neural correlates of expression and reception. Hum. Brain Mapp. 33(4), 812–823 (2011). https://doi.org/10.1002/hbm.21258

    Article  Google Scholar 

  24. Lugaresi, C., et al.: MediaPipe: a framework for perceiving and processing reality. In: Proceedings of the 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1–9 (2019). https://doi.org/10.48550/arXiv.1906.08172

  25. Madapana, N., Gonzalez, G., Rodgers, R., Zhang, L., Wachs, J.P.: Gestures for picture archiving and communication systems (PACS) operation in the operating room: is there any standard? PLoS ONE 13(6), 1–13 (2018). https://doi.org/10.1371/journal.pone.0198092

    Article  Google Scholar 

  26. Maricchiolo, F., Gnisci, A., Bonaiuto, M.: Coding hand gestures: a reliable taxonomy and a multi-media support. In: Esposito, A., Esposito, A.M., Vinciarelli, A., Hoffmann, R., Müller, V.C. (eds.) Cognitive Behavioural Systems. LNCS, vol. 7403, pp. 405–416. Springer, Heidelberg (2012). https://doi.org/10.1007/978-3-642-34584-5_36

    Chapter  Google Scholar 

  27. McDonald, N., Schoenebeck, S., Forte, A.: Reliability and inter-rater reliability in qualitative research: norms and guidelines for CSCW and HCI practice. Proc. ACM Hum.-Comput. Interact. 3(CSCW), 1–23 (2019). https://doi.org/10.1145/3359174

  28. McNeill, D.: Hand and Mind: What Gestures Reveal about Thought. University of Chicago Press (1992). https://doi.org/10.2307/1576015

  29. Morgado, L.: Cultural awareness and personal customization of gestural commands using a shamanic interface. Procedia Comput. Sci. 27, 449–459 (2014). https://doi.org/10.1016/j.procs.2014.02.049

    Article  Google Scholar 

  30. Morris, M.R., Wobbrock, J.O., Wilson, A.D.: Understanding users’ preferences for surface gestures. In: Proceedings of the 2010 Graphics Interface, pp. 261–268 (2010). https://doi.org/10.5555/1839214.1839260

  31. Nielsen, M., Störring, M., Moeslund, T.B., Granum, E.: A procedure for developing intuitive and ergonomic gesture interfaces for HCI. In: Camurri, A., Volpe, G. (eds.) GW 2003. LNCS (LNAI), vol. 2915, pp. 409–420. Springer, Heidelberg (2004). https://doi.org/10.1007/978-3-540-24598-8_38

    Chapter  Google Scholar 

  32. Nielsen, S., Nellemann, L.J., Larsen, L.B., Stec, K.: The social acceptability of peripheral interaction with 3D gestures in a simulated setting. In: Proceedings of the 22nd International Conference on Human-Computer Interaction, pp. 77–95 (2020). https://doi.org/10.1007/978-3-030-49062-1_5

  33. Norman, D.: Gesture Wars (2011). https://www.core77.com/posts/20272/gesture-wars-20272

  34. Norman, D.A.: Natural user interfaces are not natural. Interactions 17(3), 6–10 (2010). https://doi.org/10.1145/1744161.1744163

    Article  Google Scholar 

  35. Rakkolainen, I., et al.: State of the art in extended reality – multimodal interaction. Techreport, Tampere University (TAU), Finland (2021)

    Google Scholar 

  36. Rico, J., Brewster, S.: Usable gestures for mobile interfaces: evaluating social acceptability. In: Proceedings of the 2010 CHI Conference on Human Factors in Computing Systems, pp. 887–896 (2010). https://doi.org/10.1145/1753326.1753458

  37. Rubin, Z.L.: A framework for cross-cultural product design: the designer’s guide to cultural research and design. Georgia Institute of Technology (2012)

    Google Scholar 

  38. Sarma, D., Bhuyan, M.K.: Methods, databases and recent advancement of vision-based hand gesture recognition for HCI systems: a review. SN Comput. Sci. 2, 436 (2021). https://doi.org/10.1007/s42979-021-00827-x

    Article  Google Scholar 

  39. Shamma, D.A., Marlow, J., Denoue, L.: Interacting with smart consumer cameras: exploring gesture, voice, and AI control in video streaming. In: Proceedings of the 2019 ACM International Conference Interaction Experiences TV Online Video, pp. 137–144 (2019). https://doi.org/10.1145/3317697.3323359

  40. Taralle, F., Paljic, A., Manitsaris, S., Grenier, J., Guettier, C.: A consensual and non-ambiguous set of gestures to interact with UAV in infantrymen. In: Proceedings of the 2015 Annual ACM Conference Extended Abstracts on Human Factors in Computing Systems, pp. 797–803 (2015). https://doi.org/10.1145/2702613.2702971

  41. Tsandilas, T.: Fallacies of agreement: a critical review of consensus assessment methods for gesture elicitation. ACM Trans. Comput.-Hum. Interact. 25(3), 1–49 (2018). https://doi.org/10.1145/3182168

  42. Urakami, J.: Developing and testing a human-based gesture vocabulary for tabletop systems. Hum. Factors J. Hum. Factors Ergon. Soc. 54(4), 636–653 (2012). https://doi.org/10.1177/0018720811433052

  43. Vatavu, R.D.: Gesture-Based Interaction, pp. 1–47. Springer, Cham (2023). https://doi.org/10.1007/978-3-319-27648-9_20-1

  44. Vatavu, R.D.: iFAD gestures: understanding users’ gesture input performance with index-finger augmentation devices. In: Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems, pp. 1–17 (2023). https://doi.org/10.1145/3544548.3580928

  45. Vatavu, R.D., Wobbrock, J.O.: Formalizing agreement analysis for elicitation studies: new measures, significance test, and toolkit. In: Proceedings of the 2015 CHI Conference on Human Factors in Computing Systems, pp. 1325–1334 (2015). https://doi.org/10.1145/2702123.2702223

  46. Vatavu, R.D., Wobbrock, J.O.: Clarifying agreement calculations and analysis for end-user elicitation studies. ACM Trans. Comput.-Hum. Interact. 29(1), 5:1–5:70 (2022). https://doi.org/10.1145/3476101

  47. Vuletic, T., Duffy, A., Hay, L., McTeague, C., Campbell, G., Grealy, M.: Systematic literature review of hand gestures used in human computer interaction interfaces. Int. J. Hum.-Comput. Stud. 129, 74–94 (2019). https://doi.org/10.1016/j.ijhcs.2019.03.011

    Article  Google Scholar 

  48. Vuletic, T., et al.: A novel user-based gesture vocabulary for conceptual design. Int. J. Hum.-Comput. Stud. 150, 1–25 (2021). https://doi.org/10.1016/j.ijhcs.2021.102609

    Article  Google Scholar 

  49. Wei, X.L., Xi, R., Hou, W.J.: User-centric AR sceneized gesture interaction design. In: Proceedings of the 22nd International Conference on Human Factors in Computing Systems, pp. 367–378 (2020). https://doi.org/10.1007/978-3-030-49695-1_24

  50. Willms, J., Letter, M., Marchandise, E., Wolf, K.: Pull outperforms push as vibrotactile wristband feedback for mid-air gesture guidance. In: Proceedings of the 2023 Mensch Computer, pp. 138–148 (2023). https://doi.org/10.1145/3603555.3603579

  51. Wobbrock, J.O., Aung, H.H., Rothrock, B., Myers, B.A.: Maximizing the guessability of symbolic input. In: Proceedings of the 2005 Annual ACM Conference Extended Abstracts on Human Factors in Computing Systems, pp. 1869–1872 (2005). https://doi.org/10.1145/1056808.1057043

  52. Wobbrock, J.O., Morris, M.R., Wilson, A.D.: User-defined gestures for surface computing. In: Proceedings of the 2009 CHI Conference on Human Factors in Computing Systems, pp. 1083–1092 (2009). https://doi.org/10.1145/1518701.1518866

  53. Wu, H., Fu, S., Yang, L., Zhang, X.L.: Exploring frame-based gesture design for immersive VR shopping environments. Behav. Inf. Technol. 41(1), 96–117 (2022). https://doi.org/10.1080/0144929x.2020.1795261

    Article  Google Scholar 

  54. Yasen, M., Jusoh, S.: A systematic review on hand gesture recognition techniques, challenges and applications. PeerJ Comput. Sci. 5(e218), 1–30 (2019). https://doi.org/10.7717/peerj-cs.218

    Article  Google Scholar 

  55. Yashas, J., Shivakumar, G.: Hand gesture recognition: a survey. In: Proceedings of the 2019 International Conference on Applied Machine Learning, pp. 3–8 (2019). https://doi.org/10.1109/icaml48257.2019.00009

  56. Zeng, S., Xu, R., Huang, S.: The application and expression of product modeling design from cross-cultural perspective. In: Proceedings of the 25th International Conference on Human-Computer Interaction, pp. 564–579 (2023). https://doi.org/10.1007/978-3-031-35936-1_42

  57. Zheng, Z.: Human gesture recognition in computer vision research. SHS Web Conf. 144, 1–5 (2022). https://doi.org/10.1051/shsconf/202214403011

    Article  Google Scholar 

  58. Zholshiyeva, L., Zhukabayeva, T., Turaev, S., Berdiyeva, M., Jambulova, D.: Hand gesture recognition methods and applications: a literature survey. In: Proceedings of the 7th International Conference on Engineering MIS, pp. 1–8 (2021). https://doi.org/10.1145/3492547.3492578

Download references

Acknowledgments

Original versions of the hand pose illustrations used in Fig. 1 (b and c) were designed by pch.vector/Freepik from https://www.freepik.com (“Human hand gestures set” pack, https://www.freepik.com/free-vector/human-hand-gestures-set_8609352.htm) released under the Freepik license, free for personal and commercial purpose with attribution.

Author information

Authors and Affiliations

  1. IT University of Copenhagen (ITU), 2300, København S., Denmark

    Elizabete Munzlinger, Fabricio Batista Narcizo & Dan Witzner Hansen

  2. GN Audio A/S (Jabra), 2750, Ballerup, Denmark

    Elizabete Munzlinger, Fabricio Batista Narcizo & Ted Vucurevich

Authors
  1. Elizabete Munzlinger
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Fabricio Batista Narcizo
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Dan Witzner Hansen
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Ted Vucurevich
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Elizabete Munzlinger .

Editor information

Editors and Affiliations

  1. University of Crete and Foundation for Research and Technology - Hellas (FORTH), Heraklion, Crete, Greece

    Constantine Stephanidis

  2. Foundation for Research and Technology - Hellas (FORTH), Heraklion, Crete, Greece

    Margherita Antona

  3. Foundation for Research and Technology - Hellas (FORTH), Heraklion, Crete, Greece

    Stavroula Ntoa

  4. University of Central Florida, Orlando, FL, USA

    Gavriel Salvendy

Ethics declarations

Disclosure of Interests

The first author is supported by a grant from Innovation Fund Denmark (Award No. 3129-00046B) as part of an industrial Ph.D. research program in collaboration with GN Audio/AS and IT University of Copenhagen.

Rights and permissions

Reprints and permissions

Copyright information

© 2024 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Munzlinger, E., Batista Narcizo, F., Witzner Hansen, D., Vucurevich, T. (2024). Universal Hand Gesture Interaction Vocabulary for Cross-Cultural Users: Challenges and Approaches. In: Stephanidis, C., Antona, M., Ntoa, S., Salvendy, G. (eds) HCI International 2024 Posters. HCII 2024. Communications in Computer and Information Science, vol 2114. Springer, Cham. https://doi.org/10.1007/978-3-031-61932-8_8

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-61932-8_8

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-61931-1

  • Online ISBN: 978-3-031-61932-8

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics