skip to main content
10.1145/3382507.3418822acmconferencesArticle/Chapter ViewAbstractPublication Pagesicmi-mlmiConference Proceedingsconference-collections
research-article

FilterJoint: Toward an Understanding of Whole-Body Gesture Articulation

Published: 22 October 2020 Publication History

Abstract

Classification accuracy of whole-body gestures can be improved by selecting gestures that have few conflicts (i.e., confusions or misclassifications). To identify such gestures, an understanding of the nuances of how users articulate whole-body gestures can help, especially when conflicts may be due to confusion among seemingly dissimilar gestures. To the best of our knowledge, such an understanding is currently missing in the literature. As a first step to enable this understanding, we designed a method that facilitates investigation of variations in how users move their body parts as they perform a motion. This method, which we call filterJoint, selects the key body parts that are actively moving during the performance of a motion. The paths along which these body parts move in space over time can then be analyzed to make inferences about how users articulate whole-body gestures. We present two case studies to show how the filterJoint method enables a deeper understanding of whole-body gesture articulation, and we highlight implications for the selection of whole-body gesture sets as a result of these insights.

Supplementary Material

ZIP File (icmi1073aux.zip)
This file details the pseudocode of our adaptation of template-based gesture recognition algorithms to multiple articulation paths. This pseudocode can be used in conjunction with motion datasets to perform recognition. On a subset of adults motions from the Kinder-Gator dataset, this algorithm achieved a recognition accuracy of 90.7% using Leave-One-Out-Cross-Validation (LOOCV) (see full paper for details).
MP4 File (3382507.3418822.mp4)
Classification accuracy of whole-body gestures can be improved by selecting gestures that have few conflicts. To identify such gestures, an understanding of the nuances of how users articulate whole-body gestures can help, especially when conflicts may be due to confusion among seemingly dissimilar gestures. However, such an understanding is currently missing in the literature. To enable this understanding, we designed the filterJoint method that selects the key body parts that are actively moving during the performance of a motion to facilitate investigation of variations in how users move their body parts. The paths along which these body parts move in space over time can then be analyzed to make inferences about how users articulate whole-body gestures. We present two case studies to show how the filterJoint method enables a deeper understanding of whole-body gesture articulation, and we highlight implications for the selection of whole-body gesture sets as a result of these insights.

References

[1]
Aishat Aloba, Gianne Flores, Julia Woodward, Alex Shaw, Amanda Castonguay, Isabella Cuba, Yuzhu Dong, Eakta Jain, and Lisa Anthony. 2018. Figure 4: Degree of agreement of children and adults for gestures in our gesture set (lower = higher agreement). Kinder-Gator: The UF kinect database of child and adult motion. In Eurographics (Short Papers), 13--16. https://doi.org/10.2312/egs.20181033
[2]
Lisa Anthony, Quincy Brown, Jaye Nias, Berthel Tate, and Shreya Mohan. 2012. Interaction and recognition challenges in interpreting children's touch and gesture input on mobile devices. In Proceedings of the 2012 ACM international conference on Interactive tabletops and surfaces (ITS '12), 225. https://doi.org/10.1145/2396636.2396671
[3]
Lisa Anthony, Radu-Daniel Vatavu, and Jacob O. Wobbrock. 2013. Understanding the consistency of users? pen and finger stroke gesture articulation. In Proceedings of the Graphics Interface Conference (GI '13), 87--94. https://doi.org/10.5555/2532129.2532145
[4]
Lisa Anthony and Jacob O. Wobbrock. 2012. $ N-Protractor?: A fast and accurate multistroke recognizer. In Proceedings of Graphics Interface (GI '12), 117--120. https://doi.org/10.5555/2305276.2305296
[5]
Victoria Bloom, Vasileios Argyriou, and Dimitrios Makris. 2015. G3di: A gaming interaction dataset with a real time detection and evaluation framework. In European Conference on Computer Vision (ECCV '15), 698--712. https://doi.org/10.1007/978--3--319--16178--5_49
[6]
Aaron F. Bobick and James W. Davis. 2001. The recognition of human movement using temporal templates. IEEE Transactions on Pattern Analysis and Machine Intelligence: 257--267. https://doi.org/http://dx.doi.org/10.1109/34.910878
[7]
Frances Cleland-Donnelly, Suzanne S Mueller, and David L Gallahue. 2017. Developmental Physical Education for All Children: Theory Into Practice.
[8]
John A. Hartigan and Manchek A. Wong. 2006. Algorithm AS 136: A K-Means clustering algorithm. Journal of the Royal Statistical Society Series C (Applied Statistics) 29, 1: 100--108. https://doi.org/10.2307/2346830
[9]
Kathleen Haywood and Nancy Getchell. 2019. Life span motor development. Human Kinetics.
[10]
Hamilton A. Hernandez, Zi Ye, T.C. Nicholas Graham, Darcy Fehlings, and Lauren Switzer. 2013. Designing action-based exergames for children with cerebral palsy. In Proceedings of the ACM SIGCHI Annual Conference on Human Factors in Computing Systems (CHI '13), 1261--1270. https://doi.org/10.1145/2470654.2466164
[11]
Mathew W. Hill, Maximilian M. Wdowski, Adam Pennell, David F. Stodden, and Michael J. Duncan. 2019. Dynamic postural control in children: Do the arms lend the legs a helping hand? Frontiers in Physiology. https://doi.org/10.3389/fphys.2018.01932
[12]
Justin J. Kavanagh, R. S. Barrett, and Steven Morrison. 2004. Upper body accelerations during walking in healthy young and elderly men. Gait and Posture 20, 3: 291--298. https://doi.org/10.1016/j.gaitpost.2003.10.004
[13]
Sven Kratz and Michael Rohs. 2010. A $3 gesture recognizer: simple gesture recognition for devices equipped with 3D acceleration sensors. In Proceeding of the international conference on Intelligent user interfaces (IUI '10), 341--344. https://doi.org/10.1145/1719970.1720026
[14]
Sven Kratz and Michael Rohs. 2011. Protractor3d: a closed-form solution to rotation-invariant 3d gestures. In Proceedings of the International Conference on Intelligent User Interfaces (IUI '11), 371--374. https://doi.org/10.1145/1943403.1943468
[15]
Yang Li. 2010. Protractor: a fast and accurate gesture recognizer. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '10), 2169--2172. https://doi.org/10.1145/1753326.1753654
[16]
Microsoft. Kinect for Windows. Retrieved January 31, 2019 from https://developer.microsoft.com/en-us/windows/kinect
[17]
Microsoft. 2019. Kinect Gesture Data Set. Retrieved October 8, 2019 from https://www.microsoft.com/en-us/download/details.aspx?id=52283
[18]
Lewis M. Nashner. 1980. Balance adjustments of humans perturbed while walking. Journal of neurophysiology 44, 4: 650--664. https://doi.org/10.1152/jn.1980.44.4.650
[19]
Chuong V. Nguyen, Shahram Izadi, and David Lovell. 2012. Modeling kinect sensor noise for improved 3D reconstruction and tracking. In Proceedings of the international conference on 3D imaging, modeling, visualization, & transmission (3DIMPVT '12), 524--530. https://doi.org/10.1109/3DIMPVT.2012.84
[20]
Yi Chung Pai and James Patton. 1997. Center of mass velocity-position predictions for balance control. Journal of Biomechanics 30, 4: 347--354. https://doi.org/10.1016/S0021--9290(96)00165-0
[21]
Otto Parra, Sergio España, and Oscar Pastor. 2016. GestUI: A model-driven method and tool for including gesture-based interaction in user interfaces. Complex Systems Informatics and Modeling Quarterly 6: 73--92. https://doi.org/10.7250/csimq.2016--6.05
[22]
Radu-Daniel Vatavu. 2017. Beyond Features for Recognition: Human-Readable Measures to Understand Users? Whole-Body Gesture Performance. International Journal of Human Computer Interaction 33, 9: 713--730. https://doi.org/http://dx.doi.org/10.1080/10447318.2017.1278897
[23]
Radu-Daniel Vatavu, Lisa Anthony, and Jacob O. Wobbrock. 2013. Relative accuracy measures for stroke gestures. In International Conference on Multimodal Interaction (ICMI '13), 279--286. https://doi.org/10.1145/2522848.2522875
[24]
Radu-Daniel Vatavu, Lisa Anthony, and Jacob O Wobbrock. 2012. Gestures as point clouds: A $P recognizer for user interface prototypes. In ACM International Conference on Multimedia Interaction (ICMI '12), 273--280. https://doi.org/10.1145/2388676.2388732
[25]
Radu Daniel Vatavu, Lisa Anthony, and Jacob O. Wobbrock. 2014. Gesture heatmaps: Understanding gesture performance with colorful visualizations. In Proceedings of the International Conference on Multimodal Interaction (ICMI '14), 172--179. https://doi.org/10.1145/2663204.2663256
[26]
Stefan Waldherr, Roseli Romero, and Sebastian Thrun. 2000. Gesture based interface for human-robot interaction. Autonomous Robots 9, 2: 151--173. https://doi.org/10.1023/A:1008918401478
[27]
Daniel Weinland, Remi Ronfard, and Edmond Boyer. 2006. Free viewpoint action recognition using motion history volumes. Computer Vision and Image Understanding 104, 2--3: 249--257. https://doi.org/10.1016/j.cviu.2006.07.013
[28]
David A. Winter. 1995. Human balance and posture control during standing and walking. Gait and Posture 3, 4: 193--214. https://doi.org/10.1016/09666362(96)82849--9
[29]
Jacob O. Wobbrock, Andrew D. Wilson, and Yang Li. 2007. Gestures without libraries, toolkits or training: a $1 recognizer for user interface prototypes. In ACM symposium on User interface software and technology (UIST '07), 159--168. https://doi.org/10.1145/1294211.1294238
[30]
Svante Wold, Kim Esbensen, and Paul Geladi. 1987. Principal component analysis. Chemometrics and Intelligent Laboratory Systems 2, 1--3: 37--52. https://doi.org/10.1016/0169--7439(87)80084--9
[31]
Tzu Tsung Wong. 2015. Performance evaluation of classification algorithms by k-fold and leave-one-out cross validation. Pattern Recognition 48, 9: 2839-- 2846. https://doi.org/10.1016/j.patcog.2015.03.009

Cited By

View all
  • (2024)AI as Modality in Human Augmentation: Toward New Forms of Multimodal Interaction with AI-Embodied ModalitiesProceedings of the 26th International Conference on Multimodal Interaction10.1145/3678957.3678958(591-595)Online publication date: 4-Nov-2024
  • (2021)Characterizing Children's Motion Qualities: Implications for the Design of Motion Applications for ChildrenProceedings of the 2021 International Conference on Multimodal Interaction10.1145/3462244.3479941(229-238)Online publication date: 18-Oct-2021
  • (2021)GestureMap: Supporting Visual Analytics and Quantitative Analysis of Motion Elicitation Data by Learning 2D EmbeddingsProceedings of the 2021 CHI Conference on Human Factors in Computing Systems10.1145/3411764.3445765(1-12)Online publication date: 6-May-2021

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences
ICMI '20: Proceedings of the 2020 International Conference on Multimodal Interaction
October 2020
920 pages
ISBN:9781450375818
DOI:10.1145/3382507
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 22 October 2020

Permissions

Request permissions for this article.

Check for updates

Author Tags

  1. gesture recognition
  2. kinect
  3. motions
  4. template matching
  5. whole-body gesture articulation
  6. whole-body gestures

Qualifiers

  • Research-article

Funding Sources

  • National Science Foundation

Conference

ICMI '20
Sponsor:
ICMI '20: INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION
October 25 - 29, 2020
Virtual Event, Netherlands

Acceptance Rates

Overall Acceptance Rate 453 of 1,080 submissions, 42%

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)13
  • Downloads (Last 6 weeks)3
Reflects downloads up to 03 Mar 2025

Other Metrics

Citations

Cited By

View all
  • (2024)AI as Modality in Human Augmentation: Toward New Forms of Multimodal Interaction with AI-Embodied ModalitiesProceedings of the 26th International Conference on Multimodal Interaction10.1145/3678957.3678958(591-595)Online publication date: 4-Nov-2024
  • (2021)Characterizing Children's Motion Qualities: Implications for the Design of Motion Applications for ChildrenProceedings of the 2021 International Conference on Multimodal Interaction10.1145/3462244.3479941(229-238)Online publication date: 18-Oct-2021
  • (2021)GestureMap: Supporting Visual Analytics and Quantitative Analysis of Motion Elicitation Data by Learning 2D EmbeddingsProceedings of the 2021 CHI Conference on Human Factors in Computing Systems10.1145/3411764.3445765(1-12)Online publication date: 6-May-2021

View Options

Login options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Figures

Tables

Media

Share

Share

Share this Publication link

Share on social media