skip to main content
10.1145/3025453.3026002acmconferencesArticle/Chapter ViewAbstractPublication PageschiConference Proceedingsconference-collections
research-article

Jackknife: A Reliable Recognizer with Few Samples and Many Modalities

Published: 02 May 2017 Publication History

Abstract

Despite decades of research, there is yet no general rapid prototyping recognizer for dynamic gestures that can be trained with few samples, work with continuous data, and achieve high accuracy that is also modality-agnostic. To begin to solve this problem, we describe a small suite of accessible techniques that we collectively refer to as the Jackknife gesture recognizer. Our dynamic time warping based approach for both segmented and continuous data is designed to be a robust, go-to method for gesture recognition across a variety of modalities using only limited training samples. We evaluate pen and touch, Wii Remote, Kinect, Leap Motion, and sound-sensed gesture datasets as well as conduct tests with continuous data. Across all scenarios we show that our approach is able to achieve high accuracy, suggesting that Jackknife is a capable recognizer and good first choice for many endeavors.

Supplementary Material

suppl.mov (pn4460-file3.mp4)
Supplemental video

References

[1]
Lisa Anthony and Jacob O. Wobbrock. 2010. A Lightweight Multistroke Recognizer for User Interface Prototypes. In Proceedings of Graphics Interface 2010 (GI '10). Canadian Information Processing Society, Toronto, Ont., Canada, Canada, 245--252.
[2]
Lisa Anthony and Jacob O. Wobbrock. 2012. $N-protractor: A Fast and Accurate Multistroke Recognizer. In Proceedings of Graphics Interface 2012 (GI '12). Canadian Information Processing Society, Toronto, Ont., Canada, Canada, 117--120.
[3]
Anthony Bagnall, Aaron Bostrom, James Large, and Jason Lines. 2016. The Great Time Series Classification Bake Off: An Experimental Evaluation of Recently Proposed Algorithms. Extended Version. arXiv preprint arXiv:1602.01711 (2016).
[4]
Gustavo E.A.P.A. Batista, Xiaoyue Wang, and Eamonn J. Keogh. 2011. A Complexity-Invariant Distance Measure for Time Series. Chapter 60, 699--710.
[5]
Saša Bodiroža, Guillaume Doisy, and Verena Vanessa Hafner. 2013. Position-invariant, Real-time Gesture Recognition Based on Dynamic Time Warping. In Proceedings of the 8th ACM/IEEE International Conference on Human-robot Interaction (HRI '13). IEEE Press, Piscataway, NJ, USA, 87--88.
[6]
Sait Celebi, Ali Selman Aydin, Talha Tarik Temiz, and Tarik Arici. 2013. Gesture Recognition using Skeleton Data with Weighted Dynamic Time Warping. In VISAPP (1). 620--625.
[7]
Salman Cheema, Michael Hoffman, and Joseph J. LaViola Jr. 2013. 3D Gesture classification with linear acceleration and angular velocity sensing devices for video games. Entertainment Computing 4, 1 (2013), 11--24.
[8]
Heng Du and TszHang To. 2011. Hand gesture recognition using Kinect. Techical Report, Boston University (2011).
[9]
Chris Ellis, Syed Zain Masood, Marshall F. Tappen, Joseph J. Laviola, Jr., and Rahul Sukthankar. 2013. Exploring the Trade-off Between Accuracy and Observational Latency in Action Recognition. International Journal of Computer Vision 101, 3 (Feb. 2013), 420--436.
[10]
Rafael Giusti and Gustavo E. A. P. A. Batista. 2013. An Empirical Comparison of Dissimilarity Measures for Time Series Classification. In Proceedings of the 2013 Brazilian Conference on Intelligent Systems (BRACIS '13). IEEE Computer Society, Washington, DC, USA, 82--88.
[11]
Sidhant Gupta, Daniel Morris, Shwetak Patel, and Desney Tan. 2012. Soundwave: using the doppler effect to sense gestures. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM, 1911--1914.
[12]
Noor A. Ibraheem and Rafiqul Z. Khan. 2012. Survey on Various Gesture Recognition Technologies and Techniques. International Journal of Computer Applications 50, 7 (2012).
[13]
Eamonn Keogh and Chotirat Ann Ratanamahatana. 2005. Exact indexing of dynamic time warping. Knowledge and information systems 7, 3 (2005), 358--386.
[14]
Sven Kratz and Michael Rohs. 2010. The $3 recognizer: simple 3D gesture recognition on mobile devices. In Proceedings of the 15th international conference on Intelligent user interfaces. ACM, 419--420.
[15]
Sven Kratz and Michael Rohs. 2011. Protractor3D: a closed-form solution to rotation-invariant 3D gestures. In Proceedings of the 16th international conference on Intelligent User Interfaces. ACM, 371--374.
[16]
Joseph J LaViola. 2013. 3d gestural interaction: The state of the field. International Scholarly Research Notices 2013 (2013).
[17]
Yang Li. 2010. Protractor: A Fast and Accurate Gesture Recognizer. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '10). ACM, New York, NY, USA, 2169--2172.
[18]
Jiayang Liu, Lin Zhong, Jehan Wickramasuriya, and Venu Vasudevan. 2009. uWave: Accelerometer-based personalized gesture recognition and its applications. Pervasive and Mobile Computing 5, 6 (2009), 657--675. PerCom 2009.
[19]
Xiao-Hui Liu and Chin-Seng Chua. 2010. Rejection of non-meaningful activities for HMM-based activity recognition system. Image and Vision Computing 28, 6 (2010), 865--871.
[20]
Robert Macrae and Simon Dixon. 2010. Accurate real-time windowed time warping. In in ISMIR, 2010. 423--428.
[21]
S. Mitra and T. Acharya. 2007. Gesture Recognition: A Survey. IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews) 37, 3 (May 2007), 311--324.
[22]
Corey Pittman, Pamela Wisniewski, Conner Brooks, and Joseph J. LaViola Jr. 2016. Multiwave: Doppler Effect Based Gesture Recognition in Multiple Dimensions. In Proceedings of the 2016 CHI Conference Extended Abstracts on Human Factors in Computing Systems. ACM, 1729--1736.
[23]
Thanawin Rakthanmanon, Bilson Campana, Abdullah Mueen, Gustavo Batista, Brandon Westover, Qiang Zhu, Jesin Zakaria, and Eamonn Keogh. 2012. Searching and mining trillions of time series subsequences under dynamic time warping. In Proceedings of the 18th ACM SIGKDD international conference on Knowledge discovery and data mining. ACM, 262--270.
[24]
Chotirat Ann Ratanamahatana and Eamonn Keogh. 2004. Everything you know about dynamic time warping is wrong. In Third Workshop on Mining Temporal and Sequential Data.
[25]
Siddharth S. Rautaray and Anupam Agrawal. 2015. Vision based hand gesture recognition for human computer interaction: a survey. Artificial Intelligence Review 43, 1 (2015), 1--54.
[26]
Douglas A Reynolds, Thomas F Quatieri, and Robert B Dunn. 2000. Speaker verification using adapted Gaussian mixture models. Digital signal processing 10, 1 (2000), 19--41.
[27]
Wenjie Ruan, Quan Z Sheng, Lei Yang, Tao Gu, Peipei Xu, and Longfei Shangguan. 2016. AudioGest: enabling fine-grained hand gesture detection by decoding echo signal. In Proceedings of the 2016 ACM International Joint Conference on Pervasive and Ubiquitous Computing. ACM, 474--485.
[28]
Dean Rubine. 1991. Specifying Gestures by Example. SIGGRAPH Computer Graphics 25, 4 (July 1991), 329--337.
[29]
Hiroaki Sakoe and Seibi Chiba. 1978. Dynamic programming algorithm optimization for spoken word recognition. IEEE transactions on acoustics, speech, and signal processing 26, 1 (1978), 43--49.
[30]
Arpita R. Sarkar, G. Sanyal, and S. Majumder. 2013. Hand Gesture Recognition Systems: A Survey. International Journal of Computer Applications 71, 15 (2013).
[31]
J. Suarez and R. R. Murphy. 2012. Hand gesture recognition with depth images: A review. In 2012 IEEE RO-MAN: The 21st IEEE International Symposium on Robot and Human Interactive Communication. 411--417.
[32]
Eugene M. Taranta, II, Mehran Maghoumi, Corey R. Pittman, and Joseph J. LaViola, Jr. 2016. A Rapid Prototyping Approach to Synthetic Data Generation for Improved 2D Gesture Recognition. In Proceedings of the 29th Annual Symposium on User Interface Software and Technology (UIST '16). ACM, New York, NY, USA, 873--885.
[33]
Eugene M. Taranta II, Andrés N. Vargas, and Joseph J. LaViola Jr. 2016. Streamlined and accurate gesture recognition with Penny Pincher. Computers & Graphics 55 (2016), 130 -- 142.
[34]
Radu-Daniel Vatavu. 2011. The Effect of Sampling Rate on the Performance of Template-based Gesture Recognizers. In Proceedings of the 13th International Conference on Multimodal Interfaces (ICMI '11). ACM, New York, NY, USA, 271--278.
[35]
Radu-Daniel Vatavu, Lisa Anthony, and Jacob O. Wobbrock. 2012. Gestures As Point Clouds: A $P Recognizer for User Interface Prototypes. In Proceedings of the 14th ACM International Conference on Multimodal Interaction (ICMI '12). ACM, New York, NY, USA, 273--280.
[36]
Sharad Vikram, Lei Li, and Stuart Russell. 2013. Writing and Sketching in the Air, Recognizing and Controlling on the Fly. In CHI '13 Extended Abstracts on Human Factors in Computing Systems (CHI EA '13). ACM, New York, NY, USA, 1179--1184.
[37]
Jacob O. Wobbrock, Leah Findlater, Darren Gergle, and James J. Higgins. 2011. The Aligned Rank Transform for Nonparametric Factorial Analyses Using Only Anova Procedures. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '11). ACM, New York, NY, USA, 143--146.
[38]
Jacob O. Wobbrock, Andrew D. Wilson, and Yang Li. 2007. Gestures Without Libraries, Toolkits or Training: A $1 Recognizer for User Interface Prototypes. In Proceedings of the 20th Annual ACM Symposium on User Interface Software and Technology (UIST '07). ACM, New York, NY, USA, 159--168.
[39]
J. Wu, J. Konrad, and P. Ishwar. 2013. Dynamic time warping for gesture-based user identification and authentication with Kinect. In 2013 IEEE International Conference on Acoustics, Speech and Signal Processing. 2371--2375.
[40]
Yaodong Zhang and James R Glass. 2011. An inner-product lower-bound estimate for dynamic time warping. In 2011 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 5660--5663.

Cited By

View all
  • (2025)Body Language Between Humans and MachinesBody Language Communication10.1007/978-3-031-70064-4_18(443-476)Online publication date: 2-Jan-2025
  • (2024)Take a Seat, Make a Gesture: Charting User Preferences for On-Chair and From-Chair Gesture InputProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642028(1-17)Online publication date: 11-May-2024
  • (2024)Analysis of User-Defined Radar-Based Hand Gestures Sensed Through Multiple MaterialsIEEE Access10.1109/ACCESS.2024.336666712(27895-27917)Online publication date: 2024
  • Show More Cited By

Index Terms

  1. Jackknife: A Reliable Recognizer with Few Samples and Many Modalities

      Recommendations

      Comments

      Information & Contributors

      Information

      Published In

      cover image ACM Conferences
      CHI '17: Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems
      May 2017
      7138 pages
      ISBN:9781450346559
      DOI:10.1145/3025453
      Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

      Sponsors

      Publisher

      Association for Computing Machinery

      New York, NY, United States

      Publication History

      Published: 02 May 2017

      Permissions

      Request permissions for this article.

      Check for updates

      Badges

      • Honorable Mention

      Author Tags

      1. dynamic time warping
      2. gesture customization
      3. gesture recognition
      4. rapid prototyping
      5. user evaluation

      Qualifiers

      • Research-article

      Conference

      CHI '17
      Sponsor:

      Acceptance Rates

      CHI '17 Paper Acceptance Rate 600 of 2,400 submissions, 25%;
      Overall Acceptance Rate 6,199 of 26,314 submissions, 24%

      Upcoming Conference

      CHI 2025
      ACM CHI Conference on Human Factors in Computing Systems
      April 26 - May 1, 2025
      Yokohama , Japan

      Contributors

      Other Metrics

      Bibliometrics & Citations

      Bibliometrics

      Article Metrics

      • Downloads (Last 12 months)41
      • Downloads (Last 6 weeks)2
      Reflects downloads up to 13 Feb 2025

      Other Metrics

      Citations

      Cited By

      View all
      • (2025)Body Language Between Humans and MachinesBody Language Communication10.1007/978-3-031-70064-4_18(443-476)Online publication date: 2-Jan-2025
      • (2024)Take a Seat, Make a Gesture: Charting User Preferences for On-Chair and From-Chair Gesture InputProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642028(1-17)Online publication date: 11-May-2024
      • (2024)Analysis of User-Defined Radar-Based Hand Gestures Sensed Through Multiple MaterialsIEEE Access10.1109/ACCESS.2024.336666712(27895-27917)Online publication date: 2024
      • (2023)RadarSense: Accurate Recognition of Mid-air Hand Gestures with Radar Sensing and Few Training ExamplesACM Transactions on Interactive Intelligent Systems10.1145/358964513:3(1-45)Online publication date: 11-Sep-2023
      • (2023)GestureCanvas: A Programming by Demonstration System for Prototyping Compound Freehand Interaction in VRProceedings of the 36th Annual ACM Symposium on User Interface Software and Technology10.1145/3586183.3606736(1-17)Online publication date: 29-Oct-2023
      • (2023)Designing Efficient and Customizable Radar-based Gesture InterfacesCompanion Proceedings of the 28th International Conference on Intelligent User Interfaces10.1145/3581754.3584105(208-210)Online publication date: 27-Mar-2023
      • (2023)Effective 2D Stroke-based Gesture Augmentation for RNNsProceedings of the 2023 CHI Conference on Human Factors in Computing Systems10.1145/3544548.3581358(1-13)Online publication date: 19-Apr-2023
      • (2023)iFAD Gestures: Understanding Users’ Gesture Input Performance with Index-Finger Augmentation DevicesProceedings of the 2023 CHI Conference on Human Factors in Computing Systems10.1145/3544548.3580928(1-17)Online publication date: 19-Apr-2023
      • (2023)Gesture‐Based ComputingHandbook of Human‐Machine Systems10.1002/9781119863663.ch32(397-408)Online publication date: 7-Jul-2023
      • (2022)Theoretically-Defined vs. User-Defined Squeeze GesturesProceedings of the ACM on Human-Computer Interaction10.1145/35678056:ISS(73-102)Online publication date: 14-Nov-2022
      • Show More Cited By

      View Options

      Login options

      View options

      PDF

      View or Download as a PDF file.

      PDF

      eReader

      View online with eReader.

      eReader

      Figures

      Tables

      Media

      Share

      Share

      Share this Publication link

      Share on social media