Abstract
Hand gestures are the nature and dominant interaction interfaces for VR systems. The state of the art interaction mechanism for VR system either requires expensive sensing devices or suffers from accuracy issues thus hard to perform versatile interactions. In this paper, we leverage Ultragloves, a low cost interaction system using microphone-implanted gloves to extract the hand gestures. With specifically designed signals, we manage to get both the distance and the directions in a relatively accurate manner. We then design a CNN-LSTM like learning algorithm to extract the gestures. Furthermore, to improve the accuracy of recognition, we also design a filter algorithm to filter out noisy data. The implementation shows that our method can recognize four micro-gestures in the accuracy of 82% by combining phase and frequency features.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Becker, V., Fessler, L., Sörös, G.: Gestear: combining audio and motion sensing for gesture recognition on smartwatches. pp 10–19 (2019)
Byun, S.W., Lee, S.P.: Implementation of hand gesture recognition device applicable to smart watch based on flexible epidermal tactile sensor array. Micromachines 10, 692 (2019)
Dison Techology: Synertial. http://www.disonde.com/. (2019)
Feichtenhofer, C., Pinz, A., Zisserman, A.: Convolutional two-stream network fusion for video action recognition. (2016)
Han, H., Yoon, S.W.: Gyroscope-based continuous human hand gesture recognition for multi-modal wearable input device for human–machine interaction. Sensors (Basel, Switzerland) (2019)
Jiang, S., Zhu, Y., Zhu, X.: Hand gesture recognition and finger angle estimation via wrist-worn modified barometric pressure sensing. In: IEEE Transactions on Neural Systems and Rehabilitation Engineering, pp 1–1 (2019)
Kingma, D.P., Ba, J.: Adam: a method for stochastic optimization. (2014)
Köpüklü, O., Gunduz, A., Kose, N., Rigoll, G.: Real-time hand gesture detection and classification using convolutional neural networks. (2019)
Lecun, Y., Bottou, L., Bengio, Y., Haffner, P.: Gradient-based learning applied to document recognition. Proc. IEEE 86(11), 2278–2324 (1998)
Li, S., Zhao, Y., Liu, C.: Ultragloves: lowcost finger-level interaction system for vr-applications based on ultrasonic movement tracking. In: Wen, Sheng, Zomaya, Albert, Yang, Laurence T. (eds.) Algorithms and architectures for parallel processing. (2020)
Nandakumar, R., Iyer, V., Tan, D., Gollakota, S.: Fingerio: using active sonar for fine-grained finger tracking. New York, NY, USA (2016). Association for Computing Machinery
Priandana, K., Coccia, A., Ligthart, L.P.: Fmcw radar phase-processing for automotive application. (2015)
Schuster, M., Paliwal, K.K.: Bidirectional recurrent neural networks. IEEE Trans. Signal Process. 45(11), 2673–2681 (1997)
Sun, Ke, Zhao, Ting, Wang, Wei, Xie, Lei: Vskin: Sensing touch gestures on surfaces of mobile devices using acoustic signals, 591–605, (10 2018)
Sun, K., Wang, W., Liu, A.X., Dai, H.: Depth aware finger tapping on virtual displays. New York, NY, USA, (2018). Association for Computing Machinery
Tran, Dinh-Son, Tran, Dinh-Son, Ho, Ngoc-Huynh, Yang, Hyung-Jeong, Baek, Eu-Tteum, Kim, S.H., Lee, Guee: Real-time hand gesture spotting and recognition using rgb-d camera and 3d convolutional neural network. Appl. Sci. 10, 722 (2020)
Wang, W., Liu, A.X., Sun, K.: Device-free gesture tracking using acoustic signals. MobiCom ’16, New York, NY, USA, (2016). Association for Computing Machinery
Zhang, J., Tao, J., Huangfu, J., Shi, Z.: Doppler-radar based hand gesture recognition system using convolutional neural networks. (2017)
Zhang, C., Xue, Q., Waghmare, A., Jain, S., Yiming, P., Hersek, S., Lyons, K., Cunefare, K.A., Inan, O.T., Abowd, G.D.: Soundtrak: continuous 3d tracking of a finger using active acoustics. (2017)
Zhang, Z., Tian, Z., Zhou, M.: Latern: dynamic continuous hand gesture recognition using fmcw radar sensor. IEEE Sens. J. 18(8), 3278–3289 (2018)
Zhu, G., Zhang, L., Shen, P., Song, J., Shah, S.A.A., Bennamoun, M.: Continuous gesture segmentation and recognition using 3dcnn and convolutional lstm. IEEE Trans. Multimed. 21(4), 1011–1021 (2019)
Acknowledgements
This work was supported by the National Key Research and Development Program of China under Grant 2019YFB2102002.
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Conflict of interest
On behalf of all authors, the corresponding author states that there is no conflict of interest.
Rights and permissions
About this article
Cite this article
Jiang, W., Li, S., Zhao, Y. et al. Fine-grained hand gesture recognition based on active acoustic signal for VR systems. CCF Trans. Pervasive Comp. Interact. 2, 329–339 (2020). https://doi.org/10.1007/s42486-020-00048-w
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s42486-020-00048-w