Abstract
To enhance users’ experience of inputting characters on mobile devices with small screens, this paper designed a novel virtual keyboard used on mobile devices. In particular, we introduce a novel virtual keyboard based on the MYO armband which is able to capture the electromyogram (EMG) signals of users when typing on any surfaces (such as human body or normal desktop). The actions of the three fingers are mapped to the nine keys of the T9 keyboard. After that, the signals of finger motions are translated into key sequences of the T9 keyboard. However, the identification of continuous finger motions is a critical challenge. To address the challenge, we convert the EMG signals in time domain into a 3D time-frequency map (each channel corresponds to the EMG unit of a frequency-domain feature), and extract the convolutional features with a 4-layer CNN (Convolutional Neural Network) module, an im2col module of Optical Character Recognition (OCR) and a Long Short-Term Memory (LSTM) module, and the final result is achieved as a probability graph of finger gestures. The Connection Temporal Classification (CTC) algorithm is adopted to find the best gesture sequence from the probability map. Experimental results show that our method can effectively identify different key sequences at three different input speeds with an average accuracy of 85.9%, and the integration testing with different volunteers shows that our method can achieve an average typing speed of 15.7 Word-Per-Minute (WPM).
Supported by Hangzhou Innovation Institution, Beihang University.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Barry, D.T., Gordon, K.E., Hinton, G.G.: Acoustic and surface EMG diagnosis of pediatric muscle disease. Muscle Nerve Off. J. Am. Assoc. Electrodiagn. Med. 13(4), 286–290 (1990)
Hernandez Arieta, A., Katoh, R., Yokoi, H., Wenwei, Y.: Development of a multi-DOF electromyography prosthetic system using the adaptive joint mechanism. Appl. Bionics Biomech. 3(2), 101–111 (2006)
Saponas, T.S., Tan, D.S., Morris, D., Balakrishnan, R., Turner, J., Landay, J.A.: Enabling always-available input with muscle-computer interfaces. In: Proceedings of the 22nd Annual ACM Symposium on User Interface Software and Technology, pp. 167–176. ACM (2009)
Saponas, T.S., Tan, D.S., Morris, D., Balakrishnan, R.: Demonstrating the feasibility of using forearm electromyography for muscle-computer interfaces. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pp. 515–524. ACM (2008)
Li, Y., Chen, X., Tian, J., Zhang, X., Wang, K., Yang, J.: Automatic recognition of sign language subwords based on portable accelerometer and EMG sensors. In: International Conference on Multimodal Interfaces and the Workshop on Machine Learning for Multimodal Interaction, p. 17. ACM (2010)
Wu, J., Tian, Z., Sun, L., Estevez, L., Jafari, R.: Real-time American sign language recognition using wrist-worn motion and surface EMG sensors. In: 2015 IEEE 12th International Conference on Wearable and Implantable Body Sensor Networks (BSN), pp. 1–6. IEEE (2015)
Hudgins, B., Parker, P., Scott, R.N.: A new strategy for multifunction myoelectric control. IEEE Trans. Biomed. Eng. 40(1), 82–94 (1993)
Farry, K.A., Walker, I.D., Baraniuk, R.G.: Myoelectric teleoperation of a complex robotic hand. IEEE Trans. Robot. Autom. 12(5), 775–788 (1996)
Englehart, K., Hudgins, B., Stevenson, M., Parker, P.A.: Classification of transient myoelectric signals using a dynamic feedforward neural network. In: Proceedings World Congress Neural Networks (1995)
Englehart, K., Hudgins, B., Parker, P.A., et al.: A wavelet-based continuous classification scheme for multifunction myoelectric control. IEEE Trans. Biomed. Eng. 48(3), 302–311 (2001)
Phinyomark, A., Phukpattaranont, P., Limsakul, C.: Feature reduction and selection for EMG signal classification. Expert Syst. Appl. 39(8), 7420–7431 (2012)
Kunapipat, M., Phukpattaranont, P., Neranon, P., Thongpull, K.: Sensor-assisted emg data recording system. In: 2018 15th International Conference on Electrical Engineering/Electronics, Computer, Telecommunications and Information Technology (ECTI-CON), pp. 772–775. IEEE (2018)
Kosmidou, V.E., Hadjileontiadis, L.J., Panas, S.M.: Evaluation of surface EMG features for the recognition of American sign language gestures. In: 2006 International Conference of the IEEE Engineering in Medicine and Biology Society, pp. 6197–6200. IEEE (2006)
Park, K.-H., Lee, S.-W.: Movement intention decoding based on deep learning for multiuser myoelectric interfaces. In: 2016 4th International Winter Conference on Brain-Computer Interface (BCI), pp. 1–2. IEEE (2016)
Becker, V., Oldrati, P., Barrios, L., Sörös, G.: Touchsense: classifying finger touches and measuring their force with an electromyography armband. In: Proceedings of the 2018 ACM International Symposium on Wearable Computers, pp. 1–8. ACM (2018)
Atzori, M., Cognolato, M., Müller, H.: Deep learning with convolutional neural networks applied to electromyography data: a resource for the classification of movements for prosthetic hands. Front. Neurorobotics 10, 9 (2016)
He, Y., Fukuda, O., Bu, N., Okumura, H., Yamaguchi, N.: Surface EMG pattern recognition using long short-term memory combined with multilayer perceptron. In: 2018 40th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC), pp. 5636–5639. IEEE (2018)
Yiqin, L., Chun, Yu., Yi, X., Shi, Y., Zhao, S.: BlindType: eyes-free text entry on handheld touchpad by leveraging thumb’s muscle memory. Proc. ACM Interact. Mob. Wearable Ubiquit. Technol. 1(2), 18 (2017)
Xu, Z. et al.: TipText: eyes-free text entry on a fingertip keyboard (2019)
Yi, X., Yu, C., Xu, W., Bi, X., Shi, Y.: COMPASS: rotational keyboard on non-touch smartwatches. In: Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems, pp. 705–715. ACM (2017)
Dhillon, H.S., Singla, R., Rekhi, N.S., Jha, R.: EOG and EMG based virtual keyboard: a brain-computer interface. In: 2009 2nd IEEE International Conference on Computer Science and Information Technology, pp. 259–262. IEEE (2009)
Testa, A., Gallo, D., Langella, R.: On the processing of harmonics and interharmonics: using Hanning window in standard framework. IEEE Trans. Power Deliv. 19(1), 28–34 (2004)
LeCun, Y., Bottou, L., Bengio, Y., Haffner, P., et al.: Gradient-based learning applied to document recognition. Proc. IEEE 86(11), 2278–2324 (1998)
Graves, A., Fernández, S., Gomez, F., Schmidhuber, J.: Connectionist temporal classification: labelling unsegmented sequence data with recurrent neural networks. In: Proceedings of the 23rd International Conference on Machine Learning, pp. 369–376. ACM (2006)
Acknowledgment
This work has been supported by National Natural Science Foundation of China (61772060, 61976012, 61602024), Qianjiang Postdoctoral Foundation (2020-Y4-A-001), and CERNET Innovation Project (NGII20170315).
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2020 Springer Nature Switzerland AG
About this paper
Cite this paper
Fu, Z., Li, H., Ouyang, Z., Liu, X., Niu, J. (2020). Typing Everywhere with an EMG Keyboard: A Novel Myo Armband-Based HCI Tool. In: Qiu, M. (eds) Algorithms and Architectures for Parallel Processing. ICA3PP 2020. Lecture Notes in Computer Science(), vol 12452. Springer, Cham. https://doi.org/10.1007/978-3-030-60245-1_17
Download citation
DOI: https://doi.org/10.1007/978-3-030-60245-1_17
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-60244-4
Online ISBN: 978-3-030-60245-1
eBook Packages: Mathematics and StatisticsMathematics and Statistics (R0)