Abstract
Regaining functional independence plays a crucial role to improve the qualify of life of individuals with motor disabilities. Here, we address this problem within the framework of Body-Machine Interfaces (BoMIs). BoMIs enable individuals with restricted mobility to extend their capabilities by mapping their residual body movements into commands to control an external device. In this study, we propose a video-based marker-less interface that can track the position of the shoulders and the head using a state-of-the-art approach relying on the DeepLabCut (DLC) architecture. The high-dimensional body signal is then mapped into a lower dimensional space via non-linear variational autoencoder to obtain commands for a 2D computer cursor. First, we perform an offline test to evaluate the prediction power of the DLC fine tuned model. Then, we verify whether the proposed pipeline can be used to control a computer cursor in real-time. Results showed that the network can accurately predict the position of body landmarks. Moreover, an unimpaired participant was able to efficiently operate the computer cursor and gain a high-level of control skill after training with the interface. This enables performing experiments with video-based marker-less BoMIs for future implementation of an assistive device for people with motor disabilities.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Betke, M., Gips, J., Fleming, P.: The camera mouse: visual tracking of body features to provide computer access for people with severe disabilities. IEEE Trans. Neural Syst. Rehabil. Eng. 10(1), 1–10 (2002)
Casadio, M., et al.: Functional reorganization of upper-body movement after spinal cord injury. Exp. Brain Res. 207(3–4), 233–247 (2010)
Casadio, M., Ranganathan, R., Mussa-Ivaldi, F.A.: The body-machine interface: a new perspective on an old theme. J. Motor Behav. 44(6), 419–433 (2012)
Chen, Y.L., Tang, F.T., Chang, W.H., Wong, M.K., Shih, Y.Y., Kuo, T.S.: The new design of an infrared-controlled human-computer interface for the disabled. IEEE Trans. Rehabil. Eng. 7(4), 474–481 (1999)
Colyer, S.L., Evans, M., Cosker, D.P., Salo, A.I.: A review of the evolution of vision-based motion analysis and the integration of advanced computer vision methods towards developing a markerless system. Sports Med.-open 4(1), 24 (2018)
Cook, A.M., Polgar, J.M.: Assistive Technologies-E-Book: Principles and Practice. Elsevier Health Sciences (2014)
Di Mattia, P.A., Curran, F.X., Gips, J.: An eye control teaching device for students without language expressive capacity: EagleEyes, vol. 53. Edwin Mellen Press (2001)
Fu, Y., Huang, T.S.: hmouse: head tracking driven virtual computer mouse. In: 2007 IEEE Workshop on Applications of Computer Vision (WACV’07), pp. 30–30. IEEE (2007)
Grossi, G., Lanzarotti, R., Napoletano, P., Noceti, N., Odone, F.: Positive technology for elderly well-being: a review. Pattern Recogn. Lett. 137, 61–70 (2020)
Higgins, I., et al.: beta-vae: learning basic visual concepts with a constrained variational framework (2016)
Javanovic, R., MacKenzie, I.S.: MarkerMouse: mouse cursor control using a head-mounted marker. In: Miesenberger, K., Klaus, J., Zagler, W., Karshmer, A. (eds.) ICCHP 2010. LNCS, vol. 6180, pp. 49–56. Springer, Heidelberg (2010). https://doi.org/10.1007/978-3-642-14100-3_9
Jeong, H., Kim, J.S., Son, W.H.: An emg-based mouse controller for a tetraplegic. In: 2005 IEEE International Conference on Systems, Man and Cybernetics, vol. 2, pp. 1229–1234. IEEE (2005)
Kane, G., Lopes, G., Sanders, J., Mathis, A., Mathis, M.: Real-time, low-latency closed-loop feedback using markerless posture tracking. BioRxiv (2020)
Kim, S., Park, M., Anumas, S., Yoo, J.: Head mouse system based on gyro-and opto-sensors. In: 2010 3rd International Conference on Biomedical Engineering and Informatics, vol. 4, pp. 1503–1506. IEEE (2010)
Kingma, D.P., Welling, M.: Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114 (2013)
Mathis, A., et al.: Deeplabcut: markerless pose estimation of user-defined body parts with deep learning. Nat. Neurosci. 21(9), 1281–1289 (2018)
Meihlbradt, J., et al.: Data-driven body-machine interface for the accurate control of drones. Proc. Natl. Acad. Sci. 115(31), 7913–7918 (2018)
Moro, M., Marchesi, G., Odone, F., Casadio, M.: Markerless gait analysis in stroke survivors based on computer vision and deep learning: a pilot study. In: Proceedings of the 35th Annual ACM Symposium on Applied Computing, pp. 2097–2104 (2020)
Pierella, C., et al.: Learning new movements after paralysis: results from a home-based study. Sci. Rep. 7(1), 1–11 (2017)
Rizzoglio, F., Pierella, C., De Santis, D., Mussa-Ivaldi, F.A., Casadio, M.: Ahybrid body-machine interface integrating signals from muscles and motions. J. Neural Eng. (2020)
Thorp, E.B., et al.: Upper body-based power wheelchair control interface for individuals with tetraplegia. IEEE Tans. Neural Syst. Rehabil. Eng. 24(2), 249–260 (2015)
Zhou, H., Hu, H.: Human motion tracking for rehabilitation–a survey. Biomed. Signal Process. Control 3(1), 1–18 (2008)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2021 Springer Nature Switzerland AG
About this paper
Cite this paper
Moro, M., Rizzoglio, F., Odone, F., Casadio, M. (2021). A Video-Based MarkerLess Body Machine Interface: A Pilot Study. In: Del Bimbo, A., et al. Pattern Recognition. ICPR International Workshops and Challenges. ICPR 2021. Lecture Notes in Computer Science(), vol 12662. Springer, Cham. https://doi.org/10.1007/978-3-030-68790-8_19
Download citation
DOI: https://doi.org/10.1007/978-3-030-68790-8_19
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-68789-2
Online ISBN: 978-3-030-68790-8
eBook Packages: Computer ScienceComputer Science (R0)
