Skip to main content
SpringerLink
Log in
Book cover

International Conference on Human-Computer Interaction

HCII 2020: Digital Human Modeling and Applications in Health, Safety, Ergonomics and Risk Management. Posture, Motion and Health pp 335–348Cite as

End-User Programming Architecture for Physical Movement Assessment: An Interactive Machine Learning Approach

  • Conference paper
  • First Online:

Part of the book series: Lecture Notes in Computer Science ((LNISA,volume 12198))

Abstract

In this article, we propose an end-user adaptive architecture for movement assessment from RGB videos. Our method allows physiotherapists to add customized exercises for patients from only a few video training examples. The main idea is to take leverage of Deep learning-based pose estimation frameworks to track in real-time the key-body joints from the image data. Our system mimics the traditional physical rehabilitation process, where the therapist guides patients through demonstrative examples, and the patients repeat these examples while the physiotherapist monitors their movements. We evaluate our proposed method on four physiotherapeutic exercises for shoulder strengthening. Results indicate that our approach contributes both to reduce physiotherapist time needed to train the system, and to automatically assess the patients’ movements without direct monitoring from the physiotherapist.

This is a preview of subscription content, log in via an institution.

Chapter
USD   29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD   39.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD   54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Learn about institutional subscriptions

References

  1. Afyouni, I., et al.: A therapy-driven gamification framework for hand rehabilitation. User Model. User-Adap. Interact. 27(2), 215–265 (2017). https://doi.org/10.1007/s11257-017-9191-4. http://link.springer.com/10.1007/s11257-017-9191-4

    Article  Google Scholar 

  2. American Physical Therapy Association: Interactive guide to physical therapist practice (2019). http://guidetoptpractice.apta.org/. Accessed 08 Oct 2019

  3. Amershi, S., Cakmak, M., Knox, W.B., Kulesza, T.: Power to the people: the role of humans in interactive machine learning. AI Mag. 35(4), 105–120 (2014)

    Article  Google Scholar 

  4. Barricelli, B.R., Cassano, F., Fogli, D., Piccinno, A.: End-user development, end-user programming and end-user software engineering: a systematic mapping study. J. Syst. Softw. 149, 101–137 (2019)

    Article  Google Scholar 

  5. Braz, P., Felipe David, V., Raposo, A., Barbosa, S.D.J., de Souza, C.S.: An alternative design perspective for technology supporting youngsters with autism. In: Schmorrow, D.D., Fidopiastis, C.M. (eds.) AC 2014. LNCS (LNAI), vol. 8534, pp. 279–287. Springer, Cham (2014). https://doi.org/10.1007/978-3-319-07527-3_26

    Chapter  Google Scholar 

  6. Cao, Z., Hidalgo, G., Simon, T., Wei, S.E., Sheikh, Y.: Openpose: realtime multi-person 2d pose estimation using part affinity fields (2018). arXiv preprint arXiv:1812.08008

  7. Capecci, M., et al.: Physical rehabilitation exercises assessment based on hidden semi-markov model by kinect v2. In: 2016 IEEE-EMBS International Conference on Biomedical and Health Informatics (BHI). pp. 256–259. IEEE (2016)

    Google Scholar 

  8. Celebi, S., Aydin, A.S., Temiz, T.T., Arici, T.: Gesture recognition using skeleton data with weighted dynamic time warping. In: VISAPP no. 1, pp. 620–625 (2013)

    Google Scholar 

  9. Chen, S., Yang, R.: Pose trainer: correcting exercise posture using pose estimation (2018)

    Google Scholar 

  10. Cypher, A., Halbert, D.C.: Watch What I Do: Programming by Demonstration. MIT press, Cambridge (1993)

    Google Scholar 

  11. Da Silva, M.L., Gonçalves, D., Silva, H.: User-tuned content customization for children with Autism Spectrum Disorders. In: Procedia Computer Science (2013). https://doi.org/10.1016/j.procs.2014.02.048

  12. Devanne, M., Remy-Neris, O., Le Gals-Garnett, B., Kermarrec, G., Thepaut, A., et al.: A co-design approach for a rehabilitation robot coach for physical rehabilitation based on the error classification of motion errors. In: 2018 Second IEEE International Conference on Robotic Computing (IRC), pp. 352–357. IEEE (2018)

    Google Scholar 

  13. Garzotto, F., Gonella, R.: An open-ended tangible environment for disabled children’s learning. In: Proceedings of the 10th International Conference on Interaction Design and Children, pp. 52–61. ACM (2011)

    Google Scholar 

  14. Godse, S.P., Singh, S., Khule, S., Yadav, V., Wakhare, S.: Musculoskeletal physiotherapy using artificial intelligence and machine learning. Int. J. Innov. Sci. Res. Technol. 4(11), 592–598 (2019)

    Google Scholar 

  15. Görer, B., Salah, A.A., Akın, H.L.: A robotic fitness coach for the elderly. In: Augusto, J.C., Wichert, R., Collier, R., Keyson, D., Salah, A.A., Tan, A.-H. (eds.) AmI 2013. LNCS, vol. 8309, pp. 124–139. Springer, Cham (2013). https://doi.org/10.1007/978-3-319-03647-2_9

    Chapter  Google Scholar 

  16. Hamzabegovic, J., Kalpić, D.: A proposal for development of software to support specific learning difficulties. In: Proceedings of the 12th International Conference on Telecommunications, pp. 207–214. IEEE (2013)

    Google Scholar 

  17. Houmanfar, R., Karg, M., Kulić, D.: Movement analysis of rehabilitation exercises: Distance metrics for measuring patient progress. IEEE Syst. J. 10(3), 1014–1025 (2014)

    Article  Google Scholar 

  18. Jain, N.K., Saini, R.K., Mittal, P.: A review on traffic monitoring system techniques. In: Ray, K., Sharma, T.K., Rawat, S., Saini, R.K., Bandyopadhyay, A. (eds.) Soft Computing: Theories and Applications. AISC, vol. 742, pp. 569–577. Springer, Singapore (2019). https://doi.org/10.1007/978-981-13-0589-4_53

    Chapter  Google Scholar 

  19. Kojima, H., Kitano, M., Yokota, K., Ooi, S., Sano, M.: Cleaning behavior estimation for self-supported cognitive rehabilitation system. In: 2018 IEEE International Conference on Artificial Intelligence in Engineering and Technology (IICAIET), pp. 1–6. IEEE (2018)

    Google Scholar 

  20. Lieberman, H.: Your Wish is My Command: Programming by Example. Morgan Kaufmann, Burlington (2001)

    Google Scholar 

  21. Liu, F.T., Ting, K.M., Zhou, Z.H.: Isolation forest. In: 2008 Eighth IEEE International Conference on Data Mining, pp. 413–422. IEEE (2008)

    Google Scholar 

  22. Liu, Z., Liu, X., Ma, J., Gao, H.: An optimized computational framework for isolation forest. Math. Prob. Eng. 2018 (2018)

    Google Scholar 

  23. Maat, R. and Malali, A.: A Multipurpose Library for Synthetic Time Series in Python (2019). https://github.com/TimeSynth/TimeSynth. Accessed 09 Oct 2019

  24. Mohammadi, S., Perina, A., Kiani, H., Murino, V.: Angry crowds: detecting violent events in videos. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9911, pp. 3–18. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-46478-7_1

    Chapter  Google Scholar 

  25. Ross, M.K., Broz, F., Baillie, L.: Towards an adaptive robot for sports and rehabilitation coaching (2019). arXiv preprint arXiv:1909.08052

  26. Rousseeuw, P.J., Hubert, M.: Anomaly detection by robust statistics. Wiley Interdisc. Rev. Data Min. Knowl. Disc. 8(2), e1236 (2018)

    Google Scholar 

  27. Saha, S., Pal, M., Konar, A., Janarthanan, R.: Neural network based gesture recognition for elderly health care using kinect sensor. In: Panigrahi, B.K., Suganthan, P.N., Das, S., Dash, S.S. (eds.) SEMCCO 2013. LNCS, vol. 8298, pp. 376–386. Springer, Cham (2013). https://doi.org/10.1007/978-3-319-03756-1_34

    Chapter  Google Scholar 

  28. Schneider, P., Memmesheimer, R., Kramer, I., Paulus, D.: Gesture recognition in RGB videos usinghuman body keypoints and dynamic time warping (2019). arXiv preprint arXiv:1906.12171

  29. Su, C.J., Chiang, C.Y., Huang, J.Y.: Kinect-enabled home-based rehabilitation system using dynamic time warping and fuzzy logic. Appl. Soft Comput. 22, 652–666 (2014)

    Article  Google Scholar 

  30. Sultani, W., Chen, C., Shah, M.: Real-world anomaly detection in surveillance videos. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 6479–6488 (2018)

    Google Scholar 

  31. Tanguy, P., Rémy-Néris, O., et al.: Computational architecture of a robot coach for physical exercises in kinaesthetic rehabilitation. In: 2016 25th IEEE International Symposium on Robot and Human Interactive Communication (RO-MAN), pp. 1138–1143. IEEE (2016)

    Google Scholar 

  32. Taranta II, E.M., Maghoumi, M., Pittman, C.R., LaViola Jr, J.J.: A rapid prototyping approach to synthetic data generation for improved 2d gesture recognition. In: Proceedings of the 29th Annual Symposium on User Interface Software and Technology, pp. 873–885. ACM (2016)

    Google Scholar 

  33. Tetteroo, D., et al.: Lessons learnt from deploying an end-user development platform for physical rehabilitation. In: Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems - CHI 2015, pp. 4133–4142. ACM Press, New York (2015). https://doi.org/10.1145/2702123.2702504, http://dl.acm.org/citation.cfm?doid=2702123.2702504

  34. Tukey, J.W.: Exploratory Data Analysis, vol. 2. Reading, Mass (1977)

    MATH  Google Scholar 

  35. Velloso, E., Bulling, A., Gellersen, H.: Motionma: motion modelling and analysis by demonstration. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pp. 1309–1318. ACM (2013)

    Google Scholar 

  36. Vox, J.P., Wallhoff, F.: Recognition of human motion exercises using skeleton data and SVM for rehabilitative purposes. In: 2017 IEEE Life Sciences Conference (LSC), pp. 266–269. IEEE (2017)

    Google Scholar 

  37. Wang, Z., Ding, Z.: Rehabilitation system for children with cerebral palsy based on body vector analysis and gmfm-66 standard. J. Phys. Conf. Ser. 1325(1), 012088 (2019)

    Google Scholar 

  38. World Health Organization: Rehabilitation. In: World Report on Disability, chap. 4, p. 350. World Health Organization (2011). https://www.who.int/disabilities/world_report/2011/chapter4.pdf

  39. World Health Organization: WHO — Rehabilitation in health systems. In: WHO, World Health Organization (2019). http://www.who.int/disabilities/rehabilitation_health_systems/en/

  40. Zhao, W., Reinthal, M.A., Espy, D.D., Luo, X.: Rule-based human motion tracking for rehabilitation exercises: realtime assessment, feedback, and guidance. IEEE Access 5, 21382–21394 (2017)

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of Informatics, Pontifical Catholic University of Rio de Janeiro (PUC-Rio), Gávea, Rio de Janeiro, 22451-900, Brazil

    Jessica M. Palomares-Pecho & Alberto Barbosa Raposo

  2. Tecgraf Institute, Pontifical Catholic University of Rio de Janeiro (PUC-Rio), Gávea, Rio de Janeiro, 22451-900, Brazil

    Greis Francy M. Silva-Calpa, César A. Sierra-Franco & Alberto Barbosa Raposo

Authors
  1. Jessica M. Palomares-Pecho
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Greis Francy M. Silva-Calpa
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. César A. Sierra-Franco
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Alberto Barbosa Raposo
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding authors

Correspondence to Jessica M. Palomares-Pecho , Greis Francy M. Silva-Calpa , César A. Sierra-Franco or Alberto Barbosa Raposo .

Editor information

Editors and Affiliations

  1. Purdue University, West Lafayette, IN, USA

    Vincent G. Duffy

Rights and permissions

Reprints and permissions

Copyright information

© 2020 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Palomares-Pecho, J.M., Silva-Calpa, G.F.M., Sierra-Franco, C.A., Barbosa Raposo, A. (2020). End-User Programming Architecture for Physical Movement Assessment: An Interactive Machine Learning Approach. In: Duffy, V. (eds) Digital Human Modeling and Applications in Health, Safety, Ergonomics and Risk Management. Posture, Motion and Health. HCII 2020. Lecture Notes in Computer Science(), vol 12198. Springer, Cham. https://doi.org/10.1007/978-3-030-49904-4_25

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-49904-4_25

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-49903-7

  • Online ISBN: 978-3-030-49904-4

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation