Skip to main content
SpringerLink
Log in

Continues online exercise monitoring and assessment system with visual guidance feedback for stroke rehabilitation

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

Exercise therapy is a conventional intervention for stroke rehabilitation. Performance monitoring and feedback have shown to further improve the outcome of exercise therapy. This paper proposes a vision based system for monitoring exercise therapy which consists of 3 components: online exercise recognition, exercise performance analysis, and automatic visual feedback generation. The Microsoft Kinect was used for data acquisition. The exercise recognition component utilizes Kinect joints to continuously recognize and track the exercises. Upon completion of each exercise, joint flexibility and compensatory trunk motions are extracted for performance analysis. The visual feedback is a virtual skeleton augmented on top of the Kinect skeleton which displays the correct exercise path during execution. The Kinect skeleton and exercise definitions were applied to a motion hierarchy and animated using forward kinematics. Two additional experiments were also conducted to find accurate methods for calculating joint flexibility based on ROM measurement and trunk representation. Several datasets were created for system design and evaluation: 336 exercise sequences for exercise recognition, 25 records for ROM measurement, and 63 records for finding a suitable trunk representation method and compensatory motion detection. System evaluations showed that each component of the system is capable of producing outputs with significant accuracy.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15

Similar content being viewed by others

Notes

  1. doi.org/10.6084/m9.figshare.6741713

  2. https://figshare.com/s/9ab16ddb61c587b8cd97

References

  1. Abdollahi F, Case Lazarro ED, Listenberger M, Kenyon RV, Kovic M, Bogey RA, Hedeker D, Jovanovic BD, Patton JL (2014) Error augmentation enhancing arm recovery in individuals with chronic stroke: a randomized crossover design. Neurorehabil Neural Repair 28:120–128

    Article  Google Scholar 

  2. Aggarwal JK, Xia L (2014) Human activity recognition from 3d data: A review. Pattern Recogn Lett 48:70–80

    Article  Google Scholar 

  3. Ayoade M, Baillie L (2014) A novel knee rehabilitation system for the home. In: Proceedings of the 32nd annual ACM conference on Human factors in computing systems, 2014. ACM, pp 2521-2530

  4. Barnachon M, Bouakaz S, Boufama B, Guillou E (2014) Ongoing human action recognition with motion capture. Pattern Recogn 47(1):238–247

    Article  Google Scholar 

  5. Bloom V, Argyriou V, Makris D (2017) Linear latent low dimensional space for online early action recognition and prediction. Pattern Recogn 72:532–547

    Article  Google Scholar 

  6. Bonnechere B, Jansen B, Salvia P, Bouzahouene H, Omelina L, Moiseev F, Sholukha V, Cornelis J, Rooze M, Jan SVS (2014) Validity and reliability of the Kinect within functional assessment activities: comparison with standard stereophotogrammetry. Gait Posture 39:593–598

    Article  Google Scholar 

  7. Chandra H, Oakley I, Silva H (2012) Designing to support prescribed home exercises: understanding the needs of physiotherapy patients. In: Proceedings of the 7th Nordic Conference on Human-Computer Interaction: Making Sense Through Design. ACM, pp 607-616

  8. Chang Y-J, Chen S-F, Huang J-D (2011) A Kinect-based system for physical rehabilitation: A pilot study for young adults with motor disabilities. Res Dev Disabil 32:2566–2570

    Article  Google Scholar 

  9. Chang Y-J, Han W-Y, Tsai Y-C (2013) A Kinect-based upper limb rehabilitation system to assist people with cerebral palsy. Res Dev Disabil 34:3654–3659

    Article  Google Scholar 

  10. Chang X, Yu Y-L, Yang Y, Xing EP (2017) Semantic pooling for complex event analysis in untrimmed videos. IEEE Trans Pattern Anal Mach Intell 39(8):1617–1632

    Article  Google Scholar 

  11. Conner C, Poor GM (2016) Correcting exercise form using body tracking. In: Proceedings of the 2016 CHI Conference Extended Abstracts on Human Factors in Computing Systems, 2016. ACM, pp 3028-3034

  12. Conradsson D, Nero H, Löfgren N, Hagströmer M, Franzén E (2017) Monitoring training activity during gait-related balance exercise in individuals with Parkinson’s disease: a proof-of-concept-study. BMC Neurol 17:19

    Article  Google Scholar 

  13. Crasto JA, Sayari AJ, Gray RR, Askari M (2015) Comparative analysis of photograph-based clinical goniometry to standard techniques. Hand 10:248–253

    Article  Google Scholar 

  14. Da Gama A, Chaves T, Figueiredo L, Teichrieb V (2012) Guidance and movement correction based on therapeutics movements for motor rehabilitation support systems. In: Virtual and Augmented Reality (SVR), 2012 14th Symposium on, 2012. IEEE, pp 191-200

  15. Deters JK, Rybarczyk Y (2018) Hidden Markov Model approach for the assessment of tele-rehabilitation exercises. International Journal of Artificial Intelligence 16(1):1–19

    Google Scholar 

  16. English C, Bernhardt J, Crotty M, Esterman A, Segal L, Hillier S (2015) Circuit class therapy or seven-day week therapy for increasing rehabilitation intensity of therapy after stroke (CIRCIT): a randomized controlled trial. Int J Stroke 10:594–602

    Article  Google Scholar 

  17. Flanders M, Kavanagh RC (2015) Build-A-Robot: Using virtual reality to visualize the Denavit–Hartenberg parameters. Comput Appl Eng Educ 23:846–853

    Article  Google Scholar 

  18. Gajdosik RL, Bohannon RW (1987) Clinical measurement of range of motion: review of goniometry emphasizing reliability and validity. Phys Ther 67:1867–1872

    Article  Google Scholar 

  19. Gal N, Andrei D, Nemeş DI, Nădăşan E, Stoicu-Tivadar V (2015) A Kinect based intelligent e-rehabilitation system in physical therapy. Studies in Health Technology and Informatics 210:489–493

    Google Scholar 

  20. Gauthier LV, Kane C, Borstad A, Strahl N, Uswatte G, Taub E, Morris D, Hall A, Arakelian M, Mark V (2017) Video Game Rehabilitation for Outpatient Stroke (VIGoROUS): protocol for a multi-center comparative effectiveness trial of in-home gamified constraint-induced movement therapy for rehabilitation of chronic upper extremity hemiparesis. BMC Neurol 17:109

    Article  Google Scholar 

  21. Han F, Reily B, Hoff W, Zhang H (2017) Space-time representation of people based on 3D skeletal data: A review. Comput Vis Image Underst 158:85–105

    Article  Google Scholar 

  22. Hiraoka K (2001) Rehabilitation effort to improve upper extremity function in post-stroke patients: a meta-analysis. J Phys Ther Sci 13:5–9

    Article  Google Scholar 

  23. Hsieh C-L, Sheu C-F, Hsueh I-P, Wang C-H (2002) Trunk control as an early predictor of comprehensive activities of daily living function in stroke patients. Stroke 33:2626–2630

    Article  Google Scholar 

  24. Kim E, Kim K (2015) Effect of purposeful action observation on upper extremity function in stroke patients. J Phys Ther Sci 27:2867–2869

    Article  Google Scholar 

  25. Kitsunezaki N, Adachi E, Masuda T, Mizusawa J-I (2013) KINECT applications for the physical rehabilitation. In: Medical Measurements and Applications Proceedings (MeMeA), 2013 IEEE International Symposium on, 2013. IEEE, pp 294–299

  26. Lam MY, Tatla SK, Lohse KR, Shirzad N, Hoens AM, Miller KJ, Holsti L, Virji-Babul N, Van der Loos HM (2015) Perceptions of technology and its use for therapeutic application for individuals with hemiparesis: findings from adult and pediatric focus groups. JMIR Rehabilitation and Assistive Technologies 2

  27. Lam K-Y, Tsang NW-H, Han S, Zhang W, Ng JK-Y, Nath A (2017) Activity tracking and monitoring of patients with Alzheimer’s disease. Multimed Tools Appl 76:489–521

    Article  Google Scholar 

  28. Lam AW, Varona-Marin D, Li Y, Fergenbaum M, Kulić D (2016) Automated rehabilitation system: Movement measurement and feedback for patients and physiotherapists in the rehabilitation clinic. Human–Computer Interaction 31:294–334

    Article  Google Scholar 

  29. Lee K-H (2015) The role of compensatory movements patterns in spontaneous recovery after stroke. J Phys Ther Sci 27:2671–2673

    Article  Google Scholar 

  30. Levin MF, Kleim JA, Wolf SL (2009) What do motor “recovery” and “compensation” mean in patients following stroke? Neurorehabil Neural Repair 23(4):313–319

    Article  Google Scholar 

  31. Li Z, Nie F, Chang X, Yang Y (2017) Beyond trace ratio: weighted harmonic mean of trace ratios for multiclass discriminant analysis. IEEE Trans Knowl Data Eng 29(10):2100–2110

    Article  Google Scholar 

  32. Manghisi VM, Uva AE, Fiorentino M, Bevilacqua V, Trotta GF, Monno G (2017) Real time RULA assessment using Kinect v2 sensor. Appl Ergon

  33. Mastos M, Miller K, Eliasson A-C, Imms C (2007) Goal-directed training: linking theories of treatment to clinical practice for improved functional activities in daily life. Clin Rehabil 21:47–55

    Article  Google Scholar 

  34. Mousavi Hondori H, Khademi M (2014) A review on technical and clinical impact of microsoft kinect on physical therapy and rehabilitation. Journal of Medical Engineering 2014

  35. Mündermann L, Corazza S, Andriacchi TP (2006) The evolution of methods for the capture of human movement leading to markerless motion capture for biomechanical applications. Journal of NeuroEngineering and Rehabilitation 3:6

    Article  Google Scholar 

  36. Murray RM (2017) A mathematical introduction to robotic manipulation. CRC Press, Inc., Boca Raton

  37. Nowozin S, Shotton J (2012) Action points: A representation for low-latency online human action recognition. Microsoft Research Cambridge, Tech. Rep. MSR-TR-2012-68.

  38. Piron L, Turolla A, Agostini M, Zucconi CS, Ventura L, Tonin P, Dam M (2010) Motor learning principles for rehabilitation: a pilot randomized controlled study in poststroke patients. Neurorehabil Neural Repair 24:501–508

    Article  Google Scholar 

  39. Presti LL, La Cascia M (2016) 3D skeleton-based human action classification: A survey. Pattern Recogn 53:130–147

    Article  Google Scholar 

  40. Qamar A, Rahman MA, Basalamah S (2014) Adding inverse kinematics for providing live feedback in a serious game-based rehabilitation system. In: Intelligent Systems, Modelling and Simulation (ISMS), 2014 5th International Conference on, 2014. IEEE, pp 215-220

  41. Radomski MV, Latham CAT (2008) Occupational therapy for physical dysfunction. Wolters Kluwer Health/Lippincott Williams & Wilkins, Philadelphia

  42. Rahman MA (2015) Multimedia environment toward analyzing and visualizing live kinematic data for children with Hemiplegia. Multimed Tools Appl 74:5463–5487

    Article  Google Scholar 

  43. Ranganathan R, Wang R, Gebara R, Biswas S (2017) Detecting Compensatory Trunk Movements in Stroke Survivors using a Wearable System. In: Proceedings of the 2017 Workshop on Wearable Systems and Applications, 2017. ACM, pp 29–32

  44. Raptis M, Kirovski D, Hoppe H (2011) Real-time classification of dance gestures from skeleton animation. In: Proceedings of the 2011 ACM SIGGRAPH/Eurographics symposium on computer animation. ACM, pp 147-156

  45. Reither LR, Foreman MH, Migotsky N, Haddix C, Engsberg JR (2017) Upper extremity movement reliability and validity of the Kinect version 2. Disability and Rehabilitation: Assistive Technology 1–9

  46. Roby-Brami A, Feydy A, Combeaud M, Biryukova E, Bussel B, Levin M (2003) Motor compensation and recovery for reaching in stroke patients. Acta Neurol Scand 107:369–381

    Article  Google Scholar 

  47. Rocha C, Tonetto C, Dias A (2011) A comparison between the Denavit–Hartenberg and the screw-based methods used in kinematic modeling of robot manipulators. Robot Comput Integr Manuf 27:723–728

    Article  Google Scholar 

  48. Samad R, Bakar MZA, Pebrianti D, Mustafa M, Abdullah NRH (2017) Elbow flexion and extension rehabilitation exercise system using marker-less kinect-based method. International Journal of Electrical and Computer Engineering (IJECE) 7(3):1602–1610

    Article  Google Scholar 

  49. Schmidt RA, Young DE (1991) Methodology for motor learning: a paradigm for kinematic feedback. J Mot Behav 23:13–24

    Article  Google Scholar 

  50. Seidenari L, Varano V, Berretti S, Bimbo A (2013) Pala P Recognizing actions from depth cameras as weakly aligned multi-part bag-of-poses. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops. pp 479–485

  51. Shotton J, Sharp T, Kipman A, Fitzgibbon A, Finocchio M, Blake A, Cook M, Moore R (2013) Real-time human pose recognition in parts from single depth images. Commun ACM 56:116–124

    Article  Google Scholar 

  52. Snell RS (2007) Clinical anatomy by systems. Wolters Kluwer Health/Lippincott Williams & Wilkins, Philadelphia

  53. Stief F, Böhm H, Ebert C, Döderlein L, Meurer A (2014) Effect of compensatory trunk movements on knee and hip joint loading during gait in children with different orthopedic pathologies. Gait Posture 39:859–864

    Article  Google Scholar 

  54. Sun C, Zhang T, Xu C (2015) Latent support vector machine modeling for sign language recognition with Kinect. ACM Transactions on Intelligent Systems and Technology (TIST) 6:20

    Google Scholar 

  55. Tatla SK, Shirzad N, Lohse KR, Virji-Babul N, Hoens AM, Holsti L, Li LC, Miller KJ, Lam MY, Van der Loos HM (2015) Therapists’ perceptions of social media and video game technologies in upper limb rehabilitation. JMIR Serious Games

  56. Timmermans AA, Spooren AI, Kingma H, Seelen HA (2010) Influence of task-oriented training content on skilled arm-hand performance in stroke: a systematic review. Neurorehabil Neural Repair 24:858–870

    Article  Google Scholar 

  57. Valdés BA, Schneider AN, Van der Loos HM (2017) Reducing Trunk Compensation in Stroke Survivors: A Randomized Crossover Trial Comparing Visual and Force Feedback Modalities. Arch Phys Med Rehabil 98:1932–1940

    Article  Google Scholar 

  58. Van Vliet PM, Wulf G (2006) Extrinsic feedback for motor learning after stroke: what is the evidence? Disabil Rehabil 28:831–840

    Article  Google Scholar 

  59. Veerbeek JM, van Wegen E, van Peppen R, van der Wees PJ, Hendriks E, Rietberg M, Kwakkel G (2014) What is the evidence for physical therapy poststroke? A systematic review and meta-analysis. PLoS One 9:e87987

    Article  Google Scholar 

  60. Wang Q, Kurillo G, Ofli F, Bajcsy R (2015) Evaluation of pose tracking accuracy in the first and second generations of microsoft kinect. In: Healthcare Informatics (ICHI), 2015 International Conference on, 2015. IEEE, pp 380-389

  61. Wolf SL, Sahu K, Bay RC, Buchanan S, Reiss A, Linder S, Rosenfeldt A, Alberts J (2015) The HAAPI (Home Arm Assistance Progression Initiative) trial: a novel robotics delivery approach in stroke rehabilitation. Neurorehabil Neural Repair 29:958–968

    Article  Google Scholar 

  62. Zankel H (1951) Photogoniometry; a new method of measurement of range of motion of joints. Arch Phys Med Rehabil 32:227

    Google Scholar 

  63. Zeng Z, Li Z, Cheng D, Zhang H, Zhan K, Yang Y (2018) Two-stream multirate recurrent neural network for video-based pedestrian reidentification. IEEE Transactions on Industrial Informatics 14(7):3179–3186

    Article  Google Scholar 

  64. Zennaro S, Munaro M, Milani S, Zanuttigh P, Bernardi A, Ghidoni S, Menegatti E (2015) Performance evaluation of the 1st and 2nd generation Kinect for multimedia applications. In: Multimedia and Expo (ICME), 2015 IEEE International Conference on, 2015. IEEE, pp 1-6

  65. Zhao W, Espy DD, Reinthal MA, Feng H (2014) A feasibility study of using a single kinect sensor for rehabilitation exercises monitoring: A rule based approach. In: Computational Intelligence in Healthcare and e-health (CICARE), 2014 IEEE Symposium on, 2014. IEEE, pp 1-8

  66. Zulkarnain RF, Kim G-Y, Adikrishna A, Hong HP, Kim YJ, Jeon I-H (2017) Digital data acquisition of shoulder range of motion and arm motion smoothness using Kinect v2. J Shoulder Elb Surg 26:895–901

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Cyber Space Research Institute, Shahid Beheshti University, GC, Tehran, Iran

    Fatemeh Mortazavi & Ali Nadian-Ghomsheh

Authors
  1. Fatemeh Mortazavi
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Ali Nadian-Ghomsheh
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Ali Nadian-Ghomsheh.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Electronic supplementary material

ESM 1

(AVI 7936 kb)

ESM 2

(AVI 5727 kb)

ESM 3

(AVI 5349 kb)

ESM 4

(AVI 3548 kb)

Appendices

Appendix 1

Human body movements are tri-dimensional, allowing the body to move through XY, XZ and YZ space planes. When the body and the planes are projected into the same space, joint movements can be related to these planes. The XY plane, regarded as frontal or coronal plane divides the body into front and back. The YZ plane, called sagittal or vertical plane divides the body into right and left side. XZ is the horizontal or transversal plane which divides the body into up and down portions (Fig. 16).

Fig. 16
figure 16

Comparison between representation of shoulder flexion/extension using (top row) joint degree of freedom and (bottom row) spherical systems

Full size image

  • Abduction: movement in the coronal plane that moves a limb laterally away from the body

  • Adduction: movement in the coronal plane that moves a limb medially toward or across the midline of the body

  • Extension: movement in the sagittal plane that increases the angle of a joint (straightens the joint); motion involving posterior bending of the vertebral column or returning to the upright position from a flexed position

  • Flexion: movement in the sagittal plane that decreases the angle of a joint (bends the joint); motion involving anterior bending of the vertebral column

Appendix 2

Equations for extracting joint angles with Cosine Rule (CR), Spherical Coordinates (SC), and Right Triangle Trigonometry (RTT).

 

CR

RTT

SC

\( {\cos}^{-1}\left(\frac{a^2+{b}^2-{c}^2}{2 ab}\right) \)

\( {\tan}^{-1}\left(\frac{\Delta y}{\Delta x}\right) \)

\( {\tan}^{-1}\left(\frac{\Delta z}{\Delta y}\right) \)

\( {\tan}^{-1}\left(\frac{\Delta z}{\Delta x}\right) \)

Shoulder abduction

\( a=\sqrt{{{\left({S}_l-{E}_l\right)}_x}^2+{{\left({S}_l-{E}_l\right)}_y}^2+{{\left({S}_l-{E}_l\right)}_z}^2} \)

\( b=\sqrt{{{\left({S}_l-{H}_l\right)}_x}^2+{{\left({S}_l-{H}_l\right)}_y}^2+{{\left({S}_l-{H}_l\right)}_z}^2} \)

\( c=\sqrt{{{\left({H}_l-{E}_l\right)}_x}^2+{{\left({H}_l-{E}_l\right)}_y}^2+{{\left({H}_l-{E}_l\right)}_z}^2} \)

\( {\tan}^{-1}\left(\frac{Y_{W_l}-{Y}_{S_l}}{X_{W_l}-{X}_{S_l}}\right) \)

\( {\displaystyle \begin{array}{l}{\cos}^{-1}\left(\frac{S_l{E}_l\cdot Up}{\mid {S}_l{E}_l\Big\Vert Up\mid}\right)\\ {}\end{array}} \)

Shoulder flexion

\( a=\sqrt{{{\left({S}_l-{W}_l\right)}_x}^2+{{\left({S}_l-{W}_l\right)}_y}^2+{{\left({S}_l-{W}_l\right)}_z}^2} \)

\( b=\sqrt{{{\left({S}_l-{H}_l\right)}_x}^2+{{\left({S}_l-{H}_l\right)}_y}^2+{{\left({S}_l-{H}_l\right)}_z}^2} \)

\( c=\sqrt{{{\left({H}_l-{E}_l\right)}_x}^2+{{\left({H}_l-{E}_l\right)}_y}^2+{{\left({H}_l-{E}_l\right)}_z}^2} \)

\( {\tan}^{-1}\left(\frac{Z_{W_l}-{Z}_{S_l}}{Y_{W_l}-{Y}_{S_l}}\right) \)

\( {\cos}^{-1}\left(\frac{S_l{E}_l\cdot Up}{\mid {S}_l{E}_l\Big\Vert Up\mid}\right) \)

Shoulder Horizontal adduction

\( a=\sqrt{{{\left({S}_l-{E}_l\right)}_x}^2+{{\left({S}_l-{E}_l\right)}_y}^2+{{\left({S}_l-{E}_l\right)}_z}^2} \)

\( b=\sqrt{{{\left({S}_l-{S}_r\right)}_x}^2+{{\left({S}_l-{S}_r\right)}_y}^2+{{\left({S}_l-{S}_r\right)}_z}^2} \)

\( c=\sqrt{{{\left({S}_r-{E}_l\right)}_x}^2+{{\left({S}_r-{E}_l\right)}_y}^2+{S}_r-{E}_l\Big){{}_z}^2} \)

\( {\tan}^{-1}\left(\frac{Z_{W_l}-{Z}_{S_l}}{X_{W_l}-{X}_{S_l}}\right) \)

\( {\cos}^{-1}\left(\frac{S_l{E}_l^p\cdot {S}_r{S}_l}{\mid {S}_l{E}_l^p\Big\Vert {S}_r{S}_l\mid}\right) \)

Elbow flexion

\( a=\sqrt{{{\left({E}_l-{W}_l\right)}_x}^2+{{\left({E}_l-{W}_l\right)}_y}^2+{{\left({E}_l-{W}_l\right)}_z}^2} \)

\( b=\sqrt{{{\left({S}_l-{E}_l\right)}_x}^2+{{\left({S}_l-{E}_l\right)}_y}^2+{{\left({S}_l-{E}_l\right)}_z}^2} \)

\( c=\sqrt{{{\left({S}_l-{W}_l\right)}_x}^2+{{\left({S}_l-{W}_l\right)}_y}^2+{{\left({S}_l-{W}_l\right)}_z}^2} \)

\( 90-{\tan}^{-1}\left(\frac{Y_{W_l}-{Y}_{E_l}}{Z_{W_l}-{Z}_{E_l}}\right) \)

\( {\cos}^{-1}\left(\frac{B\cdot {E}_l{W}_l}{\mid B\Big\Vert {E}_l{W}_l\mid}\right) \)

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Mortazavi, F., Nadian-Ghomsheh, A. Continues online exercise monitoring and assessment system with visual guidance feedback for stroke rehabilitation. Multimed Tools Appl 78, 32055–32085 (2019). https://doi.org/10.1007/s11042-019-08020-2

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-019-08020-2

Keywords

Search

Navigation