Skip to main content
SpringerLink
Log in

VRheab: a fully immersive motor rehabilitation system based on recurrent neural network

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

In this paper, a fully immersive serious game system that combines two Natural User Interfaces (NUIs) and a Head Mounted Display (HMD) to provide an interactive Virtual Environment (VE) for patient rehabilitation is proposed. Patients’ data are acquired in real-time by the NUIs, while by the HMD the VE is shown to them, thus allowing the interaction. A Long Short-Term Memory Recurrent Neural Network (LSTM-RNN), previously trained by healthy subjects (i.e., baseline), processes patients’ movements in real-time during the rehabilitation exercises to provide the degree of their performance. By comparing the functionalities of the proposed system with the ongoing state-of-the-art, it is worth noting that the reported fully immersive serious game system provides a concrete contribute to the current literature in terms of completeness and versatility. The results obtained by three rehabilitation exercises, chosen as reference case studies, performed on real patients affected by Parkinson’s disease have shown the effectiveness of the presented approach. Finally, the analysis of the feedbacks received by the therapists and patients who have used the system have highlighted remarkable results in terms of motivation, acceptance, and usability.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13

Similar content being viewed by others

References

  1. Ai B, Zhou Y, Yu Y, Du S (2017) Human pose estimation using deep structure guided learning. In: Winter conference on applications of computer vision (WACV), pp 1224–1231

  2. Alimanova M, Borambayeva S, Kozhamzharova D, Kurmangaiyeva N, Ospanova D, Tyulepberdinova G, Gaziz G, Kassenkhan A (2017) Gamification of hand rehabilitation process using virtual reality tools: using leap motion for hand rehabilitation. In: 1th International conference on robotic computing (IRC), pp 336–339

  3. Angra S, Ahuja S (2017) Machine learning and its applications: a review. In: International conference on big data analytics and computational intelligence (ICBDAC), pp 57–60

  4. Asadi-Aghbolaghi M, Clapés A, Bellantonio M, Escalante HJ, Ponce-López V, Baró X, Guyon I, Kasaei S, Escalera S (2017) A survey on deep learning based approaches for action and gesture recognition in image sequences. In: 12th International conference on automatic face gesture recognition (FG), pp 476–483

  5. Avola D, Spezialetti M, Placidi G (2013) Design of an efficient framework for fast prototyping of customized human –computer interfaces and virtual environments for rehabilitation. Comput Methods Programs Biomed 110(3):490–502

    Article  Google Scholar 

  6. Bengio Y, Simard P, Frasconi P (1994) Learning long-term dependencies with gradient descent is difficult. Trans Neur Netw 5(2):157–166

    Article  Google Scholar 

  7. Brau E, Jiang H (2016) 3d human pose estimation via deep learning from 2d annotations. In: 4th International conference on 3D vision (3DV), pp 582–591

  8. Brooke J (2013) Sus: a retrospective. J Usab Stud 8(2):29–40

    Google Scholar 

  9. Byeon W, Breuel TM, Raue F, Liwicki M (2015) Scene labeling with lstm recurrent neural networks. In: Conference on computer vision and pattern recognition (CVPR), pp 3547–3555

  10. Cho K, van Merrienboer B, Bahdanau D, Bengio Y (2014) On the properties of neural machine translation: encoder-decoder approaches. In: 8th Workshop on syntax, semantics and structure in statistical translation (SSST), pp 103–111

  11. Chung J, Gulcehre C, Cho K, Bengio Y (2014) Empirical evaluation of gated recurrent neural networks on sequence modeling. In: Deep learning and representation learning workshop (NIPS), pp 1–9

  12. Crosbie JH, Lennon S, McNeill MDJ, McDonough SM (2006) Virtual reality in the rehabilitation of the upper limb after stroke: The user’s perspective. Cyberpsychol Behav 9(2):137–141

    Article  Google Scholar 

  13. Desai PR, Desai PN, Ajmera KD, Mehta K (2014) A review paper on oculus rift-a virtual reality headset. Int J Eng Trends Technol 13(4):175–179

    Article  Google Scholar 

  14. Dorsey ER, Constantinescu R, Thompson JP, Biglan KM, Holloway RG, Kieburtz K, Marshall FJ, Ravina BM, Schifitto G, Siderowf A, Tanner CM (2007) Projected number of people with parkinson disease in the most populous nations, 2005 through 2030. Neurology 68(5):384–386

    Article  Google Scholar 

  15. Feng X, Liu C, Guo Q, Bai Y, Ren Y, Ren B, Bai J, Chen L (2013) Research progress in rehabilitation treatment of stroke patients: a bibliometric analysis. Neural Regen Res 8(15):1423–1430

    Google Scholar 

  16. García-Martínez S, Orihuela-Espina F, Sucar LE, Moran AL, Hernández-Franco J (2015) A design framework for arcade-type games for the upper-limb rehabilitation. In: International conference on virtual rehabilitation (ICVR), pp 235–242

  17. Gargantini A, Terzi F, Zambelli M, Bonfanti S (2015) A low-cost virtual reality game for amblyopia rehabilitation. In: 3rd Workshop on ICTs for improving patients rehabilitation research techniques (REHAB), pp 81–84

  18. Gobron SC, Zannini N, Wenk N, Schmitt C, Charrotton Y, Fauquex A, Lauria M, Degache F, Frischknecht R (2015) Serious games for rehabilitation using head-mounted display and haptic devices. In: International conference on augmented and virtual reality (AVR), pp 199–219

  19. Golomb MR, McDonald BC, Warden SJ, Yonkman J, Saykin AJ, Shirley B, Huber M, Rabin B, AbdelBaky M, Nwosu ME, Barkat-Masih M, Burdea GC (2010) In-home virtual reality videogame telerehabilitation in adolescents with hemiplegic cerebral palsy. Arch Phys Med Rehabil 91(1):1–9

    Article  Google Scholar 

  20. Guna J, Jakus G, Pogačnik M, Tomažič S, Sodnik J (2014) An analysis of the precision and reliability of the leap motion sensor and its suitability for static and dynamic tracking. Sensors 14(2):3702–3720

    Article  Google Scholar 

  21. Han J, Shao L, Xu D, Shotton J (2013) Enhanced computer vision with microsoft kinect sensor: a review. IEEE Trans Cybern 43(5):1318–1334

    Article  Google Scholar 

  22. Hansard M, Lee S, Choi O, Horaud R (2013) Time of flight cameras: principles, methods, and application. In: SpringerBriefs in computer science. 1st edn. Springer-Verlag, London, p 95

  23. Harrington MCR (2011) Empirical evidence of priming, transfer, reinforcement, and learning in the real and virtual trillium trails. IEEE Trans Learn Technol 4(2):175–186

    Article  Google Scholar 

  24. Hermans M, Schrauwen B (2013) Training and analysing deep recurrent neural networks. In: Burges C J C, Bottou L, Welling M, Ghahramani Z, Weinberger K Q (eds) Advances in neural information processing systems, pp 190-198

  25. Hirsch M, Farley B (2009) Exercise and neuroplasticity in persons living with parkinson’s disease. Eur J Phys Rehabil Med 45(2):215–229

    Google Scholar 

  26. Hochreiter S, Schmidhuber J (1997) Long short-term memory. Neural Comput 9(8):1735–1780

    Article  Google Scholar 

  27. Holzinger A, Scherer R, Seeber M, Wagner J, Müller-Putz G (2012) Computational sensemaking on examples of knowledge discovery from neuroscience data: towards enhancing stroke rehabilitation. In: 3th International conference on information technology in bio- and medical informatics (ITBAM), pp 166–168

  28. Horaud R, Hansard M, Evangelidis G, Ménier C (2016) An overview of depth cameras and range scanners based on time-of-flight technologies. Mach Vis Appl 27(7):1005–1020

    Article  Google Scholar 

  29. Huang Z, Wan C, Probst T, Van Gool L (2017) Deep learning on lie groups for skeleton-based action recognition. In: Conference on computer vision and pattern recognition (CVPR), pp 6099–6108

  30. Ijjina EP, Chalavadi KM (2017) Human action recognition in rgb-d videos using motion sequence information and deep learning. Pattern Recognit 72(Supplement C):504–516

    Article  Google Scholar 

  31. Jorissen P, Wijnants M, Lamotte M (2005) Dynamic interactions in physically realistic collaborative virtual environments. IEEE Trans Vis Comput Graphics 11(6):649–660

    Article  Google Scholar 

  32. Kato N, Tanaka T, Sugihara S, Shimizu K, Kudo N (2016) Trial operation of a cloud service-based three-dimensional virtual reality tele-rehabilitation system for stroke patients. In: 11th International conference on computer science education (ICCSE), pp 285–290

  33. Keus SH, Munneke M, Nijkrake MJ, Kwakkel G, Bloem BR (2009) Physical therapy in parkinson’s disease: evolution and future challenges. Mov Disord 24(1):1–14

    Article  Google Scholar 

  34. Knight A, Carey S, Dubey R (2016) An interim analysis of the use of virtual reality to enhance upper limb prosthetic training and rehabilitation. In: 9th ACM International conference on pervasive technologies related to assistive environments, pp 1–4

  35. Kwakkel G, de Goede C, van Wegen E (2007) Impact of physical therapy for parkinson’s disease: a critical review of the literature. Parkinson Related Disord 13(Supplement 3):S478–S487

    Article  Google Scholar 

  36. LeCun Y, Bengio Y, Hinton G (2015) Deep learning. Nature 521:436–444

    Article  Google Scholar 

  37. Li S, Zhang W, Chan AB (2017) Maximum-margin structured learning with deep networks for 3d human pose estimation. Int J Comput Vis 122(1):149–168

    Article  MathSciNet  Google Scholar 

  38. Luis MAVS, Atienza RO, Luis AMS (2016) Immersive virtual reality as a supplement in the rehabilitation program of post-stroke patients. In: 10th International conference on next generation mobile applications, security and technologies (NGMAST), pp 47–52

  39. Miljkovic D, Aleksovski D, Podpečan V, Lavrač N, Malle B, Holzinger A (2016) Machine learning and data mining methods for managing parkinson’s disease. In: Machine learning for health informatics: state-of-the-art and future challenges, pp 209–220

  40. Munroe C, Meng Y, Yanco H, Begum M (2016) Augmented reality eyeglasses for promoting home-based rehabilitation for children with cerebral palsy. In: 11th ACM/IEEE International conference on human robot interaction, pp 565–565

  41. Nielsen J, Molich R (1990) Heuristic evaluation of user interfaces. In: SIGCHI Conference on human factors in computing systems, pp 249–256

  42. Oak JW, Bae JH (2014) Development of smart multiplatform game app using unity3d engine for cpr education. Int J Multimed Ubiquit Eng 9(7):263–268

    Article  Google Scholar 

  43. Paterson RE (2015) Basics of human binocular vision. In: Human factors of stereoscopic 3D displays, pp 9–21

  44. Pei W, Xu G, Li M, Ding H, Zhang S, Luo A (2016) A motion rehabilitation self-training and evaluation system using kinect. In: 13th International conference on ubiquitous robots and ambient intelligence (URAI), pp 353–357

  45. Pellecchia MT, Grasso A, Biancardi LG, Squillante M, Bonavita V, Barone P (2004) Physical therapy in parkinson’s disease: an open long-term rehabilitation trial. J Neurol 251(5):595–598

    Article  Google Scholar 

  46. Placidi G, Avola D, Iacoviello D, Cinque L (2013) Overall design and implementation of the virtual glove. Comput Biol Med 43(11):1927–1940

    Article  Google Scholar 

  47. Placidi G, Avola D, Ferrari M, Iacoviello D, Petracca A, Quaresima V, Spezialetti M (2014) A low-cost real time virtual system for postural stability assessment at home. Comput Methods Programs Biomed 117(2):322–333

    Article  Google Scholar 

  48. Rawat S, Vats S, Kumar P (2016) Evaluating and exploring the myo armband. In: International conference system modeling advancement in research trends (SMART), pp 115–120

  49. Rego P, Moreira PM, Reis LP (2010) Serious games for rehabilitation: a survey and a classification towards a taxonomy. In: 5th Iberian conference on information systems and technologies (CISTI), pp 1–6

  50. Saini S, Rambli DRA, Sulaiman S, Zakaria MN, Shukri SRM (2012) A low-cost game framework for a home-based stroke rehabilitation system. In: International conference on computer information science (ICCIS), pp 55–60

  51. Sak H, Senior AW, Beaufays F (2014) Long short-term memory recurrent neural network architectures for large scale acoustic modeling. In: Interspeech, pp 338–342

  52. Sargano AB, Wang X, Angelov P, Habib Z (2017) Human action recognition using transfer learning with deep representations. In: International joint conference on neural networks (IJCNN), pp 463–469

  53. Schmidhuber J (2015) Deep learning in neural networks: an overview. Neural Netw 61(Supplement C):85–117

    Article  Google Scholar 

  54. Sen SL, Xiang YB, Ming ESL, Xiang KK, Fai YC, Khan QI (2015) Enhancing effectiveness of virtual reality rehabilitation system: durian runtuh. In: 10th Asian control conference (ASCC), pp 1–6

  55. Shiratuddin MF, Hajnal A, Farkas A, Wong KW, Legradi G (2012) A proposed framework for an interactive visuotactile 3d virtual environment system for visuomotor rehabilitation of stroke patients. In: International conference on computer information science (ICCIS), pp 1052–1057

  56. Singh D, Merdivan E, Psychoula I, Kropf J, Hanke S, Geist M, Holzinger A (2017) Human activity recognition using recurrent neural networks. In: International cross-domain conference on machine learning and knowledge extraction, pp 267–274

  57. Sosa GD, Sánchez J, Franco H (2015) Improved front-view tracking of human skeleton from kinect data for rehabilitation support in multiple sclerosis. In: 20th Symposium on signal processing, images and computer vision (STSIVA), pp 1–7

  58. Wasenmüller O, Stricker D (2016) Comparison of kinect v1 and v2 depth images in terms of accuracy and precision. In: Asian Conference on computer vision (ACCV), pp 34–45

  59. Weiss PL, Rand D, Katz N, Kizony R (2004) Video capture virtual reality as a flexible and effective rehabilitation tool. J NeuroEng Rehabil 1(1):1–12

    Article  Google Scholar 

  60. Zhang Z (2012) Microsoft kinect sensor and its effect. IEEE Multimed 19(2):4–10

    Article  Google Scholar 

  61. Zhang XY, Xie GS, Liu CL, Bengio Y (2017) End-to-end online writer identification with recurrent neural network. IEEE Trans Human-Mach Syst 47 (2):285–292

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of Mathematics, Computer Science and Physics, University of Udine, Via delle Scienze 206, 33100, Udine, Italy

    Danilo Avola & Gian Luca Foresti

  2. Department of Computer Science, Sapienza University, Via Salaria 113, 00198, Rome, Italy

    Luigi Cinque, Marco Raoul Marini & Daniele Pannone

Authors
  1. Danilo Avola
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Luigi Cinque
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Gian Luca Foresti
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Marco Raoul Marini
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Daniele Pannone
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Daniele Pannone.

Electronic supplementary material

Below is the link to the electronic supplementary material.

(MP4 9.35 MB)

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Avola, D., Cinque, L., Foresti, G.L. et al. VRheab: a fully immersive motor rehabilitation system based on recurrent neural network. Multimed Tools Appl 77, 24955–24982 (2018). https://doi.org/10.1007/s11042-018-5730-1

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-018-5730-1

Keywords

Search

Navigation