Skip to main content
SpringerLink
Log in

On unifying deep learning and edge computing for human motion analysis in exergames development

  • Special issue on Advances of Neural Computing phasing challenges in the era of 4th industrial revolution
  • Published:
Neural Computing and Applications Aims and scope Submit manuscript

Abstract

This work describes a novel methodology for creating exergames on an edge-native platform with the integration of multiple deep neural networks. A prototype of the platform, which includes capabilities for innovative gameplay and advanced user interactivity, has been implemented and deployed in a real-world scenario. At core of the proposed methodology is the ad hoc training of classifiers for posture classification which can be dynamically adapted to the specific requirements of the usage scenario, operational and environmental conditions allowing for real-time identification of events and advanced game control. The proposed solution is ideal for individual consumers in a home environment since is supports by-design edge platforms minimizing the cost of the system and enabling in parallel the communication with state-of-the-art hardware (i.e., GPUs, TPUs, computer boards) for real-time operation. The proposed system allows the collection and analysis of game data, which can be exploited by specialized personnel in rehabilitation centers or for other purposes in the areas of healthcare and assisted living.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13

Similar content being viewed by others

Notes

  1. A complete set of tools for building, deploying, and managing fleets of connected Linux devices: https://www.balena.io/.

References

  1. Abadi M, Barham P, Chen J, Chen Z, Davis A, Dean J, Devin M, Ghemawat S, Irving G, Isard M et al (2016) Tensorflow: a system for large-scale machine learning. In: 12th USENIX symposium on operating systems design and implementation (OSDI 16), pp 265–283 (2016)

  2. Aggarwal JK, Cai Q (1999) Human motion analysis: a review. Comput Vis Image Understand 73(3):428–440

    Article  Google Scholar 

  3. Al-Hrathi R, Karime A, Al-Osman H, El Saddik A (2012) Exerlearn bike: an exergaming system for children’s educational and physical well-being. In: 2012 IEEE international conference on multimedia and expo workshops. IEEE, pp 489–494

  4. Benzing V, Schmidt M (2018) Exergaming for children and adolescents: strengths, weaknesses, opportunities and threats. J Clin Med 7(11):422

    Article  Google Scholar 

  5. Bianchini M, Scarselli F (2014) On the complexity of shallow and deep neural network classifiers. In: ESANN. Citeseer

  6. Bonnechére B, Jansen B, Omelina L, Da Silva L, Mouraux D, Rooze M, Van Sint JS (2013) Patient follow-up using serious games. A feasibility study on low back pain patients. In: Games for health. Springer, pp 185–195

  7. Bradski G (2000) The OpenCV Library. Dr. Dobb’s Journal of Software Tools

  8. Brox E, Fernandez-Luque L, Tollefsen T (2011) Healthy gaming-video game design to promote health. Appl Clin Inform 2(2):128–142

    Article  Google Scholar 

  9. Bulat A, Tzimiropoulos G (2016) Human pose estimation via convolutional part heatmap regression. In: European conference on computer vision. Springer, pp 717–732

  10. Carling C, Bloomfield J, Nelsen L, Reilly T (2008) The role of motion analysis in elite soccer. Sports Med 38(10):839–862

    Article  Google Scholar 

  11. Cass S (2019) Taking AI to the edge: Google’s TPU now comes in a maker-friendly package. IEEE Spectr 56(5):16–17

    Article  Google Scholar 

  12. Cass S (2020) Nvidia makes it easy to embed AI: the jetson nano packs a lot of machine-learning power into diy projects-[hands on]. IEEE Spectr 57(7):14–16

    Article  Google Scholar 

  13. Chang YJ, Chen SF, Huang JD (2011) A kinect-based system for physical rehabilitation: a pilot study for young adults with motor disabilities. Res Dev Disabil 32(6):2566–2570

    Article  Google Scholar 

  14. Chen Y, Tian Y, He M (2020) Monocular human pose estimation: a survey of deep learning-based methods. Comput Vis Image Underst 192:102897

    Article  Google Scholar 

  15. Clark R, Kraemer T (2009) Clinical use of nintendo wiiTM bowling simulation to decrease fall risk in an elderly resident of a nursing home: A case report. J Geriatric Physical Ther 32(4):174–180

    Article  Google Scholar 

  16. Doukas C, Metsis V, Becker E, Le Z, Makedon F, Maglogiannis I (2011) Digital cities of the future: extending@ home assistive technologies for the elderly and the disabled. Telematics Inform 28(3):176–190

    Article  Google Scholar 

  17. Garcia JA, Navarro KF, Schoene D, Smith ST, Pisan Y (2012) Exergames for the elderly: towards an embedded kinect-based clinical test of falls risk. In: HIC, pp 51–57

  18. Goudelis G, Karpouzis K, Kollias S (2011) Robust human action recognition using history trace templates. In: 12th International workshop on image analysis for multimedia interactive services (WIAMIS), Delft, The Netherlands

  19. Ha K, Pillai P, Lewis G, Simanta S, Clinch S, Davies N, Satyanarayanan M (2013) The impact of mobile multimedia applications on data center consolidation. In: 2013 IEEE international conference on cloud engineering (IC2E). IEEE, pp 166–176

  20. Ilg W, Schatton C, Schicks J, Giese MA, Schöls L, Synofzik M (2012) Video game-based coordinative training improves ataxia in children with degenerative ataxia. Neurology 79(20):2056–2060

    Article  Google Scholar 

  21. Jiang J, Ananthanarayanan G, Bodik P, Sen S, Stoica I (2018) Chameleon: scalable adaptation of video analytics. In: Proceedings of the 2018 conference of the ACM special interest group on data communication, pp 253–266

  22. Kelly S (2016) Python, PyGame and raspberry Pi game development. Springer, New York

    Book  Google Scholar 

  23. Liang S, Srikant R (2016) Why deep neural networks for function approximation? arXiv preprint arXiv:1610.04161

  24. Matallaoui A, Koivisto J, Hamari J, Zarnekow R (2017) How effective is “exergamification”? A systematic review on the effectiveness of gamification features in exergames. In: Proceedings of the 50th Hawaii international conference on system sciences

  25. Menychtas A, Doukas C, Tsanakas P, Maglogiannis I (2017) A versatile architecture for building IOT quantified-self applications. In: 2017 IEEE 30th international symposium on computer-based medical systems (CBMS). IEEE, pp 500–505

  26. Mezari A, Maglogiannis I (2018) An easily customized gesture recognizer for assisted living using commodity mobile devices. J Healthcare Eng

  27. Newell A, Yang K, Deng J (2016) Stacked hourglass networks for human pose estimation. In: European conference on computer vision. Springer, pp 483–499

  28. Obdržálek Š, Kurillo G, Ofli F, Bajcsy R, Seto E, Jimison H, Pavel M (2012) Accuracy and robustness of kinect pose estimation in the context of coaching of elderly population. In: 2012 Annual international conference of the IEEE engineering in medicine and biology society. IEEE, pp 1188–1193

  29. Oved D, Zhu T (2019) Bodypix. https://github.com/tensorflow/tfjs-models/tree/master/body-pix

  30. Pandey G, Dukkipati A (2014) To go deep or wide in learning? arXiv preprint arXiv:1402.5634

  31. Papandreou G, Zhu T, Chen LC, Gidaris S, Tompson J, Murphy K (2018) Personlab: person pose estimation and instance segmentation with a bottom-up, part-based, geometric embedding model. In: Proceedings of the European conference on computer vision (ECCV), pp 269–286

  32. Pardos A, Menychtas A, Maglogiannis I (2020) Introducing an edge-native deep learning platform for exergames. In: IFIP international conference on artificial intelligence applications and innovations. Springer, pp 88–98 (2020)

  33. Pedregosa F, Varoquaux G, Gramfort A, Michel V, Thirion B, Grisel O, Blondel M, Prettenhofer P, Weiss R, Dubourg V, Vanderplas J, Passos A, Cournapeau D, Brucher M, Perrot M, Duchesnay E (2011) Scikit-learn: machine learning in python. J Mach Learn Res 12:2825–2830

    MathSciNet  MATH  Google Scholar 

  34. Perera C, Member CHL, Jayawardena S, Chen M (2015) Context-aware computing in the internet of things: a survey on internet of things from industrial market perspective. arXiv preprint arXiv:1502.00164

  35. Pirovano M, Lanzi PL, Mainetti R, Borghese NA(2013) Iger: a game engine specifically tailored to rehabilitation. In: Games for health. Springer, pp 85–98

  36. Pogrzeba L, Wacker M, Jung B (2012) Potentials of a low-cost motion analysis system for exergames in rehabilitation and sports medicine. In: E-learning and games for training, education, health and sports. Springer, pp 125–133

  37. Qian K, Wu C, Zhou Z, Zheng Y, Yang Z, Liu Y (2017) Inferring motion direction using commodity wi-fi for interactive exergames. In: Proceedings of the 2017 CHI conference on human factors in computing systems, pp 1961–1972

  38. Safran I, Shamir O (2017) Depth-width tradeoffs in approximating natural functions with neural networks. In: International conference on machine learning. PMLR, pp 2979–2987

  39. Schlömer T, Poppinga B, Henze N, Boll S (2008) Gesture recognition with a WII controller. In: Proceedings of the 2nd international conference on Tangible and embedded interaction, pp 11–14

  40. Senthilkumar G, Gopalakrishnan K, Kumar VS (2014) Embedded image capturing system using raspberry pi system. Int J Emerg Trends Technol Comput Sci 3(2):213–215

    Google Scholar 

  41. Shih CH, Yeh JC, Shih CT, Chang ML (2011) Assisting children with attention deficit hyperactivity disorder actively reduces limb hyperactive behavior with a nintendo wii remote controller through controlling environmental stimulation. Res Dev Disabil 32(5):1631–1637

    Article  Google Scholar 

  42. Skjæret N, Nawaz A, Morat T, Schoene D, Helbostad JL, Vereijken B (2016) Exercise and rehabilitation delivered through exergames in older adults: an integrative review of technologies, safety and efficacy. Int J Med Inform 85(1):1–16

    Article  Google Scholar 

  43. Street TD, Lacey SJ, Langdon RR (2017) Gaming your way to health: a systematic review of exergaming programs to increase health and exercise behaviors in adults. Games Health J 6(3):136–146

    Article  Google Scholar 

  44. Styliadis C, Konstantinidis E, Billis A, Bamidis P (2014) Employing affection in elderly healthcare serious games interventions. In: Proceedings of the 7th international conference on PErvasive technologies related to assistive environments, pp 1–4

  45. Susi T, Johannesson M, Backlund P (2007) Serious games: an overview

  46. Toshev A, Szegedy C (2014) Deeppose: human pose estimation via deep neural networks. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 1653–1660

  47. van der Kruk E, Reijne MM (2018) Accuracy of human motion capture systems for sport applications; state-of-the-art review. Eur J Sport Sci 18(6):806–819

    Article  Google Scholar 

  48. Wang L, Hu W, Tan T (2003) Recent developments in human motion analysis. Pattern Recogn 36(3):585–601

    Article  Google Scholar 

  49. Wong WY, Wong MS (2008) Trunk posture monitoring with inertial sensors. Eur Spine J 17(5):743–753

    Article  Google Scholar 

  50. Yi S, Hao Z, Qin Z, Li Q (2015) Fog computing: platform and applications. In: 2015 Third IEEE workshop on hot topics in web systems and technologies (HotWeb). IEEE, pp 73–78

  51. Zhang S, Wen L, Bian X, Lei Z, Li SZ (2018) Occlusion-aware r-cnn: detecting pedestrians in a crowd. In: Proceedings of the European conference on computer vision (ECCV), pp 637–653

  52. Zhang X, Wang Y, Chao L, Li C, Wu L, Peng X, Xu Z (2017) Iehouse: a non-intrusive household appliance state recognition system. In: 2017 IEEE SmartWorld, ubiquitous intelligence and computing, advanced and trusted computed, scalable computing and communications, cloud and big data computing, internet of people and smart city innovation (SmartWorld/SCALCOM/UIC/ATC/CBDCom/IOP/SCI). IEEE, pp 1–8

Download references

Acknowledgements

This research has been co-financed by the European Union and Greek national funds through the Operational Program Competitiveness, Entrepreneurship and Innovation, under the call RESEARCH-CREATE-INNOVATE (Project Code: MediLudus Personalised home care based on game and gamify elements \(T2EK\varDelta K\)-03049).

Author information

Authors and Affiliations

  1. BioAssist SA, 1 G. Mpakou St., 11524, Athens, Greece

    Antonis Pardos & Andreas Menychtas

  2. Department of Digital Systems, University of Piraeus, 80 M. Karaoli & A. Dimitriou St., 18534, Piraeus, Greece

    Ilias Maglogiannis

Authors
  1. Antonis Pardos
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Andreas Menychtas
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Ilias Maglogiannis
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Ilias Maglogiannis.

Ethics declarations

Conflict of interest

The authors declare that they have no conflict of interest.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Pardos, A., Menychtas, A. & Maglogiannis, I. On unifying deep learning and edge computing for human motion analysis in exergames development. Neural Comput & Applic 34, 951–967 (2022). https://doi.org/10.1007/s00521-021-06181-6

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00521-021-06181-6

Keywords

Search

Navigation