Skip to main content
SpringerLink
Log in

Control of adaptive running platform based on machine vision technologies and neural networks

  • Original Article
  • Published:
Neural Computing and Applications Aims and scope Submit manuscript

Abstract

The paper considers the problem of selection of the optimal method for controlling an adaptive running platform for movement organization in virtual reality. The analysis of existing approaches for the control of such systems is carried out, a list of existing methods is formed, and new functions are developed. In order to solve the problem of user positioning within the framework of an adaptive running platform, two approaches, based on virtual reality trackers and using a machine vision, are considered and implemented. The problem of the study includes the choice of the optimal method of controlling an adaptive running platform to ensure maximum user comfort when walking on it. The study methodology includes the creation of a simplified model of human interaction with a running platform, the formalization of the assessment of the quality of movement, various methods of human positioning and platform control functions, followed by an experimental part on their comparison and the search for an optimal approach to running platform control. It was proved that machine vision technologies and neural networks allow positioning a person with sufficient accuracy, and they are deprived of the disadvantages of trackers. Moreover, comparative studies of six different control functions were carried out with the tracker-based positioning method and with the use of machine vision technologies. The most universal is the nonlinear one, a detailed zonal and proportional–differential functions are recommended for the tracker-based positioning method, linear and proportional–differential functions are recommended for positioning based on machine vision. The scientific novelty of the study consists in the formalization and comparison of various control methods of adaptive running platforms (based on linear and nonlinear functions, proportional differential law and neural networks), methods of positioning a person on them (using cameras and trackers), which will expand the area of knowledge about the optimal control functions of this class of devices. The practical significance of the research lies in a comprehensive description of the solution of the problems of organizing the control of adaptive running platforms and positioning a person by various methods.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16
Fig. 17

Similar content being viewed by others

References

  1. Silva EC, Sirqueira CA, Pinto FG (2021) The immersion in virtual reality of control room activity. In: Black NL, PatrickNeumann W, Noy I (eds) Congress of the international ergonomics association. Springer, Cham, pp 82–88. https://doi.org/10.1007/978-3-030-74608-7_11

    Chapter  Google Scholar 

  2. Demirel D, Hamam A, Scott C, Karaman B, Toker O, Pena L (2021) Towards a new chemistry learning platform with virtual reality and haptics. In: International conference on human-computer interaction. Springer, Cham, pp 253–267. https://doi.org/10.1007/978-3-030-77943-6_16

  3. Li W, Xiao JX (2021) User experience in digital museums: a case study of the palace museum in Beijing. In: International conference on human-computer interaction. Springer, Cham, pp 436–448. https://doi.org/10.1007/978-3-030-77411-0_28

  4. Khasanova GF, Shageeva FT (2019) Poster: variable scenarios of the VR use in training specialists for chemical industry. In: International conference on interactive collaborative learning. Springer, Cham, pp 808–813. https://doi.org/10.1007/978-3-030-40271-6_79

  5. Mitrut O, Moldoveanu A, Petrescu L, Petrescu C, Moldoveanu F (2021) A review of virtual therapists in anxiety and phobias alleviating applications. In: International conference on human–computer interaction. Springer, Cham, pp 71–79. https://doi.org/10.1007/978-3-030-77599-5_6

  6. Ekechukwu END, Nzeakuba IC, Dada OO, Nwankwo KO, Olowoyo P, Utti VA, Owolabi MO (2021) Virtual reality, a neuroergonomic and neurorehabilitation tool for promoting neuroplasticity in stroke survivors: a systematic review with meta-analysis. In: Congress of the international ergonomics association. Springer, Cham, pp 495–508. https://doi.org/10.1007/978-3-030-74614-8_64

  7. Homayounpour M, Butter D, Vasta S, Merryweather A (2021). Validation of an inverse kinematic VR manikin in seated tasks: application in ergonomics training. In: Congress of the international ergonomics association. Springer, Cham, pp 367–373. https://doi.org/10.1007/978-3-030-74614-8_45

  8. Jiménez MC, James AM, Maureira MAG, Kniestedt I (2017) Dreadful virtualities: a comparative case study of player responses to a horror game in virtual reality and flat screen. In: International conference on advances in computer entertainment. Springer, Cham, pp 239–260. https://doi.org/10.1007/978-3-319-76270-8_17

  9. Lee J (2020) Problem-based gaming via an augmented reality mobile game and a printed game in foreign language education. Educ Inf Technol 27:1–29. https://doi.org/10.1007/s10639-020-10391-1

    Article  Google Scholar 

  10. Echtler F, Sturm F, Kindermann K, Klinker G, Stilla J, Trilk J, Najafi H (2004) The intelligent welding gun: augmented reality for experimental vehicle construction. In: Ong SK, Nee AYC (eds) Virtual and augmented reality applications in manufacturing. Springer, London, pp 333–360. https://doi.org/10.1007/978-1-4471-3873-0_17

    Chapter  Google Scholar 

  11. Inoue Y (2012) Virtual reality learning environments. In: Seel NM (ed) Encyclopedia of the sciences of learning. Springer, Boston. https://doi.org/10.1007/978-1-4419-1428-6_651

    Chapter  Google Scholar 

  12. Dhaya R (2020) Improved image processing techniques for user immersion problem alleviation in virtual reality environments. J Innov Image Process 2(02):77–84. https://doi.org/10.36548/jiip.2020.2.002

    Article  Google Scholar 

  13. Berna-Moya JL, Martinez-Plasencia D (2019) Exploring the effects of replicating shape, weight and recoil effects on VR shooting controllers. In: IFIP conference on human–computer interaction. Springer, Cham, pp 763–782. https://doi.org/10.1007/978-3-030-29381-9_45

  14. Jia D, Hermans A, Leibe B (2020) DR-SPAAM: a spatial-attention and auto-regressive model for person detection in 2D range data. In: 2020 IEEE/RSJ international conference on intelligent robots and systems (IROS). IEEE, pp 10270–10277. https://doi.org/10.1109/IROS45743.2020.9341689

  15. Hsieh MC, Lee JJ (2018) Preliminary study of VR and AR applications in medical and healthcare education. J Nurs Health Stud 3(1):12. https://doi.org/10.21767/2574-2825.100030

    Article  Google Scholar 

  16. Nilsson NC, Serafin S, Steinicke F, Nordahl R (2018) Natural walking in virtual reality: a review. Comput Entertain 16(2):1–22. https://doi.org/10.1145/3180658

    Article  Google Scholar 

  17. Habgood MJ, Moore D, Wilson D, Alapont S (2018) Rapid, continuous movement between nodes as an accessible virtual reality locomotion technique. In: 2018 IEEE conference on virtual reality and 3D user interfaces (VR). IEEE, pp 371–378. https://doi.org/10.1109/VR.2018.8446130

  18. Krompiec P, Park K (2019) Enhanced player interaction using motion controllers for first-person shooting games in virtual reality. IEEE Access 7:124548–124557. https://doi.org/10.1109/ACCESS.2019.2937937

    Article  Google Scholar 

  19. Saredakis D, Szpak A, Birckhead B, Keage HA, Rizzo A, Loetscher T (2020) Factors associated with virtual reality sickness in head-mounted displays: a systematic review and meta-analysis. Front Hum Neurosci 14:96. https://doi.org/10.1089/g4h.2017.0190

    Article  Google Scholar 

  20. Farmani Y, Teather RJ (2018) Viewpoint snapping to reduce cybersickness in virtual reality. In: Proceedings of the 44th graphics interface conference, pp 168–175. https://doi.org/10.20380/GI2018.23

  21. Yildirim C (2020) Don’t make me sick: investigating the incidence of cybersickness in commercial virtual reality headsets. Virtual Real 24(2):231–239. https://doi.org/10.1007/s10055-019-00401-0

    Article  MathSciNet  Google Scholar 

  22. Han DT, Suhail M, Ragan ED (2018) Evaluating remapped physical reach for hand interactions with passive haptics in virtual reality. IEEE Trans Vis Comput Graph 24(4):1467–1476. https://doi.org/10.1109/TVCG.2018.2794659

    Article  Google Scholar 

  23. Winter C, Kern F, Gall D et al (2021) Immersive virtual reality during gait rehabilitation increases walking speed and motivation: a usability evaluation with healthy participants and patients with multiple sclerosis and stroke. J Neuroeng Rehabil 18:68. https://doi.org/10.1186/s12984-021-00848-w

    Article  Google Scholar 

  24. Andersen K, Gaab SJ, Sattarvand J, Harris FC (2020) METS VR: mining evacuation training simulator in virtual reality for underground mines. In: 17th international conference on information technology—new generations (ITNG 2020). Springer, Cham, pp 325–332. https://doi.org/10.1007/978-3-030-43020-7_43

  25. Bouguila L, Evequoz F, Courant M, Hirsbrunner B (2004) Walking-pad: a step-in-place locomotion interface for virtual environments. In: Proceedings of the 6th international conference on multimodal interfaces, pp 77–81. https://doi.org/10.1145/1027933.1027948

  26. Solanki D, Kumar S, Raj P, Lahiri U (2019) Body weight support assisted virtual reality based treadmill walk with gait characterization. In: 2019 10th international conference on computing, communication and networking technologies (ICCCNT). IEEE, pp 1–7. https://doi.org/10.1109/ICCCNT45670.2019.8944611

  27. Lidia-Cristina B, Horatiu R, Nicu-George B (2021) Design of improved mechatronic system for Virtusphere. In: 2021 international conference on applied and theoretical electricity (ICATE). IEEE, pp 1–4. https://doi.org/10.1109/ICATE49685.2021.9464938

  28. Bozgeyikli E (2018) Locomotion in virtual reality video games. In: Lee N (ed) Encyclopedia of computer graphics and games. Springer, Cham. https://doi.org/10.1007/978-3-319-08234-9_186-1

    Chapter  Google Scholar 

  29. Kemeny A, Chardonnet JR, Colombet F (2020) Visualization and motion systems. In: Getting rid of cybersickness. Springer, Cham. https://doi.org/10.1007/978-3-030-59342-1_3

  30. Warren LE, Bowman DA (2017) User experience with semi-natural locomotion techniques in virtual reality: the case of the Virtuix Omni. In: Proceedings of the 5th symposium on spatial user interaction, pp 163–163. https://doi.org/10.1145/3131277.3134359

  31. Obukhov A, Siukhin A, Dedov D (2020) The model of the automatic control system for a treadmill based on neural networks. In: 2020 international multi-conference on industrial engineering and modern technologies (FarEastCon). IEEE, pp 1–5. https://doi.org/10.1109/FarEastCon50210.2020.9271589

  32. Mittal R, Karre SA, Reddy YR (2021) Designing limitless path in virtual reality environment. In: Chen JYC, Fragomeni G (eds) Virtual, augmented and mixed reality. HCII 2021. Lecture notes in computer science, vol 12770. Springer, Cham. https://doi.org/10.1007/978-3-030-77599-5_7

    Chapter  Google Scholar 

  33. De Regt A, Barnes SJ, Plangger K (2020) The virtual reality value chain. Bus Horiz 63(6):737–748. https://doi.org/10.1016/j.bushor.2020.08.002

    Article  Google Scholar 

  34. Auralius M, Yoon JW (2011) An automatic speed control system of a treadmill with ultrasonic sensors. J Inst Control Robot Syst 17(5):505–511. https://doi.org/10.5302/J.ICROS.2011.17.5.505

    Article  Google Scholar 

  35. Qi Y, Soh CB, Gunawan E, Low KS (2014) Ambulatory measurement of three-dimensional foot displacement during treadmill walking using wearable wireless ultrasonic sensor network. IEEE J Biomed Health Inform 19(2):446–452. https://doi.org/10.1109/JBHI.2014.2316998

    Article  Google Scholar 

  36. Gembalczyk G, Duda S, Kciuk S, Gąsiorek D, Mężyk A (2019) Mechatronic treadmill for gait reeducation with control algorithm of treadmill speed adaptation. Proc Inst Mech Eng C J Mech Eng Sci 233(7):2239–2247. https://doi.org/10.1177/0954406218790279

    Article  Google Scholar 

  37. Wu M, George Hornby T, Landry JM, Roth H, Schmit BD (2011) A cable-driven locomotor training system for restoration of gait in human SCI. Gait Posture 33(2):256–260. https://doi.org/10.1016/j.gaitpost.2010.11.016

    Article  Google Scholar 

  38. Caserman P, Garcia-Agundez A, Konrad R, Göbel S, Steinmetz R (2019) Real-time body tracking in virtual reality using a Vive tracker. Virtual Real 23(2):155–168. https://doi.org/10.1007/s10055-018-0374-z

    Article  Google Scholar 

  39. Yoon J, Asl H, Pyo S (2018) Intelligent control scheme to facilitate abrupt stopping on self-adjustable treadmills. In: IEEE international conference on robotics and automation, pp 1639–1644. https://doi.org/10.1109/ICRA.2018.8462897

  40. Kamkar S, Ghezloo F, Moghaddam HA, Borji A, Lashgari R (2020) Multiple-target tracking in human and machine vision. PLoS Comput Biol. https://doi.org/10.1371/journal.pcbi.1007698

    Article  Google Scholar 

  41. Barris S, Button C (2008) A review of vision-based motion analysis in sport. Sports Med 38:1025–1043. https://doi.org/10.2165/00007256-200838120-00006

    Article  Google Scholar 

  42. Rosdi MHAB, Ghani ASA (2022) investigation on accuracy of sensors in sensor fusion for object detection of autonomous vehicle based on 2D lidar and ultrasonic sensor. In: Recent trends in mechatronics towards industry 4.0. Springer, Singapore, pp 761–770.https://doi.org/10.1007/978-981-33-4597-3_68

  43. Gauglitz G (2005) Direct optical sensors: principles and selected applications. Anal Bioanal Chem 381:141–155. https://doi.org/10.1007/s00216-004-2895-4

    Article  Google Scholar 

  44. Ikbal MS, Ramadoss V, Zoppi M (2020) Dynamic pose tracking performance evaluation of HTC vive virtual reality system. IEEE Access 9:3798–3815. https://doi.org/10.1109/ACCESS.2020.3047698

    Article  Google Scholar 

  45. Wang L, Hu W, Tan T (2003) Recent developments in human motion analysis. Pattern Recogn 36(3):585–601. https://doi.org/10.1016/S0031-3203(02)00100-0

    Article  Google Scholar 

  46. Tsutsui H, Miura J, Shirai Y (2001) Optical flow-based person tracking by multiple cameras. In: Conference documentation international conference on multisensor fusion and integration for intelligent systems. MFI 2001 (Cat. No. 01TH8590). IEEE, pp 91–96. https://doi.org/10.1109/MFI.2001.1013514.

  47. Wang X, Garg S, Tran SN, Bai Q, Alty J (2021) Hand tremor detection in videos with cluttered background using neural network based approaches. Health Inf Sci Syst 9(1):1–14. https://doi.org/10.1007/s13755-021-00159-3

    Article  Google Scholar 

  48. Wu Y, Lin Q, Yang M, Liu J, Tian J, Kapil D, Vanderbloemen L (2022) A computer vision-based yoga pose grading approach using contrastive skeleton feature representations. Healthcare 10(1):36. https://doi.org/10.3390/healthcare10010036

    Article  Google Scholar 

  49. Cao Z, Simon T, Wei SE, Sheikh Y (2017) Realtime multi-person 2d pose estimation using part affinity fields. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 7291–7299. https://doi.org/10.1109/CVPR.2017.143

  50. Pyo S, Lee H, Yoon J (2021) Development of a novel omnidirectional treadmill-based locomotion interface device with running capability. Appl Sci 11(9):4223

    Article  Google Scholar 

  51. Souman JL, Giordano PR, Frissen I, Luca AD, Ernst MO (2010) Making virtual walking real: perceptual evaluation of a new treadmill control algorithm. ACM Trans Appl Percept 7(2):1–14

    Article  Google Scholar 

  52. Wang W, Yang K, Zhu Y, Mu H (2020) Speed adaptation and acceleration ripple suppression of treadmill user system using a virtual force moment balance model. Trans Inst Meas Control 42(2):322–329. https://doi.org/10.1177/0142331219868924

    Article  Google Scholar 

  53. Bowtell MV, Tan H, Wilson AM (2009) The consistency of maximum running speed measurements in humans using a feedback-controlled treadmill, and a comparison with maximum attainable speed during overground locomotion. J Biomech 42(15):2569–2574. https://doi.org/10.1016/j.jbiomech.2009.07.024

    Article  Google Scholar 

  54. Zhou Y, Chen S (2020) Fuzzy control for treadmill permanent magnet synchronous motor speed system. In: 2020 5th international conference on advanced robotics and mechatronics (ICARM). IEEE, pp 646–651. https://doi.org/10.1109/ICARM49381.2020.9195339

  55. Weng K, Turk B, Dolores L, Nguyen TN, Celler B, Su S, Nguyen HT (2010) Fast tracking of a given heart rate profile in treadmill exercise. In: 2010 annual international conference of the IEEE engineering in medicine and biology. IEEE, pp 2569–2572. https://doi.org/10.1109/IEMBS.2010.5626650

  56. Kim J, Chang M, Jeon D (2018) The AI supervisor for the effective treadmill training system of rehabilitation and exercise. In: International conference on NeuroRehabilitation. Springer, Cham, pp 195–199. https://doi.org/10.1007/978-3-030-01845-0_39

  57. Li L, Luo M (2020) Research on an improved single neuron PI control strategy. In: 2020 international conference on computer network, electronic and automation (ICCNEA). IEEE, pp 409–413. : https://doi.org/10.1109/ICCNEA50255.2020.00090

  58. Manju A, Valarmathie P (2021) Video analytics for semantic substance extraction using OpenCV in python. J Ambient Intell Humaniz Comput 12(3):4057–4066. https://doi.org/10.1007/s12652-020-01780-y

    Article  Google Scholar 

  59. Brdjanin A, Dardagan N, Dzigal D, Akagic A (2020) Single object trackers in OpenCV: a benchmark. In: 2020 international conference on INnovations in Intelligent SysTems and Applications (INISTA). IEEE, pp 1–6

  60. Hartley R, Li H (2020) Stereovision. In: Ang M, Khatib O, Siciliano B (eds) Encyclopedia of robotics. Springer, Berlin. https://doi.org/10.1007/978-3-642-41610-1_100-1

    Chapter  Google Scholar 

  61. Ahmadyan A, Hou T, Wei J, Zhang L, Ablavatski A, Grundmann M (2020) Instant 3D object tracking with applications in augmented reality. arXiv preprint arXiv:2006.13194.

  62. Kim S-h, Hwang Y (2021) A survey on deep learning based methods and datasets for monocular 3D object detection. Electronics 10(4):517. https://doi.org/10.3390/electronics10040517

    Article  Google Scholar 

  63. Ahmadyan A, Zhang L, Ablavatski A, Wei J, Grundmann M (2021) Objectron: a large scale dataset of object-centric videos in the wild with pose annotations. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 7822–7831. https://doi.org/10.1109/CVPR46437.2021.00773

  64. Obukhov AD, Krasnyanskiy MN (2021) Neural network method for automatic data generation in adaptive information systems. Neural Comput Appl. https://doi.org/10.1007/S00521-021-06169-2

    Article  Google Scholar 

Download references

Acknowledgements

The work was carried out with the support of the Laboratory of medical VR simulator systems for training, diagnostics and rehabilitation of Tambov State Technical University.

Author information

Authors and Affiliations

  1. Tambov State Technical University, Tambov, Russia

    Artem D. Obukhov, Mikhail N. Krasnyanskiy, Denis L. Dedov, Victoria V. Vostrikova, Daniil V. Teselkin & Ekaterina O. Surkova

Authors
  1. Artem D. Obukhov
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Mikhail N. Krasnyanskiy
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Denis L. Dedov
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Victoria V. Vostrikova
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Daniil V. Teselkin
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Ekaterina O. Surkova
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Artem D. Obukhov.

Ethics declarations

Conflict of interest

The authors declare that there is no conflict of interest regarding the publication of this article.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Obukhov, A.D., Krasnyanskiy, M.N., Dedov, D.L. et al. Control of adaptive running platform based on machine vision technologies and neural networks. Neural Comput & Applic 34, 12919–12946 (2022). https://doi.org/10.1007/s00521-022-07166-9

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00521-022-07166-9

Keywords

Search

Navigation