Skip to main content
SpringerLink
Log in

Exploiting virtual reality and the robot operating system to remote-control a humanoid robot

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

The availability of frameworks and applications in the robotic domain fostered in the last years a spread in the adoption of robots in daily life activities. Many of these activities include the robot teleoperation, i.e. controlling its movements remotely. Virtual Reality (VR) demonstrated its effectiveness in lowering the skill barrier for such a task. This paper discusses the engineering and implementation of a general-purpose, open-source framework for teleoperating a humanoid robot through a VR headset. It includes a VR interface for articulating different robot actions using the VR controllers, without the need for training. Besides, it exploits the Robot Operating System (ROS) for the control and synchronization of the robot hardware, the distribution of the computation and its scalability. The framework supports the extension for operating other types of robots and using different VR configurations. We carried out a user experience evaluation with twenty users using System Usability Scale questionnaires and with six stakeholders on five different scenarios using the Software Architecture Analysis Method.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9

Similar content being viewed by others

Notes

  1. https://qz.com/1383665/social-robots-will-become-family-members-in-the-homes-of-the-future/

  2. https://en.wikibooks.org/wiki/Robotics/Types_of_Robots/Wheeled

  3. https://www.softbankrobotics.com/emea/en/nao

  4. https://asimo.honda.com/

  5. https://www.bostondynamics.com/atlas

  6. http://www.icub.org/

  7. https://robots.ieee.org/robots/valkyrie/

  8. http://news.mit.edu/2017/mit-csail-new-system-teleoperating-robots-virtual-reality-1009

  9. https://unity.com/

  10. https://www.unrealengine.com/en-US/

  11. https://www.ros.org/

  12. https://www.oculus.com/rift/

  13. http://wiki.ros.org/rosbridge_suite

  14. http://wiki.ros.org/nao_robot

  15. http://doc.aldebaran.com/2-4/software/choregraphe/

  16. The Choreagraphe suite guarantees compatibility with other SoftBank robots. It supports NAO and also the Pepper robots except for the latest version, which uses qiskd instead https://developer.softbankrobotics.com/nao6/naoqi-developer-guide/getting-started.

  17. https://developer.oculus.com/documentation/unity/unity-utilities-overview/

  18. http://doc.aldebaran.com/2-1/naoqi/motion/control-cartesian-api.html

  19. https://github.com/triguz/Nao_RealTime_Control.

  20. https://www.youtube.com/watch?v=J8I892TnaIA

References

  1. Ajili I, Mallem M, Didier J-Y (2017) Gesture recognition for humanoid robot teleoperation, pp 115–1120. https://doi.org/10.1109/ROMAN.2017.8172443. https://www.scopus.com/inward/record.uri?eid=2-s2.0-85045843514&doi=10.1109%2fROMAN.2017.8172443&partnerID=40&md5=29010026d05e5013037fd01834458106

  2. Ali Babar M, Gorton I (2004) Comparison of scenario-based software architecture evaluation methods, pp 600–607. https://doi.org/10.1109/APSEC.2004.38

  3. Alonso R, Concas E, Recupero DR (2019) A flexible and scalable social robot architecture employing voice assistant technology. In: Workshop on Adapted Interaction with Social Robots. https://caesar2020.di.unito.it/pdf/cAESAR2020_paper_10.pdf

  4. Atzeni M, Recupero DR (2018) Deep learning and sentiment analysis for human-robot interaction. In: The Semantic Web: ESWC 2018 Satellite Events - ESWC 2018 Satellite Events, Heraklion, Revised Selected Papers, pp 14–18. https://doi.org/10.1007/978-3-319-98192-5_3

  5. Atzeni M, Recupero DR (2020) Multi-domain sentiment analysis with mimicked and polarized word embeddings for human-robot interaction. Fut Gener Comput Syst 110:984–999. https://doi.org/10.1016/j.future.2019.10.012

    Article  Google Scholar 

  6. Babington P (2020) Robot operating system (ros), 4. Springer

  7. Bass L, Clements P, Kazman R (2003) Software architecture in practice

  8. Berg J, Lu S (2020) Review of interfaces for industrial human-robot interaction. Current Robot Rep 1(2):27–34. https://doi.org/10.1007/s43154-020-00005-6

    Article  Google Scholar 

  9. Brooke J (1996) SUS-A quick and dirty usability scale. Usability Eval Industry 189(194):4–7

    Google Scholar 

  10. Clements P (2000) Active reviews for intermediate designs, pp 26

  11. Dahl TS, Boulos MNK (2014) Robots in health and social care: a complementary technology to home care and telehealthcare?. Robotics 3(1):1–21. https://doi.org/10.3390/robotics3010001, https://www.mdpi.com/2218-6581/3/1/1

    Article  Google Scholar 

  12. Dehnavi S, Sedaghatbaf A, Salmani B, Sirjani M, Kargahi M, Khamespanah E (2019) Towards an actor-based approach to design verified ros-based robotic programs using rebeca. Procedia Comput Sci 155:59–68. https://doi.org/10.1016/j.procs.2019.08.012, http://www.sciencedirect.com/science/article/pii/S1877050919309263. The 16th International Conference on Mobile Systems and Pervasive Computing (MobiSPC 2019), The 14th International Conference on Future Networks and Communications (FNC-2019), The 9th International Conference on Sustainable Energy Information Technology

    Article  Google Scholar 

  13. Elbasiony R, Gomaa W (2018) Humanoids skill learning based on real-time human motion imitation using kinect. Intel Serv Robot 11(2):149–169. https://doi.org/10.1007/s11370-018-0247-z, https://www.scopus.com/inward/record.uri?eid=2-s2.0-85042100893&doi=10.%1007%2fs11370-018-0247-z&partnerID=40&md5=3932f5591b3ab0ba75ae761c4e3dba2c. cited By 5

    Article  Google Scholar 

  14. Franzluebbers A, Johnsen K (2019) Remote robotic arm teleoperation through virtual reality. https://www.scopus.com/inward/record.uri?eid=2-s2.0-85077022654&doi=10.%1145%2f3357251.3359444&partnerID=40&md5=58f5ad5120dbc91ff9bbb318d90e743b. cited By 0

  15. Franzluebbers A, Johnson K (2019) Remote robotic arm teleoperation through virtual reality, p1–2

  16. Gerina F, Massa SM, Moi F, Recupero DR, Riboni D (2020) Recognition of cooking activities through air quality sensor data for supporting food journaling. HCIS 10:27. https://doi.org/10.1186/s13673-020-00235-9

    Google Scholar 

  17. Gerina F, Pes B, Recupero DR, Riboni D (2019) Toward supporting food journaling using air quality data mining and a social robot. In: Chatzigiannakis I, de Ruyter B E R, Mavrommati I (eds) ambient intelligence - 15th European conference, AmI 2019, proceedings, lecture notes in computer science. https://doi.org/10.1007/978-3-030-34255-5_22, vol 11912. Springer, Rome, pp 318–323

  18. Gharaybeh Z, Chizeck H, Stewart A (2019) Telerobotic control in virtual reality. https://www.scopus.com/inward/record.uri?eid=2-s2.0-85079076874&doi=10.%23919%2fOCEANS40490.2019.8962616&partnerID=40&md5=3a7f6c5f3929328e525f960392e3%88bb cited By 0

  19. Hashimoto S, Ishida A, Inami M, Igarashi T (2013) Touchme: an augmented reality interface for remote robot control. J Robot Mechatron 25(3):529–537. https://doi.org/10.20965/jrm.2013.p0529

    Article  Google Scholar 

  20. Hetrick R, Amerson N, Kim B, Rosen E, Visser EJD, Phillips E (2020) Comparing virtual reality interfaces for the teleoperation of robots. https://www.scopus.com/inward/record.uri?eid=2-s2.0-85087088497&doi=10.%1109%2fSIEDS49339.2020.9106630&partnerID=40&md5=3a192ea1bfb5e6cd6e05ae43d934e0%98. cited By 0

  21. Hong S, Park G, Lee Y, Lee W, Choi B, Sim O, Oh J-H (2018) Development of a tele-operated rescue robot for a disaster response. Int J Human Robot 15(04):1850008. https://doi.org/10.1142/S0219843618500081

    Article  Google Scholar 

  22. Ismail LI, Shamsudin S, Yussof H, Hanapiah FA, Zahari NI (2012) Robot-based intervention program for autistic children with humanoid robot nao: initial response in stereotyped behavior. Procedia Eng 41:1441–1447. https://doi.org/10.1016/j.proeng.2012.07.333, http://www.sciencedirect.com/science/article/pii/S1877705812027336. International Symposium on Robotics and Intelligent Sensors 2012 (IRIS 2012)

    Article  Google Scholar 

  23. Kasahara S, Niiyama R, Heun V, Ishii H (2013) Extouch: Spatially-aware embodied manipulation of actuated objects mediated by augmented reality. In: In: Proceedings of the 7th International Conference on Tangible, Embedded and Embodied Interaction. ACM, pp 223–228

  24. Kazman R, Bass L, Abowd G, Webb M (1994) Saam: a method for analyzing the properties of software architectures. In: Proceedings of 16th International Conference on Software Engineering, pp 81–90

  25. Kazman R, Klein M, Barbacci M, Longstaff T, Lipson H, Carrière S (1998) The architecture tradeoff analysis method, pp 68–78

  26. Lewis JR, Sauro J (2018) Item benchmarks for the system usability scale. J Usability Stud 13(3)

  27. Li C, Yang C, Liang P, Cangelosi A, Wan J (2016) Development of kinect based teleoperation of nao robot. 133–138. https://www.scopus.com/inward/record.uri?eid=2-s2.0-84998812266&doi=10.%1109%2fICARM.2016.7606908&partnerID=40&md5=9bed4eef953668f855dc76dd689ab9d3. cited By 11

  28. Makhataeva Z, Varol HA (2020) Augmented reality for robotics: A review. Robotics 9(2). https://doi.org/10.3390/robotics9020021, https://www.mdpi.com/2218-6581/9/2/21

  29. Matarić MJ, Scassellati B (2016) Socially assistive robotics. Springer Handbook of Robotics. Springer International Publishing, Cham, pp 1973–1994. https://doi.org/10.1007/978-3-319-32552-1_73

  30. Melchiorri C (2014) Robot teleoperation. Encyclopedia of Systems and Control, pp 1–14

  31. Melinte O, Vladareanu L, Munteanu L, Yu H, Cang S, Hou Z-G, Bian G-B, Wang H (2015) Haptic intelligent interfaces for nao robot hand control. https://www.scopus.com/inward/record.uri?eid=2-s2.0-84959307255&doi=10.1109%2fICAMechS.2015.7287127&partnerID=40&md5=430767b73ba56676e51c52397191bd1f. cited By 4

  32. Munoz J-M, Avalos J, Ramos OE (2017) Image-driven drawing system by a nao robot, pp pp 1–4. https://www.scopus.com/inward/record.uri?eid=2-s2.0-85049597955&doi=10.1109%2fECON.2017.8247303&partnerID=40&md5=5ad8ff0c502c4ee27f9e4289df38c812. cited By 2

  33. Naceri A, Mazzanti D, Bimbo J, Prattichizzo D, Caldwell DG, Mattos LS, Deshpande N (2019) Towards a virtual reality interface for remote robotic teleoperation. pp 284–289. https://www.scopus.com/inward/record.uri?eid=2-s2.0-85084284613&doi=10.1109%2fICAR46387.2019.8981649&partnerID=40&md5=02e9ec75ce525e44e45235cb7611be2d. cited By 0

  34. Nunez L, Dajles D, Siles F (2018) Teleoperation of a humanoid robot using an optical motion capture system. https://www.scopus.com/inward/record.uri?eid=2-s2.0-85054515350&doi=10.%1109%2fIWOBI.2018.8464136&partnerID=40&md5=8a5aa86b12e934608066de2f28484ee6, cited By 0

  35. Quigley M, Conley K, Gerkey B, Faust J, Foote T, Leibs J, Wheeler R, Ng A (2009) Ros: an open-source robot operating system 3

  36. Recupero DR, Dessì D, Concas E (2019) A flexible and scalable architecture for human-robot interaction. In: Chatzigiannakis I, de Ruyter BER, Mavrommati I (eds) ambient intelligence - 15th European conference, AmI 2019, proceedings, lecture notes in computer science. https://doi.org/10.1007/978-3-030-34255-5_21, vol 11912. Springer, Rome, pp 311–317

  37. Reforgiato Recupero D, Spiga F (2019) Knowledge acquisition from parsing natural language expressions for humanoid robot action commands. Inf Process Manag:102094. https://doi.org/10.1016/j.ipm.2019.102094, http://www.sciencedirect.com/science/article/pii/S0306457319303322

  38. Robles-Bykbaev V, Ochoa-Guaraca M, Carpio-Moreta M, Pulla-Sánchez D, Serpa-Andrade L, López-Nores M, García-Duque J (2016) Robotic assistant for support in speech therapy for children with cerebral palsy. In: 2016 IEEE International Autumn Meeting on Power, Electronics and Computing (ROPEC), pp 1–6

  39. Rodriguez I, Astigarraga A, Jauregi E, Ruiz T, Lazkano E (2014) Humanizing nao robot teleoperation using ros. In: 2014 IEEE-RAS International Conference on Humanoid Robots, pp 179–186

  40. Roldán Gómez J, Peña-Tapia E, Garzón Ramos D, De León Rivas J, Garzon M, Cerro J, Barrientos A (2019) Multi-robot systems, virtual reality and ros: developing a new generation of operator interfaces. Stud Comput Intell 778:29–64. https://doi.org/10.1007/978-3-319-91590-6_2, https://www.scopus.com/inward/record.uri?eid=2-s2.0-85049667675&doi=10.1007%2f978-3-319-91590-6_2&partnerID=40&md5=cee6d70ae6cf9801f6c2a23d63e9e062, cited By 10

    Article  Google Scholar 

  41. Sani AYM, He T, Zhao W, Yao T (2019) Hybrid underwater robot system based on ros, pp 396–400. https://www.scopus.com/inward/record.uri?eid=2-s2.0-85076638378&doi=10.%1145%2f3366194.3366264&partnerID=40&md5=d65f2cad6555b078f5bfb45f2d6bbdb8, cited By 0

  42. Schmidt L, Hegenberg J, Cramar L (2014) User studies on teleoperation of robots for plant inspection. Ind Robot: Int J:41. https://doi.org/10.1108/IR-02-2013-325

  43. Sun D, Kiselev A, Liao Q, Stoyanov T, Loutfi A (2020) A new mixed-reality-based teleoperation system for telepresence and maneuverability enhancement. IEEE Trans Mach Syst 50 (1):55–67. https://doi.org/10.1109/THMS.2019.2960676, https://www.scopus.com/inward/record.uri?eid=2-s2.0-85077905008&doi=10.%1109%2fTHMS.2019.2960676&partnerID=40&md5=99b140db47ca7254bcfeb78a1842de76, cited By 1

    Article  Google Scholar 

  44. (2020). Unity Technologies: Unity Game Engine. https://unity.com/, Accessed: 2020-07-16

  45. Vircikova M, Sincak P (2013) Experience with the children-humanoid interaction in rehabilitation therapy for spinal disorders. In: Kim J-H, Matson ET, Myung H, Xu P (eds) robot intelligence technology and applications 2012: an edition of the presented papers from the 1st international conference on robot intelligence technology and applications. https://doi.org/10.1007/978-3-642-37374-9_34. Springer, Berlin, pp 347–357

  46. Walker ME, Hedayati H, Szafir D (2019) Robot teleoperation with augmented reality virtual surrogates. In: 2019 14th ACM/IEEE International Conference on Human-Robot Interaction (HRI), pp 202–210

  47. Wang Q, Cheng Y, Jiao W, Johnson MT, Zhang Y (2019) Virtual reality human-robot collaborative welding: a case study of weaving gas tungsten arc welding. J Manuf Process 48:210–217. https://doi.org/10.1016/j.jmapro.2019.10.016, https://www.scopus.com/inward/record.uri?eid=2-s2.0-85074943614&doi=10.%1016%2fj.jmapro.2019.10.016&partnerID=40&md5=738261f2802366ba5de7e48b7cd915e7, cited By 1

    Article  Google Scholar 

  48. Yuan F, Zhang L, Zhang H, Li D, Zhang T (2019) Distributed teleoperation system for controlling heterogeneous robots based on ros. pp 7–12. https://www.scopus.com/inward/record.uri?eid=2-s2.0-85078359983&doi=10.1109%2fARSO46408.2019.8948758&partnerID=40&md5=73998ede0142a983ca9119b592974e41, cited By 0

Download references

Acknowledgements

This research was partially funded by the EU’s Marie Curie training network PhilHumans - Personal Health Interfaces Leveraging HUman-MAchine Natural interactionS (grant number 812882).

Author information

Authors and Affiliations

  1. R2M Solution s.r.l., Via Fratelli Cuzio, 42, 27100, Pavia, Italy

    Ruben Alonso

  2. Department of Mathematics and Computer Science, University of Cagliari, Via Ospedale 72, 09124, Cagliari, Italy

    Alessandro Bonini, Diego Reforgiato Recupero & Lucio Davide Spano

Authors
  1. Ruben Alonso
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Alessandro Bonini
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Diego Reforgiato Recupero
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Lucio Davide Spano
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Ruben Alonso.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Alonso, R., Bonini, A., Reforgiato Recupero, D. et al. Exploiting virtual reality and the robot operating system to remote-control a humanoid robot. Multimed Tools Appl 81, 15565–15592 (2022). https://doi.org/10.1007/s11042-022-12021-z

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-022-12021-z

Keywords

Search

Navigation