Abstract
This paper presents the preliminary design of a teleoperation system for a bimanual bartending robot, with reference to the BRILLO (Bartending Robot for Interactive Long Lasting Operations) project. The aim is to simulate the remote control of the robotic bartender by the human operator in an intuitive manner, using Virtual Reality technologies. The proposed Virtual Reality architecture is based on the use of commercial Head Mounted Display with a pair of hand controllers and the virtual simulation of the remote environment of the robot, with the robotic simulator CoppeliaSim. Originally, virtual simulations of the robot environment have allowed to identify the possible scenarios and interactions between the customers and the different robotic systems inside the automatized bar: the totem for the selection and payment of the order, the robotic bartender to prepare the cocktail and the mobile robot for the cocktail serving at the table. Secondly, focusing on a sequence of main tasks that the robotic bartender must perform for the cocktails preparation, the operator’s control on the simulated robotic system has been reproduced. In fact, the aim of this first experimental phase is to test the interaction between the human operator and the simulated immersive environment for the remote control of the robotic system. Two use cases have been reproduced: the first is related to the recovery from a failure situation such as the fall of a glass, while the second refers to the trajectory training to perform some repeating actions. Six operators (three males and three females), who already knew the taks, with an age between 25 and 40 years and a minimum experience with VR technology for personal entertainment, have been involved in the test phase. For this reason, the paper will finally discuss the perception of the involved operators about the use of the proposed VR architecture in terms of usability and mental workload.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Di Gironimo, G., Lanzotti, A.: Designing in VR. Int. J. Interact. Des. Manuf. 3(2), 51–53 (2009)
Gammieri, L., Schumann, M., Pelliccia, L., Di Gironimo, G., Klimant, P.: Coupling of a redundant manipulator with a virtual reality environment to enhance human-robot cooperation. Procedia CIRP 62, 618–623 (2017)
Kuan, C., Young, K.: VR-based teleoperation for robot compliance control. J. Intell. Robot. Syst. Theor. Appl. 30(4), 377–398 (2001)
Ke, S., Xiang, F., Zhang, Z., Zuo, Y.: A enhanced interaction framework based on VR, AR and MR in digital twin. Procedia CIRP 83, 753–758 (2019)
Rosen, R., Von Wichert, G., Lo, G., Bettenhausen, K.D.: About the importance of autonomy and digital twins for the future of manufacturing. IFAC-PapersOnLine 28, 567–572 (2015)
Cichon, T., Rossmann, J.: Simulation-based user interfaces for digital twins: pre-, in-, or post-operational analysis and exploration of virtual testbeds. In: 31st Annual European Simulation and Modelling Conference 2017, ESM 2017, pp. 365–372 (2017)
Bugalia, N., Sen, A., Kalra, P., Kumar, S.: Immersive environment for robotic tele-operation. In: ACM International Conference Proceeding Series, 02–04 July 2015 (2015)
Lipton, J.I., Fay, A.J., Rus, D.: Baxter’s homunculus: virtual reality spaces for teleoperation in manufacturing. IEEE Robot. Autom. Lett. 3(1), 179–186 (2018)
Singh, J., Srinivasan, A.R., Neumann, G., Kucukyilmaz, A.: Haptic-guided teleoperation of a 7-DoF collaborative robot arm with an identical twin master. IEEE Trans. Haptics 13(1), 246–252 (2020)
Mostefa, M., Lahouari, K.E.B., Loukil, A., Mohamed, K., Amine, D.: Design of mobile robot teleoperation system based on virtual reality. In: 3rd International Conference on Control, Engineering and Information Technology, CEIT 2015 (2015)
Mohsen, M., Ibrahim, K., Mohamed, A.M.: Design and control of a two degree of freedom teleoperated manipulator. In: IOP Conference Series: Materials Science and Engineering, vol. 435 (2018)
Whitney, D., Rosen, E., Phillips, E., Konidaris, G., Tellex, S.: Comparing robot grasping teleoperation across desktop and virtual reality with ROS reality. In: Amato, N.M., Hager, G., Thomas, S., Torres-Torriti, M. (eds.) Robotics Research. SPAR, vol. 10, pp. 335–350. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-28619-4_28
Laaki, H., Miche, Y., Tammi, K.: Prototyping a digital twin for real time remote control over mobile networks: application of remote surgery. IEEE Access 7, 20235–20336 (2019)
De Pace, F., Gorjup, G., Bai, H., Sanna, A., Liarokapis, M., Billinghurst, M.: Assessing the suitability and effectiveness of mixed reality interfaces for accurate robot teleoperation. In: Proceedings of the ACM Symposium on Virtual Reality Software and Technology, VRST (2020)
Leeper, A.E., Hsiao, K., Ciocarlie, M., Takayama, L., Gossow, D.: Strategies for human-in-the-loop robotic grasping. In: Proceedings of the 7th Annual ACM/IEEE International Conference on Human-Robot Interaction, HRI 2012, pp. 1–8 (2012)
Oculus Rift. http://www.oculus.com/rift/
Manus VR. https://www.manus-meta.com/
VR Free Glove. https://www.sensoryx.com/product/vrfree-glove-system-cv/
Kobayashi, F., Ikai, G., Fukui, W., Kojima, F.: Two-fingered haptic device for robot hand teleoperation. J. Robot. 2011, 1–8 (2011)
Fischer, M., van der Smagt, P, Hirzinger, G.: Learning techniques in a dataglove based telemanipulation system for the DLR hand (1998)
Mizera, C., Delrieu, T., Weistroffer, V., Andriot, C., Decatoire, A., Gazeau, J.-P.: Evaluation of hand-tracking systems in teleoperation and virtual dexterous manipulation. IEEE Sens. J. 20(3), 1642–1655 (2020)
Leap Motion. https://www.ultraleap.com/product/leap-motion-controller/
Jang, I., Carrasco, J., Weightman, A., Lennox, B.: Intuitive bare-hand teleoperation of a robotic manipulator using virtual reality and leap motion. In: Althoefer, K., Konstantinova, J., Zhang, K. (eds.) TAROS 2019. LNCS (LNAI), vol. 11650, pp. 283–294. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-25332-5_25
Sian, N.E., Yokoi, K., Kajita, S., Kanehiro, F., Tanie, K.: Whole body teleoperation of a humanoid robot - a method of integrating operator’s intention and robot’s autonomy. In: Proceedings of the IEEE International Conference on Robotics and Automation, vol. 2, pp. 1613–1619 (2003)
Rosen, E., Whitney, D., Phillips, E., Ullman, D., Tellex, S.: Testing robot teleoperation using a virtual reality interface with ROS reality. In: Proceedings of the 1st International Workshop on Virtual, Augmented, and Mixed Reality for HRI (VAM-HRI), pp. 1–4 (2018)
LUA Reference Manual. http://www.lua.org/manual/5.3/
KUKA LBR iiwa brochure. https://www.kuka.com/-/media/kuka-downloads/imported/9cb8e311bfd744b4b0eab25ca883f6d3/kuka_lbr_iiwa_brochure_en.pdf?rev=46b7f50a907d44c29dd2c12cae2f92cf &hash=4985B631C9AD8F95387870213AB5D614
CoppeliaSim VR Toolbox. http://github.com/BorisBogaerts/CoppeliaSim-VR-Toolbox
Bogaerts, B., Sels, S., Vanlanduit, S., Penne, R.: Connecting the coppeliasim robotics simulator to virtual reality. SoftwareX 11, 100426 (2020)
Sauro, J.: Sustisfied? Little-known system usability scale facts. https://uxpamagazine.org/sustified/
Acknowledgement
This work has been supported by the BRILLO project (Bartending Robot for Interactive Long-Lasting Operations) which has received funding from the PON I &C 2014-2020 MISE. The authors are solely responsible for the content of this manuscript.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2022 Springer Nature Switzerland AG
About this paper
Cite this paper
Buonocore, S., Grazioso, S., Di Gironimo, G. (2022). Virtual Teleoperation Setup for a Bimanual Bartending Robot. In: De Paolis, L.T., Arpaia, P., Sacco, M. (eds) Extended Reality. XR Salento 2022. Lecture Notes in Computer Science, vol 13446. Springer, Cham. https://doi.org/10.1007/978-3-031-15553-6_22
Download citation
DOI: https://doi.org/10.1007/978-3-031-15553-6_22
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-15552-9
Online ISBN: 978-3-031-15553-6
eBook Packages: Computer ScienceComputer Science (R0)