Abstract
A teleoperation system uses motion interface approaches that allow sending commands remotely to a mobile robotic base. Traditional equipment uses specific components and, many times, stocks on the operator’s body, which may hinder their locomotion and teleoperation. In addition to being limited to teleoperation of only one robotic equipment, not being able to adapt to a multi-robot system. The multi-robot systems are generally used to increase the efficiency of the applications, having two or more robots working together to achieve a goal in a cooperative form, increasing the capacity and robustness of task accomplishments. In this sense, this paper presents the development of a cooperative approach to the teleoperation of multi-robot systems coupled with robot manipulators. This work uses the framework MediaPipe Hands to capture 2D points of the hand of the operator through RGB images combined with depth data through a Kinect sensor to get 3D points of the hand of the operator. These data are used to send positions that the mobile base and the robot manipulators must assume during the teleoperation. Experiments were conducted to demonstrate the algorithm’s efficiency and precision during the realization of object pick-up tests. A video of the experiment was made and can be accessed through the link: https://youtu.be/10cVwcjPYOE.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
Notes
References
Yang, C., Luo, J., Liu, C., Li, M., Dai, S.L.: Haptics electromyography perception and learning enhanced intelligence for teleoperated robot. IEEE Trans. Autom. Sci. Eng. 16(4), 1512–1521 (2018)
Li, J.T., Liu, H.J.: Design optimization of amazon robotics. Autom. Control Intell. Syst. 4(2), 48–52 (2016)
Zhang, W., Cheng, H., Zhao, L., Hao, L., Tao, M., Xiang, C.: A gesture-based teleoperation system for compliant robot motion. Appl. Sci. 9(24), 5290 (2019)
Xue, Z., Chen, X., He, Y., Cao, H., Tian, S.: Gesture-and vision-based automatic grasping and flexible placement in teleoperation. Int. J. Adv. Manuf. Technol. 1–16 (2022)
Martinelli, D., Sousa, A.L., Augusto, M.E., Kalempa, V.C., de Oliveira, A.S., Rohrich, R.F., Teixeira, M.A.: Remote control for mobile robots using gestures captured by the rgb camera and recognized by deep learning techniques. In: 2019 Latin American Robotics Symposium (LARS), 2019 Brazilian Symposium on Robotics (SBR) and 2019 Workshop on Robotics in Education (WRE), pp. 98–103. IEEE (2019)
Motion, L.: Leap motion. San Francisco, CA, USA, (2015)
Cao, Z., Hidalgo, G., Simon, T., Wei, S.E., Sheikh, Y.: OpenPose: realtime multi-person 2D pose estimation using Part Affinity Fields (2018). arXiv:1812.08008
Lugaresi, C., Tang, J., Nash, H., McClanahan, C., Uboweja, E., Hays, M., Zhang, F., Chang, C.L., Yong, M.G., Lee, J., et al.: Mediapipe: a framework for building perception pipelines (2019). arXiv:1906.08172
Khamis, A., Hussein, A., Elmogy, A.: Multi-robot task allocation: a review of the state-of-the-art. Coop. Robots Sensor Netw. 2015, 31–51 (2015)
Fuerstner, I.: Advanced Knowledge Application in Practice. BoD-Books on Demand (2010)
Arkin, R.C., Arkin, R.C., et al.: Behavior-Based Robotics. MIT Press (1998)
Miloradović, B., Çürüklü, B., Ekström, M., Papadopoulos, A.V.: Exploiting parallelism in multi-task robot allocation problems. In: 2021 IEEE International Conference on Autonomous Robot Systems and Competitions (ICARSC), pp. 197–202. IEEE (2021)
Zhu, H., Juhl, J., Ferranti, L., Alonso-Mora, J.: Distributed multi-robot formation splitting and merging in dynamic environments. In: 2019 International Conference on Robotics and Automation (ICRA), pp. 9080–9086. IEEE (2019)
Kalempa, V.C., Piardi, L., Limeira, M., De Oliveira, A.S.: Fault-resilient collective ternary-hierarchical behavior to smart factories. IEEE Access 8, 176905–176915 (2020)
Rover, L.: Leo rover is an outdoor robotics development kit (2022). http://leorover.tech/the-rover
Rootics, T.: Phantomx pincher robot arm (2022). http://trossenrobotics.com/p/PhantomX-Pincher-Robot-Arm.aspx
LLC, G.: Mediapipe holistic. LLC, GOOGLE (2020). http://google.github.io/mediapipe/solutions/hands.html
wiki (2022). http://wiki.ros.org/openni_launch
wiki (2022). http://wiki.ros.org/freenect_launch
OpenKinect: libfreenect. OpenKinect (2020). http://github.com/OpenKinect/libfreenect
Robotics, P.: Moveit (2022). www.moveit.ros.org
Li, C., Wang, S., Zhuang, Y., Yan, F.: Deep sensor fusion between 2d laser scanner and imu for mobile robot localization. IEEE Sens. J. 21(6), 8501–8509 (2019)
Cai, G.S., Lin, H.Y., Kao, S.F.: Mobile robot localization using gps, imu and visual odometry. In: 2019 International Automatic Control Conference (CACS), pp. 1–6. IEEE (2019)
Obreja, S.G., Vulpe, A.: Evaluation of an indoor localization solution based on bluetooth low energy beacons. In: 2020 13th International Conference on Communications (COMM), pp. 227–231. IEEE (2020)
Tong, X., Li, H., Tian, X., Wang, X.: Wi-fi localization enabling self-calibration. IEEE/ACM Trans. Netw. 29(2), 904–917 (2021)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2024 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Martinelli, D., Cerbaro, J., Teixeira, M.A.S., Kalempa, V.C., de Assis Monteiro, V., de Oliveira, A.S. (2024). A Cooperative Approach to Teleoperation Through Gestures for Multi-robot Systems. In: Youssef, E.S.E., Tokhi, M.O., Silva, M.F., Rincon, L.M. (eds) Synergetic Cooperation between Robots and Humans. CLAWAR 2023. Lecture Notes in Networks and Systems, vol 811. Springer, Cham. https://doi.org/10.1007/978-3-031-47272-5_18
Download citation
DOI: https://doi.org/10.1007/978-3-031-47272-5_18
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-47271-8
Online ISBN: 978-3-031-47272-5
eBook Packages: Intelligent Technologies and RoboticsIntelligent Technologies and Robotics (R0)