skip to main content
10.1145/3655532.3655538acmotherconferencesArticle/Chapter ViewAbstractPublication PagesicrsaConference Proceedingsconference-collections
research-article
Open access

ROS-based Augmented and Virtual Reality path planning interface for industrial robotic arms: a preliminary assessment

Published: 28 June 2024 Publication History

Abstract

Optimised human machine interfaces for multi-robot systems are essential for human in the loop in cyber-physical production lines and collaborative systems. The speed of changeover and customized production processes together with the need for straightforward easy and user-friendly human-machine interaction methods demands natural and adaptable interfaces. They should be based on flexible software cores and packages that ease and speed up the development processes. This also applies to laboratory testing and assessment in the academic field, in particular when it comes to the deployment of virtual or augmented reality (AR/VR) interfaces. The standardization of human-robot interaction including control methods in the extended reality domain is a work in progress. It needs broader assessment and clearer metrics to realise efficient and reliable tools. This work presents an Extended Reality (XR) user interface for the control and teleoperation of industrial robots. The systems allows the fast integration of the digital twins of robotic arms and path planning interface in AR and VR using Robot Operating System and Unity. Furthermore, a design-of-experiment involving two different robots (ABB IRB 1200 and ABB IRB 1600) in the two geographically distributed locations is proposed along with some preliminary experimental results.

References

[1]
[n. d.]. ROS-Industrial official website. https://rosindustrial.org/ https://rosindustrial.org/.
[2]
[n. d.]. Unity Robotics Hub. https://github.com/Unity-Technologies/Unity-Robotics-Hub https://github.com/Unity-Technologies/Unity-Robotics-Hub.
[3]
2023. Accelerated, containerized application development. https://www.docker.com/ https://www.docker.com/.
[4]
Thomas Bergs, Sascha Gierlings, Thomas Auerbach, Andreas Klink, Daniel Schraknepper, and Thorsten Augspurger. 2021. The concept of digital twin and digital shadow in manufacturing. Procedia CIRP 101 (2021), 81–84.
[5]
John Brooke 1996. SUS-A quick and dirty usability scale. Usability evaluation in industry 189, 194 (1996), 4–7.
[6]
Sachin Chitta. 2016. MoveIt!: An Introduction. In Studies in Computational Intelligence. Springer International Publishing, 3–27. https://doi.org/10.1007/978-3-319-26054-9_1
[7]
Liang Gong, Åsa Fast-Berglund, and Björn Johansson. 2021. A framework for extended reality system development in manufacturing. IEEE Access 9 (2021), 24796–24813.
[8]
Sandra G Hart and Lowell E Staveland. 1988. Development of NASA-TLX (Task Load Index): Results of empirical and theoretical research. In Advances in psychology. Vol. 52. Elsevier, 139–183.
[9]
Rebecca Hetrick, Nicholas Amerson, Boyoung Kim, Eric Rosen, Ewart J. de Visser, and Elizabeth Phillips. 2020. Comparing Virtual Reality Interfaces for the Teleoperation of Robots. In 2020 Systems and Information Engineering Design Symposium (SIEDS). 1–7. https://doi.org/10.1109/SIEDS49339.2020.9106630
[10]
Kenichi Komatsu, Paula Alavesa, Antti Pauanne, Tuomo Hänninen, Olli Liinamaa, and Ari Pouttu. 2022. Leveraging 5G in cyber-physical system for low-cost robotic telepresence. In 2022 Joint European Conference on Networks and Communications & 6G Summit (EuCNC/6G Summit). IEEE, 399–404.
[11]
Vladimir Kuts, Jeremy A Marvel, Murat Aksu, Simone L Pizzagalli, Martinš Sarkans, Yevhen Bondarenko, and Tauno Otto. 2022. Digital Twin as Industrial Robots Manipulation Validation Tool. Robotics 11, 5 (2022), 113.
[12]
Bettina Laugwitz, Theo Held, and Martin Schrepp. 2008. Construction and Evaluation of a User Experience Questionnaire. USAB 2008 5298, 63–76. https://doi.org/10.1007/978-3-540-89350-9_6
[13]
Donghyeon Lee and Young Soo Park. 2018. Implementation of Augmented Teleoperation System Based on Robot Operating System (ROS). In 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). 5497–5502. https://doi.org/10.1109/IROS.2018.8594482
[14]
Praveen Kumar Reddy Maddikunta, Quoc-Viet Pham, B Prabadevi, Natarajan Deepa, Kapal Dev, Thippa Reddy Gadekallu, Rukhsana Ruby, and Madhusanka Liyanage. 2022. Industry 5.0: A survey on enabling technologies and potential applications. Journal of Industrial Information Integration 26 (2022), 100257.
[15]
Microsoft. [n. d.]. Microsoft/MixedRealityToolkit-Unity: Mixed reality toolkit (MRTK). https://github.com/microsoft/MixedRealityToolkit-Unity https://github.com/microsoft/MixedRealityToolkit-Unity.
[16]
László Monostori. 2014. Cyber-physical production systems: Roots, expectations and R&D challenges. Procedia Cirp 17 (2014), 9–13.
[17]
Dimitris Mourtzis, John Angelopoulos, and Nikos Panopoulos. 2022. Operator 5.0: A survey on enabling technologies and a framework for digital manufacturing based on extended reality. Journal of Machine Engineering 22 (2022).
[18]
Abdeldjallil Naceri, Dario Mazzanti, Joao Bimbo, Domenico Prattichizzo, Darwin G. Caldwell, Leonardo S. Mattos, and Nikhil Deshpande. 2019. Towards a Virtual Reality Interface for Remote Robotic Teleoperation. In 2019 19th International Conference on Advanced Robotics (ICAR). 284–289. https://doi.org/10.1109/ICAR46387.2019.8981649
[19]
Cristina Nuzzi, Stefano Ghidini, Roberto Pagani, Simone Pasinetti, Gabriele Coffetti, and Giovanna Sansoni. 2020. Hands-Free: a robot augmented reality teleoperation system. In 2020 17th International Conference on Ubiquitous Robots (UR). 617–624. https://doi.org/10.1109/UR49135.2020.9144841
[20]
Mikhail Ostanin, Stanislav Zaitsev, Adelia Sabirova, and Alexandr Klimchik. 2022. Interactive Industrial Robot Programming based on Mixed Reality and Full Hand Tracking. IFAC-PapersOnLine 55, 10 (2022), 2791–2796. https://doi.org/10.1016/j.ifacol.2022.10.153 10th IFAC Conference on Manufacturing Modelling, Management and Control MIM 2022.
[21]
Marco Piccinelli, Andrea Gagliardo, Umberto Castellani, and Riccardo Muradore. 2021. Trajectory planning using Mixed Reality: an experimental validation. In 2021 20th International Conference on Advanced Robotics (ICAR). 982–987. https://doi.org/10.1109/ICAR53236.2021.9659448
[22]
Chung Xue Er (Shamaine), Yuansong Qiao, Vladimir Kuts, John Henry, Ken McNevin, and Niall Murray. 2022. Teleoperation of the Industrial Robot: Augmented Reality Application. In Proceedings of the 13th ACM Multimedia Systems Conference (Athlone, Ireland) (MMSys ’22). Association for Computing Machinery, New York, NY, USA, 299–303. https://doi.org/10.1145/3524273.3532901
[23]
J Ernesto Solanes, Adolfo Muñoz, Luis Gracia, Ana Martí, Vicent Girbés-Juan, and Josep Tornero. 2020. Teleoperation of industrial robot manipulators based on augmented reality. The International Journal of Advanced Manufacturing Technology 111 (2020), 1077–1097.
[24]
Theodoros Togias, Christos Gkournelos, Panagiotis Angelakis, George Michalos, and Sotiris Makris. 2021. Virtual reality environment for industrial robot control and path design. Procedia CIRP 100 (2021), 133–138. https://doi.org/10.1016/j.procir.2021.05.021 31st CIRP Design Conference 2021 (CIRP Design 2021).
[25]
Baicun Wang, Pai Zheng, Yue Yin, Albert Shih, and Lihui Wang. 2022. Toward human-centric smart manufacturing: A human-cyber-physical systems (HCPS) perspective. Journal of Manufacturing Systems 63 (2022), 471–490.
[26]
David Whitney, Eric Rosen, Daniel Ullman, Elizabeth Phillips, and Stefanie Tellex. 2018. ROS Reality: A Virtual Reality Framework Using Consumer-Grade Hardware for ROS-Enabled Robots. In 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). 1–9. https://doi.org/10.1109/IROS.2018.8593513
[27]
Huitaek Yun and Martin B.G. Jun. 2022. Immersive and interactive cyber-physical system (I2CPS) and virtual reality interface for human involved robotic manufacturing. Journal of Manufacturing Systems 62 (2022), 234–248. https://doi.org/10.1016/j.jmsy.2021.11.018

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Other conferences
ICRSA '23: Proceedings of the 2023 6th International Conference on Robot Systems and Applications
September 2023
335 pages
ISBN:9798400708039
DOI:10.1145/3655532
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 28 June 2024

Permissions

Request permissions for this article.

Check for updates

Author Tags

  1. Augmented Reality
  2. Design and assessment methods
  3. Human Factors
  4. Human Robot Interaction
  5. User studies
  6. Virtual Reality

Qualifiers

  • Research-article
  • Research
  • Refereed limited

Conference

ICRSA 2023

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • 0
    Total Citations
  • 313
    Total Downloads
  • Downloads (Last 12 months)313
  • Downloads (Last 6 weeks)85
Reflects downloads up to 20 Feb 2025

Other Metrics

Citations

View Options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

HTML Format

View this article in HTML Format.

HTML Format

Login options

Figures

Tables

Media

Share

Share

Share this Publication link

Share on social media