skip to main content
10.1145/3543758.3543775acmotherconferencesArticle/Chapter ViewAbstractPublication PagesmundcConference Proceedingsconference-collections
research-article

Cobity: A Plug-And-Play Toolbox to Deliver Haptics in Virtual Reality

Published: 15 September 2022 Publication History

Abstract

Haptics increase the presence in virtual reality applications. However, providing room-scale haptics is an open challenge. Cobots (robotic systems that are safe for human use) are a promising approach, requiring in-depth engineering skills. Control is done on a low abstraction level and requires complex procedures and implementations. In contrast, 3D tools such as Unity allow to quickly prototype a wide range of environments for which cobots could deliver haptic feedback. To overcome this disconnect, we present Cobity, an open-source plug-and-play solution to control the cobot using the virtual environment, enabling fast prototyping of a wide range of haptic experiences. We present a Unity plugin that allows controlling the cobot using the end-effector’s target pose (cartesian position and angles); the values are then converted into velocities and streamed to the cobot inverse kinematic solver using a specially designed C++ library. Our results show that Cobity enables rapid prototyping with high precision for haptics. We argue that Cobity simplifies the creation of a wide range of haptic feedback applications enabling designers and researchers in human-computer interaction without robotics experience to quickly prototype virtual reality experiences with haptic sensations. We highlight this potential by presenting four different showcases.

References

[1]
Víctor Hugo Andaluz, Fernando A. Chicaiza, Cristian Gallardo, Washington X. Quevedo, José Varela, Jorge S. Sánchez, and Oscar Arteaga. 2016. Unity3D-MatLab Simulator in Real Time for Robotics Applications. In Augmented Reality, Virtual Reality, and Computer Graphics, Lucio Tommaso De Paolis and Antonio Mongelli (Eds.). Springer International Publishing, Cham, 246–263. https://doi.org/10.1007/978-3-319-40621-3_19
[2]
Kentaro Ariga, Masahiro Fujiwara, Yasutoshi Makino, and Hiroyuki Shinoda. 2021. Workspace Evaluation of Long-Distance Midair Haptic Display Using Curved Reflector. In 2021 IEEE World Haptics Conference (WHC). IEEE, Virtual, 85–90. https://doi.org/10.1109/WHC49131.2021.9517193
[3]
Edwin Babaians, Mohsen Tamiz, Yaser Sarfi, Amir Mogoei, and Esmaeil Mehrabi. 2018. ROS2Unity3D; High-Performance Plugin to Interface ROS with Unity3d engine. In 2018 9th Conference on Artificial Intelligence and Robotics and 2nd Asia-Pacific International Symposium. IEEE, Iran, 59–64. https://doi.org/10.1109/AIAR.2018.8769798
[4]
Christoph Bartneck, Marius Soucy, Kevin Fleuret, and Eduardo B. Sandoval. 2015. The robot engine — Making the unity 3D game engine work for HRI. In 2015 24th IEEE International Symposium on Robot and Human Interactive Communication (RO-MAN). IEEE, Kobe, Japan, 431–437. https://doi.org/10.1109/ROMAN.2015.7333561
[5]
Andrea Bisson. 2020. COMAU: Collaborative Robotics Market and Applications in Industrial Environments. In Inclusive Robotics for a Better Society, José L. Pons (Ed.). Springer International Publishing, Cham, 12–16. https://doi.org/10.1007/978-3-030-24074-5_3
[6]
Mario Bollini, Stefanie Tellex, Tyler Thompson, Nicholas Roy, and Daniela Rus. 2013. Interpreting and Executing Recipes with a Cooking Robot. Springer International Publishing, Heidelberg, 481–495. https://doi.org/10.1007/978-3-319-00065-7_33
[7]
Elodie Bouzbib, Gilles Bailly, Sinan Haliyo, and Pascal Frey. 2021. Can I Touch This?: Survey of Virtual Reality Interactions via Haptic Solutions, In 32e Conférence Francophone sur l’Interaction Homme-Machine (IHM ’20.21). ArXiv abs/2101.11278, 1–11. arxiv:2101.11278 [cs.HC]
[8]
Daniel Brice, Thomas McRoberts, and Karen Rafferty. 2019. A Proof of Concept Integrated Multi-systems Approach for Large Scale Tactile Feedback in VR. In Augmented Reality, Virtual Reality, and Computer Graphics, Lucio Tommaso De Paolis and Patrick Bourdot (Eds.). Springer International Publishing, Cham, 120–137. https://doi.org/10.1007/978-3-030-25965-5_10
[9]
Christopher Crick, Graylin Jay, Sarah Osentoski, Benjamin Pitzer, and Odest Chadwicke Jenkins. 2017. Rosbridge: ROS for Non-ROS Users. Springer International Publishing, Cham, 493–504. https://doi.org/10.1007/978-3-319-29363-9_28
[10]
Edson de Araújo Silva, Eduardo Valentin, Jose Reginaldo Hughes Carvalho, and Raimundo da Silva Barreto. 2021. A survey of Model Driven Engineering in robotics. Journal of Computer Languages 62 (2021), 101021. https://doi.org/10.1016/j.cola.2020.101021
[11]
Xavier de Tinguy, Claudio Pacchierotti, Mathieu Emily, Mathilde Chevalier, Aurélie Guignardat, Morgan Guillaudeux, Chloé Six, Anatole Lécuyer, and Maud Marchal. 2019. How Different Tangible and Virtual Objects Can Be While Still Feeling the Same?. In 2019 IEEE World Haptics Conference (WHC). IEEE, Tokyo, Japan, 580–585. https://doi.org/10.1109/WHC.2019.8816164
[12]
Xavier de Tinguy, Claudio Pacchierotti, Maud Marchal, and Anatole Lécuyer. 2018. Enhancing the Stiffness Perception of Tangible Objects in Mixed Reality Using Wearable Haptics. In 2018 IEEE Conference on Virtual Reality and 3D User Interfaces (VR). IEEE, Tuebingen, Germany, 81–90. https://doi.org/10.1109/VR.2018.8446280
[13]
Richard C Dorf and Robert H Bishop. 2011. Modern control systems. Pearson, Singapore.
[14]
Sebastian Günther, Florian Müller, Felix Hübner, Max Mühlhäuser, and Andrii Matviienko. 2021. ActuBoard: An Open Rapid Prototyping Platform to Integrate Hardware Actuators in Remote Applications. In Companion of the 2021 ACM SIGCHI Symposium on Engineering Interactive Computing Systems (Virtual Event, Netherlands) (EICS ’21). Association for Computing Machinery, New York, NY, USA, 70–76. https://doi.org/10.1145/3459926.3464757
[15]
Matthias Hoppe, Beat Rossmy, Daniel Peter Neumann, Stephan Streuber, Albrecht Schmidt, and Tonja-Katrin Machulla. 2020. A Human Touch: Social Touch Increases the Perceived Human-Likeness of Agents in Virtual Reality. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems (Honolulu, HI, USA) (CHI ’20). Association for Computing Machinery, New York, NY, USA, 1–11. https://doi.org/10.1145/3313831.3376719
[16]
Leire Amezua Hormaza, Wael M Mohammed, Borja Ramis Ferrer, Ronal Bejarano, and Jose L Martinez Lastra. 2019. On-line training and monitoring of robot tasks through virtual reality. In 2019 IEEE 17th International Conference on Industrial Informatics (INDIN), Vol. 1. IEEE, Finland, 841–846. https://doi.org/10.1109/INDIN41052.2019.8971967
[17]
Thomas Howard, Maud Marchal, Anatole Lécuyer, and Claudio Pacchierotti. 2020. PUMAH: Pan-Tilt Ultrasound Mid-Air Haptics for Larger Interaction Workspace in Virtual Reality. IEEE Transactions on Haptics 13, 1 (2020), 38–44. https://doi.org/10.1109/TOH.2019.2963028
[18]
Lentin Joseph. 2018. Robot Operating System (ROS) for Absolute Beginners. Springer, Singapore. https://doi.org/10.1007/978-1-4842-3405-1
[19]
Dahyun Kang, Sonya S. Kwak, Hanbyeol Lee, and JongSuk Choi. 2020. First Things First: A Survey Exploring Key Services and Functions of a Robot. In Companion of the 2020 ACM/IEEE International Conference on Human-Robot Interaction (Cambridge, United Kingdom) (HRI ’20). Association for Computing Machinery, New York, NY, USA, 278–280. https://doi.org/10.1145/3371382.3378317
[20]
Ben Kehoe, Sachin Patil, Pieter Abbeel, and Ken Goldberg. 2015. A Survey of Research on Cloud Robotics and Automation. IEEE Transactions on Automation Science and Engineering 12, 2(2015), 398–409. https://doi.org/10.1109/TASE.2014.2376492
[21]
Yaesol Kim, Hyun Jung Kim, and Young J. Kim. 2018. Encountered-type haptic display for large VR environment using per-plane reachability maps. Computer Animation and Virtual Worlds 29, 3-4 (2018), e1814. https://doi.org/10.1002/cav.1814 e1814 cav.1814.
[22]
Xin Li, José-Fernán Martínez, Jesús Rodríguez-Molina, and Néstor Lucas Martínez. 2016. A Survey on Intermediation Architectures for Underwater Robotics. Sensors 16, 2 (2016), 0–0. https://doi.org/10.3390/s16020190
[23]
Ali Ahmad Malik, Tariq Masood, and Arne Bilberg. 2020. Virtual reality in manufacturing: immersive and collaborative artificial-reality in design of human-robot workspace. International Journal of Computer Integrated Manufacturing 33, 1(2020), 22–37. https://doi.org/10.1080/0951192X.2019.1690685
[24]
Victor Rodrigo Mercado, Thomas Howard, Hakim Si-Mohammed, Ferran Argelaguet, and Anatole Lécuyer. 2021. Alfred: the Haptic Butler On-Demand Tangibles for Object Manipulation in Virtual Reality using an ETHD. In 2021 IEEE World Haptics Conference (WHC). IEEE, Tokyo, Japan, 373–378. https://doi.org/10.1109/WHC49131.2021.9517250
[25]
Victor Rodrigo Mercado, Maud Marchal, and Anatole Lécuyer. 2021. ENTROPiA: Towards Infinite Surface Haptic Displays in Virtual Reality Using Encountered-Type Rotating Props. IEEE Transactions on Visualization and Computer Graphics 27, 3(2021), 2237–2243. https://doi.org/10.1109/TVCG.2019.2963190
[26]
Víctor Rodrigo Mercado, Maud Marchal, and Anatole Lécuyer. 2021. “Haptics On-Demand”: A Survey on Encountered-Type Haptic Displays. IEEE Transactions on Haptics 14, 3 (2021), 449–464. https://doi.org/10.1109/TOH.2021.3061150
[27]
Joseph E. Michaelis, Amanda Siebert-Evenstone, David Williamson Shaffer, and Bilge Mutlu. 2020. Collaborative or Simply Uncaged? Understanding Human-Cobot Interactions in Automation. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems (Honolulu, HI, USA) (CHI ’20). Association for Computing Machinery, New York, NY, USA, 1–12. https://doi.org/10.1145/3313831.3376547
[28]
Carl A Moore, Michael A Peshkin, and J Edward Colgate. 2003. Cobot implementation of virtual paths and 3D virtual surfaces. IEEE Transactions on Robotics and Automation 19, 2(2003), 347–351. https://doi.org/10.1109/TRA.2003.808866
[29]
Thomas Muender, Anke V. Reinschluessel, Sean Drewes, Dirk Wenig, Tanja Döring, and Rainer Malaka. 2019. Does It Feel Real? Using Tangibles with Different Fidelities to Build and Explore Scenes in Virtual Reality. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems. Association for Computing Machinery, New York, NY, USA, 1–12. https://doi.org/10.1145/3290605.3300903
[30]
Tomi Nukarinen, Jari Kangas, Jussi Rantala, Toni Pakkanen, and Roope Raisamo. 2018. Hands-Free Vibrotactile Feedback for Object Selection Tasks in Virtual Reality. In Proceedings of the 24th ACM Symposium on Virtual Reality Software and Technology(Tokyo, Japan) (VRST ’18). Association for Computing Machinery, New York, NY, USA, Article 94, 2 pages. https://doi.org/10.1145/3281505.3283375
[31]
Carl Oechsner, Sven Mayer, and Andreas Butz. 2022. Challenges and Opportunities of Cooperative Robots as Cooking Appliances. In Proceedings of the 2022 Workshop on Engaging with Automation (2022-04-30) (AutomationXP22). CEUR Workshops, Germany, 7 pages. http://ceur-ws.org/Vol-3154/paper14.pdf
[32]
Puren R Ouyang, J Tang, WH Yue, and Shanuka Jayasinghe. 2016. Adaptive PD plus sliding mode control for robotic manipulator. In 2016 IEEE International Conference on Advanced Intelligent Mechatronics (AIM). IEEE, Alberta, Canada, 930–934. https://doi.org/10.1109/AIM.2016.7576888
[33]
Le Minh Phuoc, Philippe Martinet, Sukhan Lee, and Hunmo Kim. 2008. Damped least square based genetic algorithm with Ggaussian distribution of damping factor for singularity-robust inverse kinematics. Journal of mechanical science and technology 22, 7(2008), 1330–1338. https://doi.org/10.1007/s12206-008-0427-4
[34]
Morgan Quigley, Ken Conley, Brian P. Gerkey, Josh Faust, Tully Foote, Jeremy Leibs, Rob Wheeler, and Andrew Y. Ng. 2009. ROS: an open-source Robot Operating System. In ICRA Workshop on Open Source Software. IEEE, Stanford, 0–0.
[35]
Marina Ramon Borràs. 2020. iMOVE: Development of a hybrid control interface based on sEMG and movement signals for an assistive robotic manipulator. Master’s thesis. Universitat Politècnica de Catalunya.
[36]
Jose Luis Rubio-Tamayo, Manuel Gertrudix Barrio, and Francisco García García. 2017. Immersive Environments and Virtual Reality: Systematic Review and Advances in Communication, Interaction and Simulation. Multimodal Technologies and Interaction 1, 4 (2017), 0–0. https://doi.org/10.3390/mti1040021
[37]
Saeed Saeedvand, Masoumeh Jafari, Hadi S. Aghdasi, and Jacky Baltes. 2019. A comprehensive survey on humanoid robot development. The Knowledge Engineering Review 34 (2019), e20. https://doi.org/10.1017/S0269888919000158
[38]
Steeven Villa Salazar, Claudio Pacchierotti, Xavier de Tinguy, Anderson Maciel, and Maud Marchal. 2020. Altering the Stiffness, Friction, and Shape Perception of Tangible Objects in Virtual Reality Using Wearable Haptics. IEEE Transactions on Haptics 13, 1 (2020), 167–174. https://doi.org/10.1109/TOH.2020.2967389
[39]
Shun Suzuki, Masahiro Fujiwara, Yasutoshi Makino, and Hiroyuki Shinoda. 2019. Midair Ultrasound Haptic Display with Large Workspace. In Haptic Interaction, Hiroyuki Kajimoto, Dongjun Lee, Sang-Youn Kim, Masashi Konyo, and Ki-Uk Kyung (Eds.). Springer Singapore, Singapore, 3–5.
[40]
Unity-Technologies. 2022. Unity-Technologies/Unity-robotics-hub: Central Repository for tools, tutorials, resources, and documentation for robotics simulation in Unity.https://github.com/Unity-Technologies/Unity-Robotics-Hub
[41]
Manuela M. Veloso. 2018. The Increasingly Fascinating Opportunity for Human-Robot-AI Interaction: The CoBot Mobile Service Robots. J. Hum.-Robot Interact. 7, 1, Article 5 (May 2018), 2 pages. https://doi.org/10.1145/3209541
[42]
Dangxiao Wang, Yuan Guo, Shiyi Liu, Yuru Zhang, Weiliang Xu, and Jing Xiao. 2019. Haptic display for virtual reality: progress and challenges. Virtual Reality and Intelligent Hardware 1, 2 (2019), 136–162. https://doi.org/10.3724/SP.J.2096-5796.2019.0008
[43]
Chifu Yang, Qitao Huang, Hongzhou Jiang, O Ogbobe Peter, and Junwei Han. 2010. PD control with gravity compensation for hydraulic 6-DOF parallel manipulator. Mechanism and Machine theory 45, 4 (2010), 666–677. https://doi.org/10.1016/j.mechmachtheory.2009.12.001
[44]
Dongya Zhao, Yongrui Zhao, Shaoyuan Li, and Quanmin Zhu. 2011. A new terminal converging PD control for parallel robotic manipulators with bounded torque. In Proceedings of 2011 International Conference on Modelling, Identification and Control. IEEE, Innsbruck, Austria, 115–121. https://doi.org/10.1109/ICMIC.2011.5973686

Cited By

View all
  • (2025)Enhancing Human–Agent Interaction via Artificial Agents That Speculate About the FutureFuture Internet10.3390/fi1702005217:2(52)Online publication date: 21-Jan-2025
  • (2022)Extended Mid-air Ultrasound Haptics for Virtual RealityProceedings of the ACM on Human-Computer Interaction10.1145/35677316:ISS(500-524)Online publication date: 14-Nov-2022

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Other conferences
MuC '22: Proceedings of Mensch und Computer 2022
September 2022
624 pages
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 15 September 2022

Permissions

Request permissions for this article.

Check for updates

Author Tags

  1. cobots
  2. haptics
  3. robots
  4. toolbox
  5. unity
  6. virtual reality

Qualifiers

  • Research-article
  • Research
  • Refereed limited

Conference

MuC '22
MuC '22: Mensch und Computer 2022
September 4 - 7, 2022
Darmstadt, Germany

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)40
  • Downloads (Last 6 weeks)1
Reflects downloads up to 17 Feb 2025

Other Metrics

Citations

Cited By

View all
  • (2025)Enhancing Human–Agent Interaction via Artificial Agents That Speculate About the FutureFuture Internet10.3390/fi1702005217:2(52)Online publication date: 21-Jan-2025
  • (2022)Extended Mid-air Ultrasound Haptics for Virtual RealityProceedings of the ACM on Human-Computer Interaction10.1145/35677316:ISS(500-524)Online publication date: 14-Nov-2022

View Options

Login options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

HTML Format

View this article in HTML Format.

HTML Format

Figures

Tables

Media

Share

Share

Share this Publication link

Share on social media