research-article

Improving Collocated Robot Teleoperation with Augmented Reality

Authors:

Hooman Hedayati,

Michael Walker,

Daniel SzafirAuthors Info & Claims

HRI '18: Proceedings of the 2018 ACM/IEEE International Conference on Human-Robot Interaction

Pages 78 - 86

https://doi.org/10.1145/3171221.3171251

Published: 26 February 2018 Publication History

Abstract

Robot teleoperation can be a challenging task, often requiring a great deal of user training and expertise, especially for platforms with high degrees-of-freedom (e.g., industrial manipulators and aerial robots). Users often struggle to synthesize information robots collect (e.g., a camera stream) with contextual knowledge of how the robot is moving in the environment. We explore how advances in augmented reality (AR) technologies are creating a new design space for mediating robot teleoperation by enabling novel forms of intuitive, visual feedback. We prototype several aerial robot teleoperation interfaces using AR, which we evaluate in a 48-participant user study where participants completed an environmental inspection task. Our new interface designs provided several objective and subjective performance benefits over existing systems, which often force users into an undesirable paradigm that divides user attention between monitoring the robot and monitoring the robot»s camera feed(s).

References

[1]

Ronald Azuma, Yohan Baillot, Reinhold Behringer, Steven Feiner, Simon Julier, and Blair MacIntyre. 2001. Recent Advances in Augmented Reality. IEEE Computer Graphics and Applications 21, 6 (2001), 34--47.

Digital Library

[2]

Maria Bualat, Jonathan Barlow, Terrence Fong, Christopher Provencher, Trey Smith, and Allison Zuniga. 2015. Astrobee: Developing a Free-flying Robot forthe International Space Station. In AIAA SPACE 2015 Conference and Exposition. 4643.

[3]

Jessie YC Chen, Ellen C Haas, and Michael J Barnes. 2007. Human Performance Issues and User Interface Design for Teleoperated Robots. IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews) 37, 6 (2007), 1231--1245.

Digital Library

[4]

Mark Corcoran. 2014. Drone Journalism Takes Off. http://www.abc.net.au/news/2012-02--21/drone-journalism-takes-off/3840616. (February 2014). Accessed: 2018-01-05.

[5]

Enrico Costanza, Samuel A Inverso, Elan Pavlov, Rebecca Allen, and Pattie Maes. 2006. Eye-q: Eyeglass Peripheral Display for Subtle Intimate Notifications. In Proceedings of the ACM Conference on Human-Computer Interaction with Mobile Devices and Services. 211--218.

Digital Library

[6]

Mike Daily, Youngkwan Cho, Kevin Martin, and Dave Payton. 2003. World embedded interfaces for human-robot interaction. In System Sciences (HICSS'03), 2003. Proceedings of the 36th Annual Hawaii International Conference on. IEEE, 6--pp.

Digital Library

[7]

Anca D Dragan and Siddhartha S Srinivasa. 2013. A Policy-blending Formalism for Shared Control. The International Journal of Robotics Research 32, 7 (2013), 790--805.

Digital Library

[8]

Jill L Drury, Justin Richer, Nathan Rackliffe, and Michael A Goodrich. 2006. Comparing Situation Awareness for Two Unmanned Aerial Vehicle Human Interface Approaches. Technical Report. DTIC Document.

[9]

Steven Feiner, Blair MacIntyre, Marcus Haupt, and Eliot Solomon. 1993. Windows on the World: 2D Windows for 3D Augmented Reality. In Proceedings of the 6th annual ACM Symposium on User Interface Software and Technology (UIST'93). 145--155.

Digital Library

[10]

T. Fong, M. Micire, T. Morse, E. Park, C. Provencher, V. To, DW. Wheeler, D. Mittman, R. J. Torres, and E. Smith. 2013. Smart SPHERES: A Telerobotic Free-Flyer for Intravehicular Activities in Space. In Proceedings of AIAA Space'13.

[11]

Foyle, David C and Andre, Anthony D and Hooey, Becky L. 2005. Situation Awareness in an Augmented Reality Cockpit: Design, Viewpoints and Cognitive Glue. In Proceedings of the 11th International Conference on Human Computer Interaction. 3--9.

[12]

Michael A Goodrich, Bryan S Morse, Damon Gerhardt, Joseph L Cooper, Morgan Quigley, Julie A Adams, and Curtis Humphrey. 2008. Supporting Wilderness Search and Rescue using a Camera-equipped Mini UAV. Journal of Field Robotics 25, 1--2 (2008), 89--110.

Digital Library

[13]

Scott A Green, Mark Billinghurst, XiaoQi Chen, and J Geoffrey Chase. 2008. Human-Robot Collaboration: A Literature Review and Augmented Reality Approach in Design. International Journal of Advanced Robotic Systems 5, 1 (2008), 1.

[14]

Scott A Green, J Geoffrey Chase, XiaoQi Chen, and Mark Billinghurst. 2009. Evaluating the Augmented Reality Human-Robot Collaboration System. International Journal of Intelligent Systems Technologies and Applications 8, 1--4 (2009), 130--143.

Digital Library

[15]

Drew Harwell. 2014. This government rule could cripple commercial drone flight. https://www.washingtonpost.com/blogs/the-switch/wp/2014/11/25/this-government-rule-could-cripple-commercial-drone-flight/. (2014).

[16]

Keita Higuchi, Katsuya Fujii, and Jun Rekimoto. 2013. Flying Head: A Headsynchronization Mechanism for Flying Telepresence. In International Conference on Artificial Reality and Telexistence (ICAT'13). 28--34.

Digital Library

[17]

Yoshio Ishiguro and Jun Rekimoto. 2011. Peripheral Vision Annotation: Noninterference Information Presentation Method for Mobile Augmented Reality. In Proceedings of the ACM Augmented Human International Conference. 8.

Digital Library

[18]

J Richard Landis and Gary G Koch. 1977. The Measurement of Observer Agreement for Categorical Data. Biometrics 33 (1977), 159--174. Issue 1.

[19]

Paula Lavigne. 2014. Eyes in the Sports Sky. ESPN website. (2014). Accessed: 2018-01-05.

[20]

Li, Hua and Zhang, Xin and Shi, Guangwei and Qu, Hemeng and Wu, Yanxiong and Zhang, Jianping. 2013. Review and Analysis of Avionic Helmet-Mounted Displays. Optical Engineering 52, 11 (2013), 110901--110901.

[21]

Jeffrey I Lipton, Aidan J Fay, and Daniela Rus. 2018. Baxter's Homunculus: Virtual Reality Spaces for Teleoperation in Manufacturing. IEEE Robotics and Automation Letters 3, 1 (2018), 179--186.

[22]

M Alejandra Menchaca-Brandan, Andrew M Liu, Charles M Oman, and Alan Natapoff. 2007. Influence of Perspective-taking and Mental Rotation Abilities in Space Teleoperation. In ACM/IEEE International Conference on Human-Robot Interaction (HRI'07). 271--278.

Digital Library

[23]

Paul Milgram and Fumio Kishino. 1994. A Taxonomy of Mixed Reality Visual Displays. IEICE Transactions on Information and Systems 77, 12 (1994), 1321-- 1329.

[24]

Paul Milgram, Shumin Zhai, David Drascic, and Julius Grodski. 1993. Applications of Augmented Reality for Human-Robot Communication. In Proceedings of the IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS'93), Vol. 3. 1467--1472.

[25]

Masmoudi Mostefa, L Kaddour El Boudadi, A Loukil, Khelf Mohamed, and Dahane Amine. 2015. Design of Mobile Robot Teleoperation System based on Virtual Reality. In IEEE International Conference on Control, Engineering&Information Technology (CEIT'15). 1--6.

[26]

Robin R Murphy. 2014. Disaster Robotics. MIT Press.

Digital Library

[27]

Nicholl, Ryan. 2014. Airline Head-up Display Systems: Human Factors Considerations. Available at SSRN:https:// ssrn.com/ abstract=2384101 (2014).

[28]

Curtis W Nielsen, Michael A Goodrich, and Robert W Ricks. 2007. Ecological Interfaces for Improving Mobile Robot Teleoperation. IEEE Transactions on Robotics 23, 5 (2007), 927--941.

Digital Library

[29]

Shahin S Nudehi, Rohan Mukherjee, and Moji Ghodoussi. 2003. A Haptic Interface Design for Minimally Invasive Telesurgical Training and Collaboration in the Presence of Time Delay. In Proceedings. 42nd IEEE Conference on Decision and Control, Vol. 5. 4563--4568.

[30]

Allison M Okamura. 2004. Methods for Haptic Feedback in Teleoperated Robotassisted Surgery. Industrial Robot: An International Journal 31, 6 (2004), 499--508.

[31]

Christos Papachristos and Kostas Alexis. 2016. Augmented Reality-Enhanced Structural Inspection Using Aerial Robots. In IEEE International Symposium on Intelligent Control (ISIC'16). IEEE, 1--6.

[32]

FJ Perez-Grau, R Ragel, F Caballero, A Viguria, and A Ollero. 2017. Semiautonomous Teleoperation of UAVs in Search and Rescue Scenarios. In International Conference on Unmanned Aircraft Systems (ICUAS'17). 1066--1074.

[33]

David Pitman and Mary L Cummings. 2012. Collaborative exploration with a Micro Aerial Vehicle: A Novel Interaction Method for Controlling a MAV with a Hand-Held Device. Advances in Human-Computer Interaction 2012 (2012), 18.

Digital Library

[34]

Benjamin Poppinga, Niels Henze, Jutta Fortmann, Wilko Heuten, and Susanne Boll. 2012. AmbiGlasses-Information in the Periphery of the Visual Field. In Mensch&Computer. 153--162.

[35]

Morgan Quigley, Michael A Goodrich, and Randal W Beard. 2004. Semiautonomous human-UAV interfaces for fixed-wing mini-UAVs. In IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS'04), Vol. 3. IEEE, 2457--2462.

[36]

Emanuele Ruffaldi, Filippo Brizzi, Franco Tecchia, and Sandro Bacinelli. 2016. Third Point of View Augmented Reality for Robot Intentions Visualization. In International Conference on Augmented Reality, Virtual Reality and Computer Graphics. 471--478.

[37]

Daniel Szafir. 2015. Human Interaction with Assistive Free-Flying Robots. Ph.D. Dissertation. The University of Wisconsin-Madison.

[38]

Daniel Szafir, Bilge Mutlu, and Terrence Fong. 2017. Designing Planning and Control Interfaces to Support User Collaboration with Flying Robots. The International Journal of Robotics Research 36, 5--7, 514--542.

Digital Library

[39]

Russell H Taylor, Arianna Menciassi, Gabor Fichtinger, Paolo Fiorini, and Paolo Dario. 2016. Medical Robotics and Computer-Integrated Surgery. In Springer Handbook of Robotics. 1657--1684.

[40]

John Thomason, Photchara Ratsamee, Kiyoshi Kiyokawa, Pakpoom Kriangkomol, Jason Orlosky, Tomohiro Mashita, Yuki Uranishi, and Haruo Takemura. 2017. Adaptive View Management for Drone Teleoperation in Complex 3D Structures. In Proceedings of the ACM International Conference on Intelligent User Interfaces (IUI'17). ACM, 419--426.

Digital Library

Cited By

Ikeda BGramopadhye MNekervis LSzafir DJohal WLemaignan SBrščić DVázquez MCharisi V(2025)MARCER: Multimodal Augmented Reality for Composing and Executing Robot TasksProceedings of the 2025 ACM/IEEE International Conference on Human-Robot Interaction10.5555/3721488.3721555(529-539)Online publication date: 4-Mar-2025
https://dl.acm.org/doi/10.5555/3721488.3721555
Nguyen DBhattacharyya RColombatto CFleming SPosner IHawes NJohal WLemaignan SBrščić DVázquez MCharisi V(2025)Group Decision-Making in Robot Teleoperation: Two Heads are Better Than OneProceedings of the 2025 ACM/IEEE International Conference on Human-Robot Interaction10.5555/3721488.3721550(489-500)Online publication date: 4-Mar-2025
https://dl.acm.org/doi/10.5555/3721488.3721550
Wang XNitsche MResch GMazalek AWelsh T(2025)Mixed reality alters motor planning and controlBehavioural Brain Research10.1016/j.bbr.2024.115373480(115373)Online publication date: Mar-2025
https://doi.org/10.1016/j.bbr.2024.115373
Show More Cited By

Index Terms

Improving Collocated Robot Teleoperation with Augmented Reality

Recommendations

Assisting Manipulation and Grasping in Robot Teleoperation with Augmented Reality Visual Cues
CHI '21: Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems

Teleoperating industrial manipulators in co-located spaces can be challenging. Facilitating robot teleoperation by providing additional visual information about the environment and the robot affordances using augmented reality (AR), can improve task ...
Virtual, Augmented, and Mixed Reality for Human-Robot Interaction
HRI '18: Companion of the 2018 ACM/IEEE International Conference on Human-Robot Interaction

The 1st International Workshop on Virtual, Augmented, and Mixed Reality for Human-Robot Interactions (VAM-HRI) will bring together HRI, Robotics, Artificial Intelligence, and Mixed Reality researchers to identify challenges in mixed reality interactions ...
Communicating Robot Motion Intent with Augmented Reality
HRI '18: Proceedings of the 2018 ACM/IEEE International Conference on Human-Robot Interaction

Humans coordinate teamwork by conveying intent through social cues, such as gestures and gaze behaviors. However, these methods may not be possible for appearance-constrained robots that lack anthropomorphic or zoomorphic features, such as aerial ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

HRI '18: Proceedings of the 2018 ACM/IEEE International Conference on Human-Robot Interaction

February 2018

468 pages

ISBN:9781450349536

DOI:10.1145/3171221

General Chairs:
Takayuki Kanda
ATR, Japan
,
Selma Ŝabanović
Indiana University Bloomington, USA
,
Program Chairs:
Guy Hoffman
Cornell University, USA
,
Adriana Tapus
ENSTA-ParisTech, France

Copyright © 2018 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

SIGAI: ACM Special Interest Group on Artificial Intelligence
SIGCHI: ACM Special Interest Group on Computer-Human Interaction
IEEE-RAS: Robotics and Automation

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 26 February 2018

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

The National Aeronautics and Space Administration (NASA)

Conference

HRI '18

Sponsor:

SIGAI
SIGCHI
IEEE-RAS

HRI '18: ACM/IEEE International Conference on Human-Robot Interaction

March 5 - 8, 2018

IL, Chicago, USA

Acceptance Rates

HRI '18 Paper Acceptance Rate 49 of 206 submissions, 24%;

Overall Acceptance Rate 268 of 1,124 submissions, 24%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

121
Total Citations
View Citations
3,345
Total Downloads

Downloads (Last 12 months)289
Downloads (Last 6 weeks)15

Reflects downloads up to 03 Mar 2025

Other Metrics

View Author Metrics

Citations

Cited By

Ikeda BGramopadhye MNekervis LSzafir DJohal WLemaignan SBrščić DVázquez MCharisi V(2025)MARCER: Multimodal Augmented Reality for Composing and Executing Robot TasksProceedings of the 2025 ACM/IEEE International Conference on Human-Robot Interaction10.5555/3721488.3721555(529-539)Online publication date: 4-Mar-2025
https://dl.acm.org/doi/10.5555/3721488.3721555
Nguyen DBhattacharyya RColombatto CFleming SPosner IHawes NJohal WLemaignan SBrščić DVázquez MCharisi V(2025)Group Decision-Making in Robot Teleoperation: Two Heads are Better Than OneProceedings of the 2025 ACM/IEEE International Conference on Human-Robot Interaction10.5555/3721488.3721550(489-500)Online publication date: 4-Mar-2025
https://dl.acm.org/doi/10.5555/3721488.3721550
Wang XNitsche MResch GMazalek AWelsh T(2025)Mixed reality alters motor planning and controlBehavioural Brain Research10.1016/j.bbr.2024.115373480(115373)Online publication date: Mar-2025
https://doi.org/10.1016/j.bbr.2024.115373
Krzysiak RButail S(2024)Human Prior Knowledge Estimation from Movement Cues for Information-Based Control of Mobile Robots during SearchACM Transactions on Human-Robot Interaction10.1145/370606514:2(1-29)Online publication date: 27-Nov-2024
https://dl.acm.org/doi/10.1145/3706065
Jia RYang LCao YKalun Or CWang WPan J(2024)Learning Autonomous Viewpoint Adjustment from Human Demonstrations for TelemanipulationACM Transactions on Human-Robot Interaction10.1145/366034813:3(1-23)Online publication date: 26-Sep-2024
https://dl.acm.org/doi/10.1145/3660348
Ikeda BSzafir D(2024)PRogramAR: Augmented Reality End-User Robot ProgrammingACM Transactions on Human-Robot Interaction10.1145/364000813:1(1-20)Online publication date: 12-Jan-2024
https://dl.acm.org/doi/10.1145/3640008
Reitmann SJung BGrollman DBroadbent EJu WSoh HWilliams T(2024)VR-based Assistance System for Semi-Autonomous Robotic BoatsCompanion of the 2024 ACM/IEEE International Conference on Human-Robot Interaction10.1145/3610978.3640750(877-881)Online publication date: 11-Mar-2024
https://dl.acm.org/doi/10.1145/3610978.3640750
Luebbers MHayes BGrollman DBroadbent EJu WSoh HWilliams T(2024)Explainable Guidance and Justification for Mental Model Alignment in Human-Robot TeamsCompanion of the 2024 ACM/IEEE International Conference on Human-Robot Interaction10.1145/3610978.3638364(118-120)Online publication date: 11-Mar-2024
https://dl.acm.org/doi/10.1145/3610978.3638364
Yoo GJi Y(2024)A Study on View Sharing AR Interface for Improving Situation Awareness during Military OperationsInternational Journal of Human–Computer Interaction10.1080/10447318.2024.231694941:4(2211-2226)Online publication date: 22-Feb-2024
https://doi.org/10.1080/10447318.2024.2316949
Liao ZCai Y(2024)AR-enhanced digital twin for human–robot interaction in manufacturing systemsEnergy, Ecology and Environment10.1007/s40974-024-00327-79:5(530-548)Online publication date: 25-May-2024
https://doi.org/10.1007/s40974-024-00327-7
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten