skip to main content
10.1145/3654777.3676355acmotherconferencesArticle/Chapter ViewAbstractPublication PagesuistConference Proceedingsconference-collections
research-article

AccessTeleopKit: A Toolkit for Creating Accessible Web-Based Interfaces for Tele-Operating an Assistive Robot

Published: 11 October 2024 Publication History

Abstract

Mobile manipulator robots, which can move around and physically interact with their environments, can empower people with motor limitations to independently carry out many activities of daily living. While many interfaces have been developed for tele-operating complex robots, most of them are not accessible to people with severe motor limitations. Further, most interfaces are rigid with limited configurations and are not readily available to download and use. To address these barriers, we developed AccessTeleopKit: an open-source toolkit for creating custom and accessible robot tele-operation interfaces based on cursor-and-click input for the Stretch 3 mobile-manipulator. With AccessTeleopKit users can add, remove, and rearrange components such as buttons and camera views, and select between a variety of control modes. We describe the participatory and iterative design process that led to the current implementation of AccessTeleopKit, involving three long-term deployments of the robot in the home of a quadriplegic user. We demonstrate how AccessTeleopKit allowed the user to create different interfaces for different tasks and the diversity of tasks it allowed the user to carry out. We also present two studies involving six additional users with severe motor limitations, demonstrating the power of AccessTeleopKit in creating custom interfaces for different user needs and preferences.

Supplemental Material

MP4 File
Video Figure

References

[1]
Jenay M Beer, Cory-Ann Smarr, Tiffany L Chen, Akanksha Prakash, Tracy L Mitzner, Charles C Kemp, and Wendy A Rogers. 2012. The domesticated robot: design guidelines for assisting older adults to age in place. In Proceedings of the seventh annual ACM/IEEE international conference on Human-Robot Interaction. 335–342.
[2]
Tapomayukh Bhattacharjee, Maria E Cabrera, Anat Caspi, Maya Cakmak, and Siddhartha S Srinivasa. 2019. A Community-Centered Design Framework for Robot-Assisted Feeding Systems. In The 21st International ACM SIGACCESS Conference on Computers and Accessibility. 482–494.
[3]
Tapomayukh Bhattacharjee, Ethan K Gordon, Rosario Scalise, Maria E Cabrera, Anat Caspi, Maya Cakmak, and Siddhartha S Srinivasa. 2020. Is More Autonomy Always Better? Exploring Preferences of Users with Mobility Impairments in Robot-assisted Feeding. In Proceedings of the 2020 ACM/IEEE International Conference on Human-Robot Interaction. 181–190.
[4]
Elizabeth Broadbent, Rebecca Stafford, and Bruce MacDonald. 2009. Acceptance of healthcare robots for the older population: Review and future directions. International journal of social robotics 1, 4 (2009), 319.
[5]
Steven W Brose, Douglas J Weber, Ben A Salatin, Garret G Grindle, Hongwu Wang, Juan J Vazquez, and Rory A Cooper. 2010. The role of assistive robotics in the lives of persons with disability. American Journal of Physical Medicine & Rehabilitation (2010).
[6]
Guido Bugmann and Simon N Copleston. 2011. What can a personal robot do for you?. In Conference Towards Autonomous Robotic Systems. Springer, 360–371.
[7]
Maria E Cabrera, Tapomayukh Bhattacharjee, Kavi Dey, and Maya Cakmak. 2021. An exploration of accessible remote tele-operation for assistive mobile manipulators in the home. In International Conference on Robot & Human Interactive Communication (RO-MAN). IEEE.
[8]
Scott Carter, Amy Hurst, Jennifer Mankoff, and Jack Li. 2006. Dynamically adapting GUIs to diverse input devices. In Proceedings of the 8th international ACM SIGACCESS conference on Computers and accessibility. 63–70.
[9]
Greg Chance, Praminda Caleb-Solly, Aleksandar Jevtić, and Sanja Dogramadzi. 2017. What’s “up”?—Resolving interaction ambiguity through non-visual cues for a robotic dressing assistant. In 2017 26th IEEE international symposium on robot and human interactive communication (RO-MAN). IEEE, 284–291.
[10]
Jessie YC Chen, Ellen C Haas, and Michael J Barnes. 2007. Human performance issues and user interface design for teleoperated robots. IEEE Transactions on Systems, Man, and Cybernetics 37, 6 (2007), 1231–1245.
[11]
Matei Ciocarlie, Kaijen Hsiao, Adam Leeper, and David Gossow. 2012. Mobile manipulation through an assistive home robot. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems. IEEE, 5313–5320.
[12]
Varad Dhat, Nick Walker, and Maya Cakmak. 2024. Using 3D Mice to Control Robot Manipulators. In Proceedings of the 2024 ACM/IEEE International Conference on Human-Robot Interaction. 896–900.
[13]
Anca D Dragan and Siddhartha S Srinivasa. 2013. Formalizing assistive teleoperation. (2013).
[14]
Paul M Fitts. 1954. The information capacity of the human motor system in controlling the amplitude of movement.Journal of experimental psychology 47, 6 (1954), 381.
[15]
T. Fong, R. Berka, M. Bualat, M. Diftler, M. Micire, D. Mittman, V. SunSpiral, and C. Provencher. 2012. The human exploration telerobotics project. In Global Space Exploration Conference.
[16]
Jodi Forlizzi, Carl DiSalvo, and Francine Gemperle. 2004. Assistive robotics and an ecology of elders living independently in their homes. Human–Computer Interaction 19, 1-2 (2004), 25–59.
[17]
Krzysztof Z Gajos, Jacob O Wobbrock, and Daniel S Weld. 2007. Automatically generating user interfaces adapted to users’ motor and vision capabilities. In Proceedings of the 20th annual ACM symposium on User interface software and technology. 231–240.
[18]
Qing Gao, Jinyang Li, Yimin Zhu, Siyue Wang, Jingshu Liufu, and Jinguo Liu. 2023. Hand gesture teleoperation for dexterous manipulators in space station by using monocular hand motion capture. Acta Astronautica 204 (2023), 630–639.
[19]
Phillip M Grice and Charles C Kemp. 2019. In-home and remote use of robotic body surrogates by people with profound motor deficits. PloS one (2019).
[20]
Ioannis Havoutis and Sylvain Calinon. 2016. Learning assistive teleoperation behaviors from demonstration. In 2016 IEEE International Symposium on Safety, Security, and Rescue Robotics (SSRR). 258–263. https://doi.org/10.1109/SSRR.2016.7784308
[21]
Amy Hurst, Scott E Hudson, Jennifer Mankoff, and Shari Trewin. 2008. Automatically detecting pointing performance. In Proceedings of the 13th international conference on Intelligent user interfaces. 11–19.
[22]
Siddarth Jain and Brenna Argall. 2016. An approach for online user customization of shared autonomy for intelligent assistive devices. In Proc. of the IEEE Int. Conf. on Robot. and Autom., Stockholm, Sweden.
[23]
Shervin Javdani, Siddhartha S Srinivasa, and J Andrew Bagnell. 2015. Shared autonomy via hindsight optimization. Robotics science and systems: online proceedings 2015 (2015).
[24]
Charles C Kemp, Aaron Edsinger, Henry M Clever, and Blaine Matulevich. 2022. The design of stretch: A compact, lightweight mobile manipulator for indoor human environments. In International Conference on Robotics and Automation.
[25]
Kavita Krishnaswamy, Tim Adamson, Maya Cakmak, and Tim Oates. 2018. Multi-Perspective, Multimodal, and Machine Learning for Accessible Robotic Web Interface. In Proceedings of the 2018 Rehabilitation Engineering and Assistive Technology Society of North America Annual Conference. RESNA, 1–4.
[26]
I-Fen Lin and Hsueh-Sheng Wu. 2011. Does informal care attenuate the cycle of ADL/IADL disability and depressive symptoms in late life?Journals of Gerontology Series B: Psychological Sciences and Social Sciences 66, 5 (2011), 585–594.
[27]
Mitchell JH Lum, Diana CW Friedman, Ganesh Sankaranarayanan, Hawkeye King, Kenneth Fodero, Rainer Leuschke, Blake Hannaford, Jacob Rosen, and Mika N Sinanan. 2009. The RAVEN: Design and validation of a telesurgery system. The International Journal of Robotics Research 28, 9 (2009), 1183–1197.
[28]
R.R. Murphy. 2004. Human-robot interaction in rescue robotics. IEEE Trans. on Systems, Man, and Cybernetics (SMC), Part C: Applications and Reviews 34, 2 (2004), 138–153.
[29]
Amal Nanavati, Max Pascher, Vinitha Ranganeni, Ethan K. Gordon, Taylor Kessler Faulkner, Siddhartha S. Srinivasa, Maya Cakmak, Patrícia Alves-Oliveira, and Jens Gerken. 2024. Multiple Ways of Working with Users to Develop Physically Assistive Robots. arxiv:2403.00489 [cs.HC]
[30]
Amal Nanavati, Vinitha Ranganeni, and Maya Cakmak. 2023. Physically Assistive Robots: A Systematic Review of Mobile and Manipulator Robots That Physically Assist People with Disabilities. Annual Review of Control, Robotics, and Autonomous Systems 7 (2023).
[31]
Akhil Padmanabha, Qin Wang, Daphne Han, Jashkumar Diyora, Kriti Kacker, Hamza Khalid, Liang-Jung Chen, Carmel Majidi, and Zackory Erickson. 2023. HAT: Head-Worn Assistive Teleoperation of Mobile Manipulators. In 2023 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 12542–12548.
[32]
Daehyung Park, Yuuna Hoshi, Harshal P Mahajan, Ho Keun Kim, Zackory Erickson, Wendy A Rogers, and Charles C Kemp. 2020. Active robot-assisted feeding with a general-purpose mobile manipulator: Design, evaluation, and lessons learned. Robotics and Autonomous Systems (2020).
[33]
Vinitha Ranganeni, Noah Ponto, and Maya Cakmak. 2023. Evaluating Customization of Remote Tele-operation Interfaces for Assistive Robots. In IEEE International Conference on Robot & Human Interactive Communication (RO-MAN).
[34]
Avi Singh, Eric Jang, Alexander Irpan, Daniel Kappler, Murtaza Dalal, Sergey Levinev, Mohi Khansari, and Chelsea Finn. 2020. Scalable multi-task imitation learning with autonomous improvement. In 2020 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2167–2173.
[35]
A. Stentz, H. Herman, A. Kelly, E. Meyhofer, G.C. Haynes, D. Stager, B. Zajac, J.A. Bagnell, J. Brindza, and C. Dellin. 2015. Chimp, the cmu highly intelligent mobile platform. Journal of Field Robotics 32, 2 (2015), 209–228.
[36]
Jacob O Wobbrock, Shaun K Kane, Krzysztof Z Gajos, Susumu Harada, and Jon Froehlich. 2011. Ability-based design: Concept, principles and examples. ACM Transactions on Accessible Computing (TACCESS) 3, 3 (2011), 1–27.
[37]
Kay N Wojtowicz and Maria E Cabrera. 2023. Stretch to the Client; Re-imagining Interfaces. In Companion of the 2023 ACM/IEEE International Conference on Human-Robot Interaction. 891–892.
[38]
Yiping Xie, Xilong Hou, and Shuangyi Wang. 2023. Design of a Novel Haptic Joystick for the Teleoperation of Continuum-Mechanism-Based Medical Robots. Robotics 12, 2 (2023), 52.

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Other conferences
UIST '24: Proceedings of the 37th Annual ACM Symposium on User Interface Software and Technology
October 2024
2334 pages
ISBN:9798400706288
DOI:10.1145/3654777
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 11 October 2024

Permissions

Request permissions for this article.

Check for updates

Qualifiers

  • Research-article
  • Research
  • Refereed limited

Funding Sources

  • NIH/NIA
  • NIBIB

Conference

UIST '24

Acceptance Rates

Overall Acceptance Rate 561 of 2,567 submissions, 22%

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • 0
    Total Citations
  • 346
    Total Downloads
  • Downloads (Last 12 months)346
  • Downloads (Last 6 weeks)136
Reflects downloads up to 05 Mar 2025

Other Metrics

Citations

View Options

Login options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

HTML Format

View this article in HTML Format.

HTML Format

Figures

Tables

Media

Share

Share

Share this Publication link

Share on social media