research-article

RobotIST: Interactive Situated Tangible Robot Programming

Authors:

Yasaman S. Sefidgar,

Heather Harvey,

Maya CakmakAuthors Info & Claims

SUI '18: Proceedings of the 2018 ACM Symposium on Spatial User Interaction

Pages 141 - 149

https://doi.org/10.1145/3267782.3267921

Published: 13 October 2018 Publication History

Abstract

Situated tangible robot programming allows programmers to reference parts of the workspace relevant to the task by indicating objects, locations, and regions of interest using tangible blocks. While it takes advantage of situatedness compared to traditional text-based and visual programming tools, it does not allow programmers to inspect what the robot detects in the workspace, nor to understand any programming or execution errors that may arise. In this work we propose to use a projector mounted on the robot to provide such functionality. This allows us to provide an interactive situated tangible programming experience, taking advantage of situatedness, both in user input and system output, to reference parts of the robot workspace. We describe an implementation and evaluation of this approach, highlighting its differences from traditional robot programming.

Supplementary Material

MP4 File (p141-sefidgar.mp4)

Download
176.06 MB

References

[1]

Baris Akgun, Maya Cakmak, Jae Wook Yoo, and Andrea Lockerd Thomaz. 2012. Trajectories and keyframes for kinesthetic teaching: A human-robot interaction perspective. In Proceedings of the seventh annual ACM/IEEE international conference on Human-Robot Interaction. 391--398.

Digital Library

[2]

S. Alexamdrova, M. Cakmak, K. Hsaio, and L. Takayama. 2014. Robot Programming by Demonstration with Interactive Action Visualizations. In Robotics: Science and Systems (RSS).

[3]

Sonya Alexandrova, Zachary Tatlock, and Maya Cakmak. 2015. RoboFlow: A flow-based visual programming language for mobile manipulation tasks. In 2015 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 5537--5544.

[4]

Rasmus S Andersen, Ole Madsen, Thomas B Moeslund, and Heni Ben Amor. 2016. Projecting robot intentions into human environments. In Robot and Human Interactive Communication (ROMAN), 2016 25th IEEE International Symposium on. IEEE, 294--301.

Digital Library

[5]

Brenna D Argall, Sonia Chernova, Manuela Veloso, and Brett Browning. 2009. A survey of robot learning from demonstration. Robotics and Autonomous Systems 57, 5 (2009), 469--483.

Digital Library

[6]

Emilia I Barakova, Jan CC Gillesen, Bibi EBM Huskens, and Tino Lourens. 2013. End-user programming architecture facilitates the uptake of robots in social therapies. Robotics and Autonomous Systems 61, 7 (2013), 704--713.

[7]

Chris Beckmann and Anind Dey. 2003. Siteview: Tangibly programming active environments with predictive visualization. In adjunct Proceedings of UbiComp. 167--168.

[8]

Antonio Bicchi and Vijay Kumar. 2000. Robotic grasping and contact: A review. In Robotics and Automation, 2000. Proceedings. ICRA'00. IEEE International Conference on, Vol. 1. IEEE, 348--353.

[9]

Aude Billard, Sylvain Calinon, Ruediger Dillmann, and Stefan Schaal. 2008. Robot programming by demonstration. In Springer Handbook of Robotics. Springer, 1371--1394.

[10]

Rehj Cantrell, Paul Schermerhorn, and Matthias Scheutz. 2011. Learning actions from human-robot dialogues. In 2011 ROMAN. IEEE, 125--130.

[11]

Ravi Teja Chadalavada, Henrik Andreasson, Robert Krug, and Achim J Lilienthal. 2015. That's on my mind! robot to human intention communication through onboard projection on shared floor space. In Mobile Robots (ECMR), 2015 European Conference on. IEEE, 1--6.

[12]

Sonia Chernova and Andrea L Thomaz. 2014. Robot learning from human teachers. Synthesis Lectures on Artificial Intelligence and Machine Learning 8, 3 (2014), 1--121.

Digital Library

[13]

L. Claassen, S. Aden, J. Gaa, J. Kotlarski, and T. Ortmaier. 2014. Intuitive Robot Control with a Projected Touch Interface. In Social Robotics. Springer, 95--104.

[14]

Jennifer Cross, Christopher Bartley, Emily Hamner, and Illah Nourbakhsh. 2013. A visual robot-programming environment for multidisciplinary education. In Robotics and Automation (ICRA), 2013 IEEE International Conference on. IEEE, 445--452.

[15]

James P Diprose, Bruce A MacDonald, and John G Hosking. 2011. Ruru: A spatial and interactive visual programming language for novice robot programming. In Visual Languages and Human-Centric Computing (VL/HCC), 2011 IEEE Symposium on. IEEE, 25--32.

[16]

Mohamed Elbanhawi and Milan Simic. 2014. Sampling-based robot motion planning: A review. Ieee access 2 (2014), 56--77.

[17]

Daniel Gallardo, Carles Fernandes Julià, and Sergi Jorda. 2008. TurTan: A tangible programming language for creative exploration. In Tabletop. Citeseer, 89--92.

[18]

Guglielmo Gemignani, Emanuele Bastianelli, and Daniele Nardi. 2015. Teaching robots parametrized executable plans through spoken interaction. In Proceedings of the 2015 International Conference on Autonomous Agents and Multiagent Systems. International Foundation for Autonomous Agents and Multiagent Systems, 851--859.

Digital Library

[19]

Dylan F Glas, Takayuki Kanda, and Hiroshi Ishiguro. 2016. Human-Robot Interaction Design Using Interaction Composer: Eight Years of Lessons Learned. In The Eleventh ACM/IEEE International Conference on Human Robot Interation. IEEE Press, 303--310.

Digital Library

[20]

J Harvent, Benjamin Coudrin, Ludovic BrÃČÅąthes, Jean-JosÃČÂl' Orteu, and Michel Devy. 2013. Multi-View Dense 3D Modelling of Untextured Objects From a Moving Projector-Cameras System. 24 (11 2013).

Digital Library

[21]

Michael S Horn and Robert JK Jacob. 2007. Designing tangible programming languages for classroom use. In Proceedings of the 1st international conference on Tangible and embedded interaction. ACM, 159--162.

Digital Library

[22]

Michael S Horn and Robert JK Jacob. 2007. Tangible programming in the classroom with tern. In CHI'07 extended abstracts on Human factors in computing systems. ACM, 1965--1970.

Digital Library

[23]

Michael S Horn, Erin Treacy Solovey, R Jordan Crouser, and Robert JK Jacob. 2009. Comparing the use of tangible and graphical programming languages for informal science education. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM, 975--984.

Digital Library

[24]

Justin Huang, Tessa Lau, and Maya Cakmak. 2016. Design and evaluation of a rapid programming system for service robots. In 2016 11th ACM/IEEE International Conference on Human-Robot Interaction (HRI). IEEE, 295--302.

Digital Library

[25]

Itseez. 2015. Open Source Computer Vision Library. http://opencv.org/

[26]

Majeed Kazemitabaar, Jason McPeak, Alexander Jiao, Liang He, Thomas Outing, and Jon E Froehlich. 2017. Makerwear: A tangible approach to interactive wearable creation for children. In Proceedings of the 2017 chi conference on human factors in computing systems. ACM, 133--145.

Digital Library

[27]

Scott R Klemmer, Jack Li, James Lin, and James A Landay. 2004. Papier-Mache: toolkit support for tangible input. In Proceedings of the SIGCHI conference on Human factors in computing systems. ACM, 399--406.

Digital Library

[28]

Jens Kober and Jan Peters. 2012. Reinforcement learning in robotics: A survey. In Reinforcement Learning. Springer, 579--610.

[29]

A. Kurenkov, B. Akgun, and A. L. Thomaz. 2015. An evaluation of GUI and kinesthetic teaching methods for constrained-keyframe skills. In Intelligent Robots and Systems (IROS), 2015 IEEE/RSJ International Conference on. 3608--3613.

[30]

Jean-Claude Latombe. 2012. Robot motion planning. Vol. 124. Springer Science & Business Media.

[31]

D. Lazewatsky and W.D. Smart. 2012. Context-sensitive in-the-world interfaces for mobile manipulation robots. In IEEE Intl. Symp. on Robot Human Communication (ROMAN). IEEE, 989--994.

[32]

Tomas Lozano-Perez. 1983. Robot programming. Proc. IEEE 71, 7 (1983), 821--841.

[33]

Michal Luria, Guy Hoffman, Benny Megidish, Oren Zuckerman, and Sung Park. 2016. Designing Vyo, a robotic Smart Home assistant: Bridging the gap between device and social agent. In Robot and Human Interactive Communication (ROMAN), 2016 25th IEEE International Symposium on. IEEE, 1019--1025.

Digital Library

[34]

Cynthia Matuszek, Evan Herbst, Luke Zettlemoyer, and Dieter Fox. 2013. Learning to parse natural language commands to a robot control system. In Experimental Robotics. Springer, 403--415.

[35]

Timothy S McNerney. 2004. From turtles to Tangible Programming Bricks: explorations in physical language design. Personal and Ubiquitous Computing 8, 5 (2004), 326--337.

[36]

Dipendra K Misra, Jaeyong Sung, Kevin Lee, and Ashutosh Saxena. 2016. Tell me dave: Context-sensitive grounding of natural language to manipulation instructions. The International Journal of Robotics Research 35, 1--3 (2016), 281--300.

Digital Library

[37]

Shiwali Mohn and John Laird. 2014. Learning Goal-Oriented Hierarchical Tasks from Situated Interactive Instruction. In Proceedings of the Twenty-eighth National Conference on Artificial Intelligence (AAAI).

Digital Library

[38]

David Molyneaux and Hans Gellersen. 2009. Projected interfaces: enabling serendipitous interaction with smart tangible objects. In Proceedings of the 3rd International Conference on Tangible and Embedded Interaction. ACM, 385--392.

Digital Library

[39]

Daniel Moreno and Gabriel Taubin. 2012. Simple,accurate,and robust projector-camera calibration. In 3D Imaging, Modeling, Processing, Visualization and Transmission (3DIMPVT), 2012 Second International Conference on. IEEE, 464--471.

Digital Library

[40]

Scott Niekum. 2013. Alvar. http://wiki.ros.org/ar_track_alvar

[41]

Morgan Quigley, Ken Conley, Brian Gerkey, Josh Faust, Tully Foote, Jeremy Leibs, Rob Wheeler, and Andrew Y Ng. 2009. ROS: an open-source Robot Operating System. In ICRA workshop on open source software, Vol. 3. Kobe, Japan, 5.

[42]

Jun Rekimoto, Brygg Ullmer, and Haruo Oba. 2001. DataTiles: a modular platform for mixed physical and graphical interactions. In Proceedings of the SIGCHI conference on Human factors in computing systems. ACM, 269--276.

Digital Library

[43]

Radu Bogdan Rusu and Steve Cousins. 2011. 3D is here: Point Cloud Library (PCL). In IEEE International Conference on Robotics and Automation (ICRA). Shanghai, China.

[44]

Theodosios Sapounidis, Stavros Demetriadis, and Ioannis Stamelos. 2015. Evaluating children performance with graphical and tangible robot programming tools. Personal and Ubiquitous Computing 19, 1 (2015), 225--237.

Digital Library

[45]

Allison Sauppé and Bilge Mutlu. 2014. Design patterns for exploring and prototyping human-robot interactions. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM, 1439--1448.

Digital Library

[46]

Yasaman S Sefidgar, Prerna Agarwal, and Maya Cakmak. 2017. Situated Tangible Robot Programming. In Proceedings of the 2017 ACM/IEEE International Conference on Human-Robot Interaction. ACM, 473--482.

Digital Library

[47]

Orit Shaer and Eva Hornecker. 2010. Tangible user interfaces: past, present, and future directions. Foundations and Trends in Human-Computer Interaction 3, 1--2 (2010), 1--137.

Digital Library

[48]

Karun B Shimoga. 1996. Robot grasp synthesis algorithms: A survey. The International Journal of Robotics Research 15, 3 (1996), 230--266.

Digital Library

[49]

Arnan Sipitakiat and Nusarin Nusen. 2012. Robo-Blocks: designing debugging abilities in a tangible programming system for early primary school children. In Proceedings of the 11th International Conference on Interaction Design and Children. ACM, 98--105.

Digital Library

[50]

Maj Stenmark, Mathias Haage, and Elin Anna Topp. 2017. Simplified Programming of Re-usable Skills on a Safe Industrial Robot: Prototype and Evaluation. In Proceedings of the 2017 ACM/IEEE International Conference on Human-Robot Interaction. ACM, 463--472.

Digital Library

[51]

Ioan A. Sucan and Sachin Chitta. 2011. MoveIt! http://moveit.ros.org

[52]

Brygg Ullmer and Hiroshi Ishii. 2000. Emerging frameworks for tangible user interfaces. IBM systems journal 39, 3.4 (2000), 915--931.

Digital Library

[53]

Robert van Herk, Janneke Verhaegh, and Willem FJ Fontijn. 2009. ESPranto SDK: an adaptive programming environment for tangible applications. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM, 849--858.

Digital Library

[54]

Manuela Waldner, Jörg Hauber, Jürgen Zauner, Michael Haller, and Mark Billinghurst. 2006. Tangible tiles: design and evaluation of a tangible user interface in a collaborative tabletop setup. In Proceedings of the 18th Australia conference on Computer-Human Interaction: Design: Activities, Artefacts and Environments. ACM, 151--158.

Digital Library

Cited By

Sefidgar Y(2024)Supporting Control and Alignment in Personal Informatics ToolsAdjunct Proceedings of the 37th Annual ACM Symposium on User Interface Software and Technology10.1145/3672539.3686709(1-4)Online publication date: 13-Oct-2024
https://dl.acm.org/doi/10.1145/3672539.3686709
Kapinus MBeran VMaterna ZBambušek D(2024)Augmented reality spatial programming paradigm applied to end-user robot programmingRobotics and Computer-Integrated Manufacturing10.1016/j.rcim.2024.10277089(102770)Online publication date: Oct-2024
https://doi.org/10.1016/j.rcim.2024.102770
Öztürk Ö(2024)A Methodology for end user programming of ROS-based service robots using jigsaw metaphor and ontologiesIntelligent Service Robotics10.1007/s11370-024-00528-317:4(745-757)Online publication date: 3-Apr-2024
https://doi.org/10.1007/s11370-024-00528-3
Show More Cited By

Index Terms

RobotIST: Interactive Situated Tangible Robot Programming
1. Human-centered computing
  1. Human computer interaction (HCI)
    1. Interactive systems and tools

Recommendations

End-User Programming of Manipulator Robots in Situated Tangible Programming Paradigm
HRI '18: Companion of the 2018 ACM/IEEE International Conference on Human-Robot Interaction

While the cost of creating robots is declining, deploying them in industry remains expensive. Widespread use of robots, particularly in smaller industries, is more easily realized if robot programming is accessible to non-programmers. Our research ...
Situated Tangible Robot Programming
HRI '17: Proceedings of the 2017 ACM/IEEE International Conference on Human-Robot Interaction

This paper introduces situated tangible robot programming, whereby a robot is programmed by placing specially designed tangible "blocks" in its workspace. These blocks are used for annotating objects, locations, or regions, and specifying actions and ...
PATI: a projection-based augmented table-top interface for robot programming
IUI '19: Proceedings of the 24th International Conference on Intelligent User Interfaces

As robots begin to provide daily assistance to individuals in human environments, their end-users, who do not necessarily have substantial technical training or backgrounds in robotics or programming, will ultimately need to program and "re-task" their ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

SUI '18: Proceedings of the 2018 ACM Symposium on Spatial User Interaction

October 2018

203 pages

ISBN:9781450357081

DOI:10.1145/3267782

Copyright © 2018 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 13 October 2018

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed limited

Conference

SUI '18

Sponsor:

SUI '18: Symposium on Spatial User Interaction

October 13 - 14, 2018

Berlin, Germany

Acceptance Rates

SUI '18 Paper Acceptance Rate 19 of 61 submissions, 31%;

Overall Acceptance Rate 86 of 279 submissions, 31%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

14
Total Citations
View Citations
262
Total Downloads

Downloads (Last 12 months)36
Downloads (Last 6 weeks)4

Reflects downloads up to 25 Feb 2025

Other Metrics

View Author Metrics

Citations

Cited By

Sefidgar Y(2024)Supporting Control and Alignment in Personal Informatics ToolsAdjunct Proceedings of the 37th Annual ACM Symposium on User Interface Software and Technology10.1145/3672539.3686709(1-4)Online publication date: 13-Oct-2024
https://dl.acm.org/doi/10.1145/3672539.3686709
Kapinus MBeran VMaterna ZBambušek D(2024)Augmented reality spatial programming paradigm applied to end-user robot programmingRobotics and Computer-Integrated Manufacturing10.1016/j.rcim.2024.10277089(102770)Online publication date: Oct-2024
https://doi.org/10.1016/j.rcim.2024.102770
Öztürk Ö(2024)A Methodology for end user programming of ROS-based service robots using jigsaw metaphor and ontologiesIntelligent Service Robotics10.1007/s11370-024-00528-317:4(745-757)Online publication date: 3-Apr-2024
https://doi.org/10.1007/s11370-024-00528-3
Bennett CSabanovic SStanojevic CHenkel ZKim SLee JBaugus KPiatt JYu JOh JCollins SBethel C(2023)Enabling Robotic Pets to Autonomously Adapt Their Own Behaviors to Enhance Therapeutic Effects: A Data-Driven Approach*2023 32nd IEEE International Conference on Robot and Human Interactive Communication (RO-MAN)10.1109/RO-MAN57019.2023.10309499(1625-1632)Online publication date: 28-Aug-2023
https://doi.org/10.1109/RO-MAN57019.2023.10309499
Wang DZhao YFeng S(2022)AR-C&P: A Tangible Programming for Children Based Augmented RealityProceedings of the Tenth International Symposium of Chinese CHI10.1145/3565698.3565778(141-150)Online publication date: 22-Oct-2022
https://dl.acm.org/doi/10.1145/3565698.3565778
Fogli DGargioni LGuida GTampalini F(2022)A hybrid approach to user-oriented programming of collaborative robotsRobotics and Computer-Integrated Manufacturing10.1016/j.rcim.2021.10223473:COnline publication date: 1-Feb-2022
https://dl.acm.org/doi/10.1016/j.rcim.2021.102234
Ritschel NSawant AWeintrop DHolmes RBacchelli AGarcia RK R CMandal AFrancis PShepherd D(2022)Training industrial end‐user programmers with interactive tutorialsSoftware: Practice and Experience10.1002/spe.316753:3(729-747)Online publication date: 11-Nov-2022
https://doi.org/10.1002/spe.3167
Ajaykumar GSteele MHuang C(2021)A Survey on End-User Robot ProgrammingACM Computing Surveys10.1145/346681954:8(1-36)Online publication date: 4-Oct-2021
https://dl.acm.org/doi/10.1145/3466819
Ajaykumar G(2021)Assisted End-User Robot ProgrammingProceedings of the 2021 International Conference on Multimodal Interaction10.1145/3462244.3481276(797-801)Online publication date: 18-Oct-2021
https://dl.acm.org/doi/10.1145/3462244.3481276
Mahadevan KSousa MTang AGrossman TKitamura YQuigley AIsbister KIgarashi TBjørn PDrucker S(2021)“Grip-that-there”: An Investigation of Explicit and Implicit Task Allocation Techniques for Human-Robot CollaborationProceedings of the 2021 CHI Conference on Human Factors in Computing Systems10.1145/3411764.3445355(1-14)Online publication date: 6-May-2021
https://dl.acm.org/doi/10.1145/3411764.3445355
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten