skip to main content
10.1145/3267782.3267921acmconferencesArticle/Chapter ViewAbstractPublication PagessuiConference Proceedingsconference-collections
research-article

RobotIST: Interactive Situated Tangible Robot Programming

Published: 13 October 2018 Publication History

Abstract

Situated tangible robot programming allows programmers to reference parts of the workspace relevant to the task by indicating objects, locations, and regions of interest using tangible blocks. While it takes advantage of situatedness compared to traditional text-based and visual programming tools, it does not allow programmers to inspect what the robot detects in the workspace, nor to understand any programming or execution errors that may arise. In this work we propose to use a projector mounted on the robot to provide such functionality. This allows us to provide an interactive situated tangible programming experience, taking advantage of situatedness, both in user input and system output, to reference parts of the robot workspace. We describe an implementation and evaluation of this approach, highlighting its differences from traditional robot programming.

Supplementary Material

MP4 File (p141-sefidgar.mp4)

References

[1]
Baris Akgun, Maya Cakmak, Jae Wook Yoo, and Andrea Lockerd Thomaz. 2012. Trajectories and keyframes for kinesthetic teaching: A human-robot interaction perspective. In Proceedings of the seventh annual ACM/IEEE international conference on Human-Robot Interaction. 391--398.
[2]
S. Alexamdrova, M. Cakmak, K. Hsaio, and L. Takayama. 2014. Robot Programming by Demonstration with Interactive Action Visualizations. In Robotics: Science and Systems (RSS).
[3]
Sonya Alexandrova, Zachary Tatlock, and Maya Cakmak. 2015. RoboFlow: A flow-based visual programming language for mobile manipulation tasks. In 2015 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 5537--5544.
[4]
Rasmus S Andersen, Ole Madsen, Thomas B Moeslund, and Heni Ben Amor. 2016. Projecting robot intentions into human environments. In Robot and Human Interactive Communication (ROMAN), 2016 25th IEEE International Symposium on. IEEE, 294--301.
[5]
Brenna D Argall, Sonia Chernova, Manuela Veloso, and Brett Browning. 2009. A survey of robot learning from demonstration. Robotics and Autonomous Systems 57, 5 (2009), 469--483.
[6]
Emilia I Barakova, Jan CC Gillesen, Bibi EBM Huskens, and Tino Lourens. 2013. End-user programming architecture facilitates the uptake of robots in social therapies. Robotics and Autonomous Systems 61, 7 (2013), 704--713.
[7]
Chris Beckmann and Anind Dey. 2003. Siteview: Tangibly programming active environments with predictive visualization. In adjunct Proceedings of UbiComp. 167--168.
[8]
Antonio Bicchi and Vijay Kumar. 2000. Robotic grasping and contact: A review. In Robotics and Automation, 2000. Proceedings. ICRA'00. IEEE International Conference on, Vol. 1. IEEE, 348--353.
[9]
Aude Billard, Sylvain Calinon, Ruediger Dillmann, and Stefan Schaal. 2008. Robot programming by demonstration. In Springer Handbook of Robotics. Springer, 1371--1394.
[10]
Rehj Cantrell, Paul Schermerhorn, and Matthias Scheutz. 2011. Learning actions from human-robot dialogues. In 2011 ROMAN. IEEE, 125--130.
[11]
Ravi Teja Chadalavada, Henrik Andreasson, Robert Krug, and Achim J Lilienthal. 2015. That's on my mind! robot to human intention communication through onboard projection on shared floor space. In Mobile Robots (ECMR), 2015 European Conference on. IEEE, 1--6.
[12]
Sonia Chernova and Andrea L Thomaz. 2014. Robot learning from human teachers. Synthesis Lectures on Artificial Intelligence and Machine Learning 8, 3 (2014), 1--121.
[13]
L. Claassen, S. Aden, J. Gaa, J. Kotlarski, and T. Ortmaier. 2014. Intuitive Robot Control with a Projected Touch Interface. In Social Robotics. Springer, 95--104.
[14]
Jennifer Cross, Christopher Bartley, Emily Hamner, and Illah Nourbakhsh. 2013. A visual robot-programming environment for multidisciplinary education. In Robotics and Automation (ICRA), 2013 IEEE International Conference on. IEEE, 445--452.
[15]
James P Diprose, Bruce A MacDonald, and John G Hosking. 2011. Ruru: A spatial and interactive visual programming language for novice robot programming. In Visual Languages and Human-Centric Computing (VL/HCC), 2011 IEEE Symposium on. IEEE, 25--32.
[16]
Mohamed Elbanhawi and Milan Simic. 2014. Sampling-based robot motion planning: A review. Ieee access 2 (2014), 56--77.
[17]
Daniel Gallardo, Carles Fernandes Julià, and Sergi Jorda. 2008. TurTan: A tangible programming language for creative exploration. In Tabletop. Citeseer, 89--92.
[18]
Guglielmo Gemignani, Emanuele Bastianelli, and Daniele Nardi. 2015. Teaching robots parametrized executable plans through spoken interaction. In Proceedings of the 2015 International Conference on Autonomous Agents and Multiagent Systems. International Foundation for Autonomous Agents and Multiagent Systems, 851--859.
[19]
Dylan F Glas, Takayuki Kanda, and Hiroshi Ishiguro. 2016. Human-Robot Interaction Design Using Interaction Composer: Eight Years of Lessons Learned. In The Eleventh ACM/IEEE International Conference on Human Robot Interation. IEEE Press, 303--310.
[20]
J Harvent, Benjamin Coudrin, Ludovic BrÃČÅąthes, Jean-JosÃČÂl' Orteu, and Michel Devy. 2013. Multi-View Dense 3D Modelling of Untextured Objects From a Moving Projector-Cameras System. 24 (11 2013).
[21]
Michael S Horn and Robert JK Jacob. 2007. Designing tangible programming languages for classroom use. In Proceedings of the 1st international conference on Tangible and embedded interaction. ACM, 159--162.
[22]
Michael S Horn and Robert JK Jacob. 2007. Tangible programming in the classroom with tern. In CHI'07 extended abstracts on Human factors in computing systems. ACM, 1965--1970.
[23]
Michael S Horn, Erin Treacy Solovey, R Jordan Crouser, and Robert JK Jacob. 2009. Comparing the use of tangible and graphical programming languages for informal science education. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM, 975--984.
[24]
Justin Huang, Tessa Lau, and Maya Cakmak. 2016. Design and evaluation of a rapid programming system for service robots. In 2016 11th ACM/IEEE International Conference on Human-Robot Interaction (HRI). IEEE, 295--302.
[25]
Itseez. 2015. Open Source Computer Vision Library. http://opencv.org/
[26]
Majeed Kazemitabaar, Jason McPeak, Alexander Jiao, Liang He, Thomas Outing, and Jon E Froehlich. 2017. Makerwear: A tangible approach to interactive wearable creation for children. In Proceedings of the 2017 chi conference on human factors in computing systems. ACM, 133--145.
[27]
Scott R Klemmer, Jack Li, James Lin, and James A Landay. 2004. Papier-Mache: toolkit support for tangible input. In Proceedings of the SIGCHI conference on Human factors in computing systems. ACM, 399--406.
[28]
Jens Kober and Jan Peters. 2012. Reinforcement learning in robotics: A survey. In Reinforcement Learning. Springer, 579--610.
[29]
A. Kurenkov, B. Akgun, and A. L. Thomaz. 2015. An evaluation of GUI and kinesthetic teaching methods for constrained-keyframe skills. In Intelligent Robots and Systems (IROS), 2015 IEEE/RSJ International Conference on. 3608--3613.
[30]
Jean-Claude Latombe. 2012. Robot motion planning. Vol. 124. Springer Science & Business Media.
[31]
D. Lazewatsky and W.D. Smart. 2012. Context-sensitive in-the-world interfaces for mobile manipulation robots. In IEEE Intl. Symp. on Robot Human Communication (ROMAN). IEEE, 989--994.
[32]
Tomas Lozano-Perez. 1983. Robot programming. Proc. IEEE 71, 7 (1983), 821--841.
[33]
Michal Luria, Guy Hoffman, Benny Megidish, Oren Zuckerman, and Sung Park. 2016. Designing Vyo, a robotic Smart Home assistant: Bridging the gap between device and social agent. In Robot and Human Interactive Communication (ROMAN), 2016 25th IEEE International Symposium on. IEEE, 1019--1025.
[34]
Cynthia Matuszek, Evan Herbst, Luke Zettlemoyer, and Dieter Fox. 2013. Learning to parse natural language commands to a robot control system. In Experimental Robotics. Springer, 403--415.
[35]
Timothy S McNerney. 2004. From turtles to Tangible Programming Bricks: explorations in physical language design. Personal and Ubiquitous Computing 8, 5 (2004), 326--337.
[36]
Dipendra K Misra, Jaeyong Sung, Kevin Lee, and Ashutosh Saxena. 2016. Tell me dave: Context-sensitive grounding of natural language to manipulation instructions. The International Journal of Robotics Research 35, 1--3 (2016), 281--300.
[37]
Shiwali Mohn and John Laird. 2014. Learning Goal-Oriented Hierarchical Tasks from Situated Interactive Instruction. In Proceedings of the Twenty-eighth National Conference on Artificial Intelligence (AAAI).
[38]
David Molyneaux and Hans Gellersen. 2009. Projected interfaces: enabling serendipitous interaction with smart tangible objects. In Proceedings of the 3rd International Conference on Tangible and Embedded Interaction. ACM, 385--392.
[39]
Daniel Moreno and Gabriel Taubin. 2012. Simple,accurate,and robust projector-camera calibration. In 3D Imaging, Modeling, Processing, Visualization and Transmission (3DIMPVT), 2012 Second International Conference on. IEEE, 464--471.
[40]
Scott Niekum. 2013. Alvar. http://wiki.ros.org/ar_track_alvar
[41]
Morgan Quigley, Ken Conley, Brian Gerkey, Josh Faust, Tully Foote, Jeremy Leibs, Rob Wheeler, and Andrew Y Ng. 2009. ROS: an open-source Robot Operating System. In ICRA workshop on open source software, Vol. 3. Kobe, Japan, 5.
[42]
Jun Rekimoto, Brygg Ullmer, and Haruo Oba. 2001. DataTiles: a modular platform for mixed physical and graphical interactions. In Proceedings of the SIGCHI conference on Human factors in computing systems. ACM, 269--276.
[43]
Radu Bogdan Rusu and Steve Cousins. 2011. 3D is here: Point Cloud Library (PCL). In IEEE International Conference on Robotics and Automation (ICRA). Shanghai, China.
[44]
Theodosios Sapounidis, Stavros Demetriadis, and Ioannis Stamelos. 2015. Evaluating children performance with graphical and tangible robot programming tools. Personal and Ubiquitous Computing 19, 1 (2015), 225--237.
[45]
Allison Sauppé and Bilge Mutlu. 2014. Design patterns for exploring and prototyping human-robot interactions. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM, 1439--1448.
[46]
Yasaman S Sefidgar, Prerna Agarwal, and Maya Cakmak. 2017. Situated Tangible Robot Programming. In Proceedings of the 2017 ACM/IEEE International Conference on Human-Robot Interaction. ACM, 473--482.
[47]
Orit Shaer and Eva Hornecker. 2010. Tangible user interfaces: past, present, and future directions. Foundations and Trends in Human-Computer Interaction 3, 1--2 (2010), 1--137.
[48]
Karun B Shimoga. 1996. Robot grasp synthesis algorithms: A survey. The International Journal of Robotics Research 15, 3 (1996), 230--266.
[49]
Arnan Sipitakiat and Nusarin Nusen. 2012. Robo-Blocks: designing debugging abilities in a tangible programming system for early primary school children. In Proceedings of the 11th International Conference on Interaction Design and Children. ACM, 98--105.
[50]
Maj Stenmark, Mathias Haage, and Elin Anna Topp. 2017. Simplified Programming of Re-usable Skills on a Safe Industrial Robot: Prototype and Evaluation. In Proceedings of the 2017 ACM/IEEE International Conference on Human-Robot Interaction. ACM, 463--472.
[51]
Ioan A. Sucan and Sachin Chitta. 2011. MoveIt! http://moveit.ros.org
[52]
Brygg Ullmer and Hiroshi Ishii. 2000. Emerging frameworks for tangible user interfaces. IBM systems journal 39, 3.4 (2000), 915--931.
[53]
Robert van Herk, Janneke Verhaegh, and Willem FJ Fontijn. 2009. ESPranto SDK: an adaptive programming environment for tangible applications. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM, 849--858.
[54]
Manuela Waldner, Jörg Hauber, Jürgen Zauner, Michael Haller, and Mark Billinghurst. 2006. Tangible tiles: design and evaluation of a tangible user interface in a collaborative tabletop setup. In Proceedings of the 18th Australia conference on Computer-Human Interaction: Design: Activities, Artefacts and Environments. ACM, 151--158.

Cited By

View all
  • (2024)Supporting Control and Alignment in Personal Informatics ToolsAdjunct Proceedings of the 37th Annual ACM Symposium on User Interface Software and Technology10.1145/3672539.3686709(1-4)Online publication date: 13-Oct-2024
  • (2024)Augmented reality spatial programming paradigm applied to end-user robot programmingRobotics and Computer-Integrated Manufacturing10.1016/j.rcim.2024.10277089(102770)Online publication date: Oct-2024
  • (2024)A Methodology for end user programming of ROS-based service robots using jigsaw metaphor and ontologiesIntelligent Service Robotics10.1007/s11370-024-00528-317:4(745-757)Online publication date: 3-Apr-2024
  • Show More Cited By

Index Terms

  1. RobotIST: Interactive Situated Tangible Robot Programming

    Recommendations

    Comments

    Information & Contributors

    Information

    Published In

    cover image ACM Conferences
    SUI '18: Proceedings of the 2018 ACM Symposium on Spatial User Interaction
    October 2018
    203 pages
    ISBN:9781450357081
    DOI:10.1145/3267782
    Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

    Sponsors

    Publisher

    Association for Computing Machinery

    New York, NY, United States

    Publication History

    Published: 13 October 2018

    Permissions

    Request permissions for this article.

    Check for updates

    Author Tags

    1. Direct Manipulation
    2. Robot Programming
    3. Situated Programming
    4. Tangible Programming
    5. Transparency

    Qualifiers

    • Research-article
    • Research
    • Refereed limited

    Conference

    SUI '18
    SUI '18: Symposium on Spatial User Interaction
    October 13 - 14, 2018
    Berlin, Germany

    Acceptance Rates

    SUI '18 Paper Acceptance Rate 19 of 61 submissions, 31%;
    Overall Acceptance Rate 86 of 279 submissions, 31%

    Contributors

    Other Metrics

    Bibliometrics & Citations

    Bibliometrics

    Article Metrics

    • Downloads (Last 12 months)36
    • Downloads (Last 6 weeks)4
    Reflects downloads up to 25 Feb 2025

    Other Metrics

    Citations

    Cited By

    View all
    • (2024)Supporting Control and Alignment in Personal Informatics ToolsAdjunct Proceedings of the 37th Annual ACM Symposium on User Interface Software and Technology10.1145/3672539.3686709(1-4)Online publication date: 13-Oct-2024
    • (2024)Augmented reality spatial programming paradigm applied to end-user robot programmingRobotics and Computer-Integrated Manufacturing10.1016/j.rcim.2024.10277089(102770)Online publication date: Oct-2024
    • (2024)A Methodology for end user programming of ROS-based service robots using jigsaw metaphor and ontologiesIntelligent Service Robotics10.1007/s11370-024-00528-317:4(745-757)Online publication date: 3-Apr-2024
    • (2023)Enabling Robotic Pets to Autonomously Adapt Their Own Behaviors to Enhance Therapeutic Effects: A Data-Driven Approach*2023 32nd IEEE International Conference on Robot and Human Interactive Communication (RO-MAN)10.1109/RO-MAN57019.2023.10309499(1625-1632)Online publication date: 28-Aug-2023
    • (2022)AR-C&P: A Tangible Programming for Children Based Augmented RealityProceedings of the Tenth International Symposium of Chinese CHI10.1145/3565698.3565778(141-150)Online publication date: 22-Oct-2022
    • (2022)A hybrid approach to user-oriented programming of collaborative robotsRobotics and Computer-Integrated Manufacturing10.1016/j.rcim.2021.10223473:COnline publication date: 1-Feb-2022
    • (2022)Training industrial end‐user programmers with interactive tutorialsSoftware: Practice and Experience10.1002/spe.316753:3(729-747)Online publication date: 11-Nov-2022
    • (2021)A Survey on End-User Robot ProgrammingACM Computing Surveys10.1145/346681954:8(1-36)Online publication date: 4-Oct-2021
    • (2021)Assisted End-User Robot ProgrammingProceedings of the 2021 International Conference on Multimodal Interaction10.1145/3462244.3481276(797-801)Online publication date: 18-Oct-2021
    • (2021)“Grip-that-there”: An Investigation of Explicit and Implicit Task Allocation Techniques for Human-Robot CollaborationProceedings of the 2021 CHI Conference on Human Factors in Computing Systems10.1145/3411764.3445355(1-14)Online publication date: 6-May-2021
    • Show More Cited By

    View Options

    Login options

    View options

    PDF

    View or Download as a PDF file.

    PDF

    eReader

    View online with eReader.

    eReader

    Figures

    Tables

    Media

    Share

    Share

    Share this Publication link

    Share on social media