skip to main content
10.1145/2909824.3020215acmconferencesArticle/Chapter ViewAbstractPublication PageshriConference Proceedingsconference-collections
research-article
Public Access

Code3: A System for End-to-End Programming of Mobile Manipulator Robots for Novices and Experts

Published: 06 March 2017 Publication History

Abstract

This paper introduces Code3, a system for user-friendly, rapid programming of mobile manipulator robots. The system is designed to let non-roboticists and roboticists alike program end-to-end manipulation tasks. To accomplish this, Code3 provides three integrated components for perception, manipulation, and high-level programming. The perception component helps users define a library of object and scene parts that the robot can later detect. The manipulation component lets users define actions for manipulating objects or scene parts through programming by demonstration. Finally, the high-level programming component provides a drag-and-drop interface with which users can program the logic and control flow to accomplish a task using their previously specified perception and manipulation capabilities. We present findings from an observational user study with non-roboticist programmers (N=10) that demonstrate their ability to quickly learn Code3 and program a PR2 robot to do manipulation tasks. We also demonstrate how the system is expressive enough for an expert to rapidly program highly complex manipulation tasks like playing tic-tac-toe and reconfiguring an object to be graspable.

References

[1]
Beer me, robot. In Willow Garage Inc. Blog. http://www.willowgarage.com/blog/2010/07/06/beer-me-robot, 2010.
[2]
B. Akgun, M. Cakmak, J. W. Yoo, and A. L. Thomaz. Trajectories and keyframes for kinesthetic teaching: A human-robot interaction perspective. In Proceedings of the Seventh Annual ACM/IEEE International Conference on Human-Robot Interaction, pages 391--398, 2012.
[3]
S. Alexandrova, M. Cakmak, K. Hsiao, and L. Takayama. Robot programming by demonstration with interactive action visualizations. In Robotics: Science and Systems (RSS), pages 48--56, 2014.
[4]
S. Alexandrova, Z. Tatlock, and M. Cakmak. RoboFlow: A flow-based visual programming language for mobile manipulation tasks. In 2015 IEEE International Conference on Robotics and Automation (ICRA), pages 5537--5544. IEEE, 2015.
[5]
B. D. Argall, S. Chernova, M. Veloso, and B. Browning. A survey of robot learning from demonstration. Robotics and Autonomous Systems, 57(5):469--483, 2009.
[6]
C. G. Atkeson and S. Schaal. Robot learning from demonstration. In ICML, volume 97, pages 12--20, 1997.
[7]
A. Bangor, P. T. Kortum, and J. T. Miller. An empirical evaluation of the system usability scale. Intl. Journal of Human-Computer Interaction, 24(6):574--594, 2008.
[8]
P. J. Besl and N. D. McKay. A method for registration of 3-D shapes. IEEE Trans. Pattern Anal. Mach. Intell., 14:239--256, 1992.
[9]
A. Billard, S. Calinon, R. Dillmann, and S. Schaal. Robot programming by demonstration. In Springer Handbook of Robotics, pages 1371--1394. Springer, 2008.
[10]
J. Brooke et al. SUS-a quick and dirty usability scale. Usability Evaluation in Industry, 189(194):4--7, 1996.
[11]
M. Cakmak. Labs. CSE 481C website. https://sites.google.com/site/cse481a/labs, 2013.
[12]
M. Cakmak and A. L. Thomaz. Eliciting good teaching from humans for machine learners. Artificial Intelligence, 217:198--215, 2014.
[13]
S. Calinon and A. Billard. Statistical learning by imitation of competing constraints in joint space and task space. Advanced Robotics, 23(15):2059--2076, 2009.
[14]
S. Chernova and A. L. Thomaz. Robot learning from human teachers. Synthesis Lectures on Artificial Intelligence and Machine Learning, 8(3):1--121, 2014.
[15]
N. Correll, K. E. Bekris, D. Berenson, O. Brock, A. Causo, K. Hauser, K. Okada, A. Rodriguez, J. M. Romano, and P. R. Wurman. Analysis and observations from the first Amazon Picking Challenge. IEEE Transactions on Automation Science and Engineering, 2016.
[16]
C. Datta, C. Jayawardena, I. H. Kuo, and B. A. MacDonald. RoboStudio: A visual programming environment for rapid authoring and customization of complex services on a personal service robot. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, pages 2352--2357. IEEE, 2012.
[17]
S. Ekvall and D. Kragic. Robot learning from demonstration: A task-level planning approach. International Journal of Advanced Robotic Systems, 5(3):223--234, 2008.
[18]
M. A. Fischler and R. C. Bolles. Random sample consensus: A paradigm for model fitting with applications to image analysis and automated cartography. Communications of the ACM, 24(6):381--395, 1981.
[19]
D. Glas, S. Satake, T. Kanda, and N. Hagita. An interaction design framework for social robots. In Robotics: Science and Systems, volume 7, page 89, 2012.
[20]
D. F. Glas, T. Kanda, and H. Ishiguro. Human-robot interaction design using Interaction Composer: Eight years of lessons learned. In The Eleventh ACM/IEEE International Conference on Human Robot Interaction, pages 303--310. IEEE Press, 2016.
[21]
D. D. Hils. Visual languages and computing survey: Data flow visual programming languages. Journal of Visual Languages & Computing, 3(1):69--101, 1992.
[22]
J. Huang and M. Cakmak. Programming by demonstration with user-specified perceptual landmarks. arXiv preprint arXiv:1612.00565, 2016.
[23]
J. Huang, T. Lau, and M. Cakmak. Design and evaluation of a rapid programming system for service robots. In 2016 11th ACM/IEEE International Conference on Human-Robot Interaction (HRI), pages 295--302. IEEE, 2016.
[24]
E. S. Kim, D. Leyzberg, K. M. Tsui, and B. Scassellati. How people talk when teaching a robot. In Human-Robot Interaction (HRI), 2009 4th ACM/IEEE International Conference on, pages 23--30. IEEE, 2009.
[25]
T. Lourens. TiViPE-Tino's visual programming environment. In Computer Software and Applications Conference, 2004. COMPSAC 2004. Proceedings of the 28th Annual International, pages 10--15. IEEE, 2004.
[26]
T. Lourens and E. Barakova. User-friendly robot environment for creation of social scenarios. In International Work-Conference on the Interplay between Natural and Artificial Computation, pages 212--221. Springer, 2011.
[27]
T. Lozano-Perez. Robot programming. Proceedings of the IEEE, 71(7):821--841, 1983.
[28]
A. Mohseni-Kabir, C. Rich, S. Chernova, C. L. Sidner, and D. Miller. Interactive hierarchical task learning from a single demonstration. In Proceedings of the Tenth Annual ACM/IEEE International Conference on Human-Robot Interaction, pages 205--212. ACM, 2015.
[29]
H. Nguyen, M. Ciocarlie, K. Hsiao, and C. C. Kemp. ROS Commander (ROSCo): Behavior creation for home robots. In Robotics and Automation (ICRA), 2013 IEEE International Conference on, pages 467--474. IEEE, 2013.
[30]
S. Niekum, S. Osentoski, G. Konidaris, and A. G. Barto. Learning and generalization of complex tasks from unstructured demonstrations. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 5239--5246, 2012.
[31]
M. Pardowitz, S. Knoop, R. Dillmann, and R. Zollner. Incremental learning of tasks from user demonstrations, past experiences, and vocal comments. Systems, Man, and Cybernetics, Part B: Cybernetics, IEEE Transactions on, 37(2):322--332, 2007.
[32]
E. Pot, J. Monceaux, R. Gelin, and B. Maisonnier. Choregraphe: A graphical tool for humanoid robot programming. In RO-MAN 2009-The 18th IEEE International Symposium on Robot and Human Interactive Communication, pages 46--51. IEEE, 2009.
[33]
M. Quigley, K. Conley, B. Gerkey, J. Faust, T. Foote, J. Leibs, R. Wheeler, and A. Y. Ng. ROS: an open-source robot operating system. In ICRA Workshop on Open Source Software, volume 3, page 5. Kobe, Japan, 2009.
[34]
A. Sauppé and B. Mutlu. Design patterns for exploring and prototyping human-robot interactions. In Proceedings of the 32nd Annual ACM Conference on Human Factors in Computing Systems, pages 1439--1448. ACM, 2014.
[35]
S. Schaal, J. Peters, J. Nakanishi, and A. Ijspeert. Learning movement primitives. In International Symposium on Robotics Research., pages 561--572, 2005.
[36]
H. B. Suay, R. Toris, and S. Chernova. A practical comparison of three robot learning from demonstration algorithm. International Journal of Social Robotics, 4(4):319--330, 2012.
[37]
A. L. Thomaz and C. Breazeal. Experiments in socially guided exploration: Lessons learned in building robots that learn with and without human teachers. Connection Science, 20(2--3):91--110, 2008.

Cited By

View all
  • (2025)Preparation of Personalized Medicines through Collaborative Robots: A Hybrid Approach to the End-User Development of Robot ProgramsACM Journal on Responsible Computing10.1145/3715852Online publication date: 28-Jan-2025
  • (2024)Design Goals for End-User Development of Robot-Assisted Physical Training Activities: A Participatory Design StudyProceedings of the ACM on Human-Computer Interaction10.1145/36646328:EICS(1-31)Online publication date: 17-Jun-2024
  • (2024)Understanding On-the-Fly End-User Robot ProgrammingProceedings of the 2024 ACM Designing Interactive Systems Conference10.1145/3643834.3660721(2468-2480)Online publication date: 1-Jul-2024
  • Show More Cited By

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences
HRI '17: Proceedings of the 2017 ACM/IEEE International Conference on Human-Robot Interaction
March 2017
510 pages
ISBN:9781450343367
DOI:10.1145/2909824
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 06 March 2017

Permissions

Request permissions for this article.

Check for updates

Author Tags

  1. end-user programming
  2. mobile manipulation
  3. programming by demonstration
  4. programming tools

Qualifiers

  • Research-article

Funding Sources

Conference

HRI '17
Sponsor:

Acceptance Rates

HRI '17 Paper Acceptance Rate 51 of 211 submissions, 24%;
Overall Acceptance Rate 268 of 1,124 submissions, 24%

Upcoming Conference

HRI '25
ACM/IEEE International Conference on Human-Robot Interaction
March 4 - 6, 2025
Melbourne , VIC , Australia

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)249
  • Downloads (Last 6 weeks)34
Reflects downloads up to 15 Feb 2025

Other Metrics

Citations

Cited By

View all
  • (2025)Preparation of Personalized Medicines through Collaborative Robots: A Hybrid Approach to the End-User Development of Robot ProgramsACM Journal on Responsible Computing10.1145/3715852Online publication date: 28-Jan-2025
  • (2024)Design Goals for End-User Development of Robot-Assisted Physical Training Activities: A Participatory Design StudyProceedings of the ACM on Human-Computer Interaction10.1145/36646328:EICS(1-31)Online publication date: 17-Jun-2024
  • (2024)Understanding On-the-Fly End-User Robot ProgrammingProceedings of the 2024 ACM Designing Interactive Systems Conference10.1145/3643834.3660721(2468-2480)Online publication date: 1-Jul-2024
  • (2024)Integrating ChatGPT with Blockly for End-User Development of Robot TasksCompanion of the 2024 ACM/IEEE International Conference on Human-Robot Interaction10.1145/3610978.3640653(478-482)Online publication date: 11-Mar-2024
  • (2024)End-User Programming of Robot-Assisted Physical Training Activities through Behaviour-Driven DevelopmentCompanion of the 2024 ACM/IEEE International Conference on Human-Robot Interaction10.1145/3610978.3640618(387-391)Online publication date: 11-Mar-2024
  • (2024)Goal-Oriented End-User Programming of RobotsProceedings of the 2024 ACM/IEEE International Conference on Human-Robot Interaction10.1145/3610977.3634974(582-591)Online publication date: 11-Mar-2024
  • (2024)Alchemist: LLM-Aided End-User Development of Robot ApplicationsProceedings of the 2024 ACM/IEEE International Conference on Human-Robot Interaction10.1145/3610977.3634969(361-370)Online publication date: 11-Mar-2024
  • (2024)Design and usability evaluation of the EvaSIM simulator for a socially assistive robotMultimedia Tools and Applications10.1007/s11042-024-18338-183:27(69897-69922)Online publication date: 3-Feb-2024
  • (2023)Enabling End-Users in Designing and Executing of Complex, Collaborative Robotic ProcessesApplied System Innovation10.3390/asi60300566:3(56)Online publication date: 12-May-2023
  • (2023)Crowdsourcing Task Traces for Service RoboticsCompanion of the 2023 ACM/IEEE International Conference on Human-Robot Interaction10.1145/3568294.3580112(389-393)Online publication date: 13-Mar-2023
  • Show More Cited By

View Options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Login options

Figures

Tables

Media

Share

Share

Share this Publication link

Share on social media