Skip to main content
SpringerLink
Log in

Synthesizing Robot Programs with Interactive Tutor Mode

  • Research Article
  • Published:
International Journal of Automation and Computing Aims and scope Submit manuscript

Abstract

With the rapid development of the robotic industry, domestic robots have become increasingly popular. As domestic robots are expected to be personal assistants, it is important to develop a natural language-based human-robot interactive system for end-users who do not necessarily have much programming knowledge. To build such a system, we developed an interactive tutoring framework, named “Holert”, which can translate task descriptions in natural language to machine-interpretable logical forms automatically. Compared to previous works, Holert allows users to teach the robot by further explaining their intentions in an interactive tutor mode. Furthermore, Holert introduces a semantic dependency model to enable the robot to “understand” similar task descriptions. We have deployed Holert on an open-source robot platform, Turtlebot 2. Experimental results show that the system accuracy could be significantly improved by 163.9% with the support of the tutor mode. This system is also efficient. Even the longest task session with 10 sentences can be handled within 0.7 s.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. J. Scholtz. Theory and evaluation of human robot interactions. In Proceedings of the 36th Annual Hawaii International Conference on System Sciences, IEEE, Big Island, USA, 2003. DOI: https://doi.org/10.1109/HICSS.2003.1174284.

    Google Scholar 

  2. I. G. Alonso, M. Fernández, J. M. Maestre, M. del Pilar Almudena García Fuente. Service Robotics within the Digital Home: Applications and Future Prospects, Dordrecht, Holland: Springer, 2011. DOI: https://doi.org/10.1007/978-94-007-1491-5.

    Book  Google Scholar 

  3. R Borja, J. R. De La Pinta, A. Álvarez, J. M. Maestre. Integration of service robots in the smart home by means of UPnP: A surveillance robot case study. Robotics and Autonomous Systems, vol. 61, no. 2, pp. 153–160, 2013. DOI: https://doi.org/10.1016/j.robot.2012.10.005.

    Article  Google Scholar 

  4. C. Zhou, M. H. Jin, Y. C. Liu, Z. Zhang, Y. Liu, H. Liu. Singularity robust path planning for real time base attitude adjustment of free-floating space robot. International Journal of Automation and Computing, vol. 14, no. 2, pp. 169–178, 2017. DOI: https://doi.org/10.1007/s11633-017-1055-1.

    Article  Google Scholar 

  5. K. C. D. Fu, Y. Nakamura, T. Yamamoto, H. Ishiguro. Analysis of motor synergies utilization for optimal movement generation for a human-like robotic arm. International Journal of Automation and Computing, vol. 10, no. 6, pp. 515–524, 2013. DOI: https://doi.org/10.1007/s11633-013-0749-2.

    Article  Google Scholar 

  6. S. Alexandrova, Z. Tatlock, M. Cakmak. RoboFlow: A flow-based visual programming language for mobile manipulation tasks. In Proceedings of IEEE International Conference on Robotics and Automation, Seattle, USA, pp. 5537–5544, 2015. DOI: https://doi.org/10.1109/ICRA.2015.7139973.

    Google Scholar 

  7. C. Datta, C. Jayawardena, I. H. Kuo, B. A. MacDonald. RoboStudio: A visual programming environment for rapid authoring and customization of complex services on a personal service robot. In Proceedings of IEEE/RSJ International Conference on Intelligent Robots and Systems, Vilamoura, Portugal, pp. 2352–2357, 2012. DOI: https://doi.org/10.1109/IROS.2012.6386105.

    Google Scholar 

  8. J. Li, A. Q. Xu, G. Dudek. Graphical state space programming: A visual programming paradigm for robot task specification. In Proceedings of IEEE International Confer-ence on Robotics and Automation, Shanghai, China, pp. 4846–4853, 2011. DOI: https://doi.org/10.1109/ICRA.2011.5979630.

    Google Scholar 

  9. M. A. Goodrich, A. C. Schultz. Human-robot interaction: A survey. Foundations and Trends in Human-computer Interaction, vol. 1, no. 3, pp. 203–275, 2007. DOI: https://doi.org/10.1561/1100000005.

    Article  Google Scholar 

  10. J. Dzifcak, M. Scheutz, C. Baral, P. Schermerhorn. What to do and how to do it: Translating natural language directives into temporal and dynamic logic representation for goal management and action execution. In Proceedings of IEEE International Conference on Robotics and Automation, Kobe, Japan, pp. 4163–4168, 2009. DOI: https://doi.org/10.1109/ROBOT.2009.5152776.

    Google Scholar 

  11. C. Matuszek, E. Herbst, L. Zettlemoyer, D. Fox. Learning to parse natural language commands to a robot control system. In Proceedings of the 13th International Symposium on Experimental Robotics, Springer, Heidelberg, Germany, 403–415, 2013. DOI: https://doi.org/10.1007/978-3-319-00065-7_28.

    Chapter  Google Scholar 

  12. R. F. Ge, R. J. Mooney. A statistical semantic parser that integrates syntax and semantics. In Proceedings of the 9th Conference on Computational Natural Language Learning, Association for Computational Linguistics, Ann Arbor, USA, 9–16, 2005. DOI: https://doi.org/10.3115/1706543.1706546.

    Google Scholar 

  13. K. Zhao, L. Huang. Type-driven incremental semantic parsing with polymorphism. In Proceedings of Human Language Technologies: The Annual Conference of the North American Chapter of the ACL, Denver, USA, 2015. DOI: https://doi.org/10.3115/v1/N15-1162.

    Google Scholar 

  14. Y. Artzi, L. Zettlemoyer. Bootstrapping semantic parsers from conversations. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, Edinburgh, UK, 421–432, 2011.

    Google Scholar 

  15. J. Berant, A. Chou, R. Frostig, P. Liang. Semantic parsing on freebase from question-answer pairs. In Proceedings of Conference on Empirical Methods in Natural Language Processing, Washington, USA, 2013.

    Google Scholar 

  16. L. Dong, M. Lapata. Language to logical form with neural attention. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, Berlin, Germany, 2016. DOI: https://doi.org/10.18653/v1/P16-1004.

    Google Scholar 

  17. Y. Kim. Convolutional neural networks for sentence classification. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, Doha, Qatar, 2014. DOI: https://doi.org/10.3115/v1/D14-1181.

    Google Scholar 

  18. Y. N. Dauphin, D. Z. Hakkani-Tur, G. Tur, L. P. Heck. Deep learning for semantic parsing including semantic utterance classification, USA. Patent 20150310862, October 2015.

    Google Scholar 

  19. R. Collobert, J. Weston. A unified architecture for natural language processing: Deep neural networks with multitask learning. In Proceedings of the 25th International Conference on Machine Learning, ACM, Helsinki, Finland, 160–167, 2008. DOI: https://doi.org/10.1145/1390156.1390177.

    Google Scholar 

  20. Z. P. Tu, Z. D. Lu, Y. Liu, X. H. Liu, H. Li. Modeling coverage for neural machine translation. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, Berlin, Germany, 2016. DOI: https://doi.org/10.18653/v1/P16-1008.

    Google Scholar 

  21. A. M. Rush, S. Chopra, J. Weston. A neural attention model for sentence summarization. In Proceedings of Conference on Empirical Methods in Natural Language Processing, Lisbon, Portugal, 2015. DOI: https://doi.org/10.18653/v1/D15-1044.

    Google Scholar 

  22. M. Quigley, K. Conley, B. Gerkey, J. Faust, T. Foote, J. Leibs, R. Wheeler, A. Y. Ng. ROS: An open-source robot operating system. In Proceedings of ICRA Workshop on Open Source Software, Kobe, Japan, 2009.

    Google Scholar 

  23. A. Voutilainen. Part-of-speech tagging. The Oxford Handbook of Computational Linguistics, R. Mitkov, Ed., Oxford, UK: Oxford University Press, 219–232, 2003.

    Google Scholar 

  24. D. Q. Chen, C. Manning. A fast and accurate dependency parser using neural networks. In Proceedings of Conference on Empirical Methods in Natural Language Processing, Association for Computational Linguistics, Doha, Qatar, 2014.

    Google Scholar 

  25. G. Look, B. Kottahachchi, R. Laddaga, H. Shrobe. A location representation for generating descriptive walking directions. In Proceedings of the 10th International Conference on Intelligent User Interfaces, ACM, San Diego, USA, 122–129, 2005. DOI: https://doi.org/10.1145/1040830.1040862.

    Chapter  Google Scholar 

  26. T. Kollar, S. Tellex, D. Roy, N. Roy. Toward understanding natural language directions. In Proceedings of the 5th ACM/IEEE International Conference on Human-robot Interaction, Osaka, Japan, 259–266, 2010. DOI: https://doi.org/10.1109/HRI.2010.5453186.

    Google Scholar 

  27. G. Bugmann, E. Klein, S. Lauria, T. Kyriacou. Corpusbased robotics: A route instruction example. In Proceedings of the 8th International Conference on Intelligent Autonomous Systems, Amsterdam, Netherlands, 96–103, 2004.

    Google Scholar 

  28. M. MacMahon, B. Stankiewicz, B. Kuipers. Walk the talk: Connecting language, knowledge, and action in route instructions. In Proceedings of National Conference on Artificial Intelligence, AAAI, Austin, UK, 2006.

    Google Scholar 

  29. P. E. Rybski, J. Stolarz, K. Yoon, M. Veloso. Using dialog and human observations to dictate tasks to a learning robot assistant. Intelligent Service Robotics, vol. 1, no. 2, pp. 159–167, 2008. DOI: https://doi.org/10.1007/s11370-008-0016-5.

    Article  Google Scholar 

  30. J. Thomason, S. Q. Zhang, R. J. Mooney, P. Stone. Learning to interpret natural language commands through human-robot dialog. In Proceedings of the 24th International Conference on Artificial Intelligence, AAAI, Buenos Aires, Argentina, 1923–1929, 2015.

    Google Scholar 

  31. L. B. She, S. H. Yang, Y. Cheng, Y. Y. Jia, J. Y. Chai, N. Xi. Back to the blocks world: Learning new actions through situated human-robot dialogue. In Proceedings of the 15th Annual Meeting of the Special Interest Group on Discourse and Dialogue, Philadelphia, USA, 89–97, 2014.

    Chapter  Google Scholar 

  32. E. Grefenstette, P. Blunsom, N. de Freitas, K. M. Hermann. A deep architecture for semantic parsing. In Proceedings of the ACL Workshop on Semantic Parsing, Association for Computational Linguistics, Baltimore, USA, 2014.

    Google Scholar 

  33. N. Papernot, P. McDaniel, S. Jha, M. Fredrikson, Z. B. Celik, A. Swami. The limitations of deep learning in adversarial settings. In Proceedings of IEEE European Symposium on Security and Privacy, Saarbrucken, Germany, 372–387, 2016. DOI: https://doi.org/10.1109/EuroSP.2016.36.

    Google Scholar 

  34. B. Biggio, B. Nelson, P. Laskov. Support vector machines under adversarial label noise. In Proceedings of the 3rd Asian Conference on Machine Learning, Taoyuan, China, 97–112, 2011.

    Google Scholar 

Download references

Acknowledgements

This work was supported by Tsinghua University Initiative Scientific Research Program (No. 20141081140).

Author information

Authors and Affiliations

  1. Tsinghua National Laboratory for Information Science and Technology (TNList), Department of Computer Science and Technology, Tsinghua University, Beijing, 100084, China

    Hao Li, Yu-Ping Wang & Tai-Jiang Mu

Authors
  1. Hao Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Yu-Ping Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Tai-Jiang Mu
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Yu-Ping Wang.

Additional information

Recommended by Associate Editor James Whidborne

Hao Li received the B. Sc. degree in computer science and technology from Xidian University, China in 2012. He is now a Ph.D. degree candidate at Tsinghua University, China under the supervision of professor Shi-Min Hu. His work has been published in journals including Communications in Information and Systems, International Journal of Software Engineering and Knowledge Engineering.

His research interests include program synthesis and system reliability.

Yu-Ping Wang received the Ph. D. degree in computer science and technology from Tsinghua University, China in 2009. He is currently an associate professor of Tsinghua University, China. He has published papers in important journals and conferences, including IEEE Transactions on Visualization and Computer Graphics, IEEE Transactions on Computers, Journal of Systems and Software, USENIX Annual Technical Conference, International Symposium on Code Generation and Optimization, International Symposium on Software Reliability Engineering, IEEE International Conference on Computers, Software and Applications (COMPSAC) and Asia-Pacific Software Engineering Conference. He received the COMPSAC 2014 Best Paper Award.

His research interests include robotic system and system reliability.

Tai-Jiang Mu received the B. Sc. and Ph. D. degrees in computer science and technology from Tsinghua University, China in 2011 and 2016, respectively. He is currently a postdoctoral researcher in Department of Computer Science and Technology, Tsinghua University, China.

His research interests include computer graphics, image/video processing and human-robot interaction.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Li, H., Wang, YP. & Mu, TJ. Synthesizing Robot Programs with Interactive Tutor Mode. Int. J. Autom. Comput. 16, 462–474 (2019). https://doi.org/10.1007/s11633-018-1154-7

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11633-018-1154-7

Keywords

Search

Navigation