Abstract
This paper presents an integrated system for generating, troubleshooting, and executing correct-by-construction controllers for autonomous robots using natural language input, allowing non-expert users to command robots to perform high-level tasks. This system unites the power of formal methods with the accessibility of natural language, providing controllers for implementable high-level task specifications, easy-to-understand feedback on those that cannot be achieved, and natural language explanation of the reason for the robot’s actions during execution. The natural language system uses domain-general components that can easily be adapted to cover the vocabulary of new applications. Generation of a linear temporal logic specification from the user’s natural language input uses a novel data structure that allows for subsequent mapping of logical propositions back to natural language, enabling natural language feedback about problems with the specification that are only identifiable in the logical form. We demonstrate the robustness of the natural language understanding system through a user study where participants interacted with a simulated robot in a search and rescue scenario. Automated analysis and user feedback on unimplementable specifications is demonstrated using an example involving a robot assistant in a hospital.








Similar content being viewed by others
Explore related subjects
Discover the latest articles, news and stories from top researchers in related subjects.Notes
This arguably unintuitive translation is due to specifications in LTLMoP being restricted to the GR(1) fragment of LTL.
Even though it was the role of the operator, not the robot, to rescue hostages, we label these examples as tagging errors because a command was given to the system and it was not properly understood. The desired response in this situation is to understand the requested action but report that the robot cannot perform it.
References
Berant, J., & Liang, P. (2014). Semantic parsing via paraphrasing. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) (pp. 1415–1425).
Bhatia, A., Kavraki, L. E., & Vardi, M.Y. (2010). Sampling-based motion planning with temporal goals. In IEEE International Conference on Robotics and Automation (ICRA), IEEE (pp. 2689–2696).
Biere, A. (2008). PicoSAT essentials. Journal on Satisfiability Boolean Modeling and Computation (JSAT), 4, 75–97.
Bikel, D. M. (2004). Intricacies of Collins’ parsing model. Computational Linguistics, 30(4), 479–511.
Bobadilla, L., Sanchez, O., Czarnowski, J., Gossman, K., & LaValle, S. (2011). Controlling wild bodies using linear temporal logic. In Robotics: Science and Systems (RSS).
Brooks, D., Lignos, C., Finucane, C., Medvedev, M., Perera, I., Raman, V., Kress-Gazit, H., Marcus, M., & Yanco, H. (2012). Make it so: Continuous, flexible natural language interaction with an autonomous robot. In Proceedings of the Grounding Language for Physical Systems Workshop at the 76th AAAI Conference on Artificial Intelligence.
Chen, D. L., & Mooney, R.J. (2011). Learning to interpret natural language navigation instructions from observations. In Proceedings of the 25th AAAI Conference on Artifical Intelligence (pp. 859–865).
Cizelj, I., & Belta, C. (2013). Negotiating the probabilistic satisfaction of temporal logic motion specifications. In IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) (pp. 4320–4325).
Clarke, E. M., Grumberg, O., & Peled, D. A. (1999). Model checking. Cambridge, MA: MIT Press.
Dzifcak, J., Scheutz, M., Baral, C., & Schermerhorn, P. (2009). What to do and how to do it: Translating natural language directives into temporal and dynamic logic representation for goal management and action execution. In IEEE International Conference on Robotics and Automation (ICRA) (pp. 4163–4168).
Fainekos, G. E. (2011). Revising temporal logic specifications for motion planning. In IEEE International Conference on Robotics and Automation (ICRA) (pp. 40–45).
Finucane, C., Jing, G., & Kress-Gazit, H. (2010). LTLMoP: Experimenting with language, temporal logic and robot control. In IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) (pp. 1988–1993).
Gabbard, R., Marcus, M., & Kulick, S. (2006). Fully parsing the Penn Treebank. In Human Language Technology Conference of the North American Chapter of the Association of Computational Linguistics (NAACL HLT) (pp. 184–191).
Karaman, Frazzoli. (2009). Sampling-based motion planning with deterministic \(\mu \)-calculus specifications. In IEEE Conference on Decision and Control (CDC) (pp. 2222–2229).
Kim, K., Fainekos, G. E., & Sankaranarayanan, S. (2012). On the revision problem of specification automata. In IEEE International Conference on Robotics and Automation (ICRA) (pp. 5171–5176).
Kloetzer, M., & Belta, C. (2008). A fully automated framework for control of linear systems from temporal logic specifications. IEEE Transactions on Automatic Control, 53(1), 287–297.
Kress-Gazit, H., Fainekos, G. E., & Pappas, G. J. (2008). Translating structured english to robot controllers. Advanced Robotics, 22(12), 1343–1359.
Kress-Gazit, H., Fainekos, G. E., & Pappas, G. J. (2009). Temporal-logic-based reactive mission and motion planning. IEEE Transactions on Robotics, 25(6), 1370–1381.
Matuszek, C., Fox, D., & Koscher, K. (2010). Following directions using statistical machine translation. In Human-Robot Interaction (HRI) (pp. 251–258).
Matuszek, C., FitzGerald, N., Zettlemoyer, L., Bo, L., & Fox, D. (2012). A joint model of language and perception for grounded attribute learning. In Proceedings of the 29th International Conference on Machine Learning (ICML) (pp. 1671–1678).
Matuszek, C., Herbst, E., Zettlemoyer, L., & Fox, D. (2013). Learning to parse natural language commands to a robot control system. Experimental Robotics, 88, 403–415.
Piterman, N., Pnueli, A., & Sa’ar, Y. (2006). Synthesis of reactive(1) designs. In Verification, Model Checking, and Abstract Interpretation (VMCAI) (pp. 364–380).
Poon, H., & Domingos, P. (2009). Unsupervised semantic parsing. In Proceedings of the 2009 Conference on Empirical Methods in Natural Language Processing (EMNLP) (pp. 1–10).
Raman, V., & Kress-Gazit, H. (2011). Analyzing unsynthesizable specifications for high-level robot behavior using LTLMoP. In Computer Aided Verification (CAV) (pp. 663–668).
Raman, V., & Kress-Gazit, H. (2013a). Explaining impossible high-level robot behaviors. IEEE Transactions on Robotics, 29, 94–104.
Raman, V., & Kress-Gazit, H. (2013b). Towards minimal explanations of unsynthesizability for high-level robot behaviors. In IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) (pp. 757–762).
Raman, V., & Kress-Gazit, H. (2014). Unsynthesizable cores: Minimal explanations for unsynthesizable high-level robot behaviors. arXiv:1409.1455.
Raman, V., Lignos, C., Finucane, C., Lee, KCT., Marcus, M., & Kress-Gazit, H. (2013). Sorry Dave, I’m afraid I can’t do that: Explaining unachievable robot tasks using natural language. In Robotics: Science and Systems (RSS).
Schuler, K. (2005). Verbnet: A broad-coverage, comprehensive verb lexicon. PhD thesis, University of Pennsylvania.
Tellex, S., Kollar, T., Dickerson, S., Walter, M. R., Banerjee, A. G., Teller, S. J., & Roy, N. (2011). Understanding natural language commands for robotic navigation and mobile manipulation. In Proceedings of the 25th AAAI Conference on Artifical Intelligence (pp. 1507–1514).
Toutanova, K., Klein, D., Manning, C. D., & Singer, Y. (2003). Feature-rich part-of-speech tagging with a cyclic dependency network. In Proceedings of the 2003 Conference of the North American Chapter of the Association for Computational Linguistics on Human Language Technology (NAACL HLT) - Volume 1 (pp. 173–180).
Wongpiromsarn, T., Topcu, U., & Murray, R. M. (2010). Receding horizon control for temporal logic specifications. In Hybrid Systems: Computation and Control (HSCC) (pp. 101–110).
Acknowledgments
We would like to thank Taylor Turpen, Israel Geselowitz, and Kenton Lee for their assistance with software development and data collection. This work was supported in part by: ARO MURI (SUBTLE) W911NF-07-1-0216, NSF CAREER CNS-0953365, DARPA N66001-12-1-4250, and TerraSwarm, one of six centers of STARnet, a Semiconductor Research Corporation program sponsored by MARCO and DARPA.
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Lignos, C., Raman, V., Finucane, C. et al. Provably correct reactive control from natural language. Auton Robot 38, 89–105 (2015). https://doi.org/10.1007/s10514-014-9418-8
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10514-014-9418-8