Abstract
In human teams, visual and auditory cues are often used to communicate information about the task and/or environment that may not otherwise be directly observable. Analogously, robots that primarily rely on visual sensors cannot directly observe some attributes of objects that may be necessary for reference resolution or task execution. The experiments in this paper address natural language interaction in human-robot teams for tasks where multi-modal (e.g. visual, auditory, haptic, etc) observations are necessary for robust execution. We present a probabilistic model, verified through physical experiments, that allows robots to acquire knowledge about the latent aspects of the workspace through language and physical interaction in an efficient manner. The model’s effectiveness is demonstrated on a mobile and a stationary manipulator in real-world scenarios by following instructions under partial knowledge of object states in the environment.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Arkin, J., Howard, T.M.: Experiments in proactive symbol grounding for efficient physically situated human-robot dialogue. In: Late-breaking Track at the SIGDIAL Special Session on Physically Situated Dialogue (RoboDIAL-18), July 2018
Bishop, C.M.: Probability distributions. In: Jordan, M., Kleinberg, J., Schölkopf, B. (eds.) Pattern Recognition and Machine Learning, Chapter 2. Springer-Verlag, New York (2006)
Chu, V., McMahon, I., Riano, L., McDonald, C.G., He, Q., Perez-Tejada, J.M., Arrigo, M., Darrell, T., Kuchenbecker, K.J.: Robotic learning of haptic adjectives through physical interaction. Robot. Auton. Syst. 63, 279–292 (2015)
Duvallet, F., Walter, M.R., Howard, T.M., Hemachandra, S., Oh, J., Teller, S., Roy, N., Stentz, A.: Inferring maps and behaviors from natural language instructions. In: Experimental Robotics, pp. 373–388. Springer (2016)
Hemachandra, S., Duvallet, F., Howard, T.M., Roy, N., Stentz, A., Walter, M.R.: Learning models for following natural language directions in unknown environments. In: Proceedings of IEEE International Conference on Robotics and Automation (ICRA), Seattle, WA, May 2015
Hogman, V., Bjorkman, M., Kragic, D.: Interactive object classification using sensorimotor contingencies. In: 2013 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 2799–2805. IEEE (2013)
Howard, T.M., Tellex, S., Roy, N.: A natural language planner interface for mobile manipulators. In: Proceedings of IEEE International Conference on Robotics and Automation (ICRA), pp. 6652–6659. IEEE (2014)
Kollar, T., Krishnamurthy, J., Strimel, G.P.: Toward interactive grounded language acqusition. Robot. Sci. Syst. 1, 721–732 (2013)
Kollar, T., Perera, V., Nardi, D., Veloso, M.: Learning environmental knowledge from task-based human-robot dialog. In: Proceedings of IEEE International Conference on Robotics and Automation (ICRA), pp. 4304–4309. IEEE (2013)
Matuszek, C., Bo, L., Zettlemoyer, L., Fox, D.: Learning from unscripted deictic gesture and language for human-robot interactions. In: Proceedings of National Conference on Artificial Intelligence (AAAI) (2014)
Matuszek, C., FitzGerald, N., Zettlemoyer, L., Bo, L., Fox, D.: A joint model of language and perception for grounded attribute learning. In: Proceedings of the: International Conference on Machine Learning, p. 2012. Edinburgh, Scotland (2012)
Oh, J., Howard, T.M., Walter, M.R., Barber, D., Zhu, M., Park, S., Suppe, A., Navarro-Serment, L., Duvallet, F., Boularias, A., Romero, O., Vinkkrov, J., Keegan, T., Dean, R., Lennon, C., Bodt, B., Childers, M., Shi, J., Daniilidis, K., Roy, N., Lebiere, C., Hebert, M., Stentz, A.: Integrated intelligence for human-robot teams. In: Proceedings of the 2016 International Symposium on Experimental Robotics, October 2016
Paul, R., Arkin, J., Aksaray, D., Roy, N., Howard, T.M.: Efficient grounding of abstract spatial concepts for natural language interaction with robot platforms. Int. J. Robot. Res. 37, 1269–1299 (2018)
Paul, R., Barbu, A., Felshin, S., Katz, B., Roy, N.: Temporal grounding graphs for language understanding with accrued visual-linguistic context. In: Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence, pp. 4506–4514 (2017)
Perera, I.E., Allen, J.F.: Sall-e: Situated agent for language learning. In: Proceedings of the 27th AAAI Conference on Artificial Intelligence, pp. 1241–1247 (2013)
Sinapov, J., Schenck, C., Staley, K., Sukhoy, V., Stoytchev, A.: Grounding semantic categories in behavioral interactions: Experiments with 100 objects. Robot. Auton. Syst. 62(5), 632–645 (2014)
Sinapov, J., Stoytchev, A.: From acoustic object recognition to object categorization by a humanoid robot. In: Proceedings of the RSS: Workshop on Mobile Manipulation, p. 2009. Seattle, WA (2009)
Thomason, J., Sinapov, J., Svetlik, M., Stone, P., Mooney, R.J.: Learning multi-modal grounded linguistic semantics by playing “I Spy”. In: IJCAI, pp. 3477–3483 (2016)
Walter, M.R., Hemachandra, S., Homberg, B., Tellex, S., Teller, S.: A framework for learning semantic maps from grounded natural language descriptions. Int. J. Robot. Res. 33(9), 1167–1190 (2014)
Whitney, D., Eldon, M., Oberlin, J., Tellex, S.: Interpreting multimodal referring expressions in real time. In: Proceedings IEEE International Conference on Robotics and Automation (ICRA), pp. 3331–3338. IEEE (2016)
Acknowledgements
Authors gratefully acknowledge funding support in part by the Robotics Consortium of the U.S. Army Research Laboratory under the Collaborative Technology Alliance Program (RCTA) and the Toyota Research Institute (TRI) Award Number LP-C000765-SR. We thank our colleagues in the lab for helpful feedback on this paper.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2020 Springer Nature Switzerland AG
About this paper
Cite this paper
Arkin, J., Paul, R., Park, D., Roy, S., Roy, N., Howard, T.M. (2020). Real-Time Human-Robot Communication for Manipulation Tasks in Partially Observed Environments. In: Xiao, J., Kröger, T., Khatib, O. (eds) Proceedings of the 2018 International Symposium on Experimental Robotics. ISER 2018. Springer Proceedings in Advanced Robotics, vol 11. Springer, Cham. https://doi.org/10.1007/978-3-030-33950-0_39
Download citation
DOI: https://doi.org/10.1007/978-3-030-33950-0_39
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-33949-4
Online ISBN: 978-3-030-33950-0
eBook Packages: Intelligent Technologies and RoboticsIntelligent Technologies and Robotics (R0)