Abstract
Effective situated interaction hinges on the well-coordinated operation of a set of competencies, including computer vision, speech recognition, and natural language, as well as higher-level inferences about turn taking and engagement. Systems often rely on a set of hand-coded and machine-learned components organized into several sensing and decision-making pipelines. Given their complexity and inter-dependencies, developing and debugging such systems can be challenging. “In-the-wild” deployments outside of controlled lab conditions bring further challenges due to unanticipated phenomena, including unexpected interactions such as playful engagements. We present a methodology for assessing performance, identifying problems, and diagnosing the root causes and influences of different types of failures on the overall performance of a situated interaction system functioning in the wild. We apply the methodology to a dataset of interactions collected with a robot deployed in a public space inside an office building. The analyses identify and characterize multiple types of failures, their causes, and their relationship to overall performance. We employ models that predict overall interaction quality from various combinations of failures. Finally, we discuss lessons learned with such a diagnostic methodology for improving situated systems deployed in the wild.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Sculley, D., Holt, G., Golovin, D., Davydov, E., Phillips, T., Ebner, D., Chaudhary, V., Young, M., Crespo, J., Dennison, D.: Hidden technical debt in machine learning systems. In: Advances in Neural Information Processing Systems, pp. 2503–2511 (2015)
Nushi, B., Kamar, E., Kossmann, D., Horvitz, E.: On human intellect and machine failures: troubleshooting integrative machine learning systems. In: Proceedings of AAAI 2017 (2017)
Mirnig, N., Weiss, A., Skantze, G., Al Moubayed, S., Gustafson, J., Beskow, J., Granström, B., Tscheligi, M.: Face-to-face with a robot: what do we actually talk about? Int. J. Hum. Robot. 10(01), 23 (2013)
Skantze, G., Al Moubayed, S., Gustafson, J., Beskow, J., Granström, B.: Furhat at robotville: a robot head harvesting the thoughts of the public through multi-party dialogue. In: Proceedings of the International Conference on Intelligent Virtual Agents (2012)
Parikh, D., Zitnick, C.L.: Human-debugging of machines. In: The Second Workshop on Computational Social Science and the Wisdom of Crowds (NIPS 2011) (2011)
Georgiladakis, S., Athanasopoulou, G., Meena, R., Lopes, J., Chorianopoulou, A., Palogiannidi, E., Iosif, E., Skantze, G., Potamianos, A.: Root cause analysis of miscommunication hotspots in spoken dialogue systems. In: Interspeech 2016, pp. 1156–1160 (2016)
Walker, M.A., Litman, D.J., Kamm, C.A., Abella. A.: PARADISE: A framework for evaluating spoken dialogue agents. In: Proceedings of the Eighth Conference on European Chapter of the Association for Computational Linguistics, pp. 271–280 (1987)
Bohus, D., Saw, C.W., Horvitz, E.: Directions robot: In-the-wild experiences and lessons learned. In: Proceedings of AAMAS 2014, Paris, France (2014)
Bohus, D., Horvitz, E.: Managing human-robot engagement with forecasts and … um … hesitations. In: Proceedings of ICMI 2014, Istanbul, Turkey (2014)
Andrist, S., Bohus, D., Yu, Z., Horvitz, E.: Are you messing with me?: querying about the sincerity of interactions in the open world. In: Proceedings of HRI 2016, Piscataway, NJ, USA, pp. 409–410. IEEE Press (2016)
Schmitt, A., Schatz, B., Minker, W.: Modeling and predicting quality in spoken human-computer interaction. In: Proceedings of the SIGDIAL 2011 Conference, pp. 173–184 (2011)
Walker, M.A., Litman, D.J., Kamm, C.A., Abella, A.: PARADISE: A framework for evaluating spoken dialogue agents. In: Proceedings of the Eighth Conference on European Chapter of the Association for Computational Linguistics (1997)
Meena, R., Lopes, J., Skantze, G., Gustafson, J.: Automatic detection of miscommunication in spoken dialogue systems. In: Proceeding of SIGDIAL (2015)
Walker, M., Langkilde, I., Wright, J., Gorin, A., Litman, D.: Learning to predict problematic situations in a spoken dialogue system: experiments with how may I help you?. In: Proceedings of the 1st North American Chapter of the Association for Computational Linguistics Conference (2000)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2017 Springer International Publishing AG
About this paper
Cite this paper
Andrist, S., Bohus, D., Kamar, E., Horvitz, E. (2017). What Went Wrong and Why? Diagnosing Situated Interaction Failures in the Wild. In: Kheddar, A., et al. Social Robotics. ICSR 2017. Lecture Notes in Computer Science(), vol 10652. Springer, Cham. https://doi.org/10.1007/978-3-319-70022-9_29
Download citation
DOI: https://doi.org/10.1007/978-3-319-70022-9_29
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-70021-2
Online ISBN: 978-3-319-70022-9
eBook Packages: Computer ScienceComputer Science (R0)