poster

Public Access

Study on Text-based and Voice-based Dialogue Interfaces for Human-Computer Interactions in a Blocks World

Authors:

Sujeong Kim,

David Salter,

Luke DeLuccia,

Amir TamrakarAuthors Info & Claims

HAI '20: Proceedings of the 8th International Conference on Human-Agent Interaction

Pages 227 - 229

https://doi.org/10.1145/3406499.3418754

Published: 10 November 2020 Publication History

PDF eReader

Abstract

We conducted a small scale user study to understand user experiences with two different forms of dialogue interfaces - text-based and voice-based - to interact with a virtual agent in a Blocks World environment while perform tower building tasks. The participants also had the option of using deictic gestures in addition to the speech modality. We identify common types of errors/issues that led to communication failures and share our observations about how users reacted to these issues. We also present survey data that reflects users' evaluations of the dialogue interfaces and their interactions with the virtual agent.

Supplementary Material

MP4 File (3406499.3418754.mp4)

We conducted a small scale user study to understand user experiences with two different forms of dialogue interfaces - text-based and voice-based - to interact with a virtual agent in a Blocks World environment while perform tower building tasks. The participants also had the option of using deictic gestures in addition to the speech modality. We identify common types of errors/issues that led to communication failures and share our observations about how users reacted to these issues. We also present survey data that reflects users? evaluations of the dialogue interfaces and their interactions with the virtual agent.

Download
79.85 MB

References

[1]

Dan Bohus and Alexander I. Rudnicky. 2005. Sorry, I didn't catch that! -- An investigation of non-understanding errors and recovery strategies. In In Proceedings of the 6th SIGdial Workshop on Discourse and Dialogue. 128--143.

Google Scholar

[2]

Rolf Carlson, Jens Edlund, Mattias Heldner, Anna Hjalmarsson, David House, and Gabriel Skantze. 2006. Towards human-like behaviour in spoken dialog systems. Proceedings of Swedish Language Technology Conference (SLTC 2006) (2006).

Google Scholar

[3]

SRI International. 2019. Play with SMILEE. https://sites.google.com/view/playwithsmilee/home

Google Scholar

[4]

Sujeong Kim, David Salter, Luke DeLuccia, Kilho Son, Mohamed R. Amer, and Amir Tamrakar. 2018. SMILEE: Symmetric Multi-modal Interactions with Language-gesture Enabled (AI) Embodiment. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Demonstrations. Association for Computational Linguistics, New Orleans, Louisiana, 86--90. https://doi.org/10.18653/v1/N18-5018

Crossref

Google Scholar

[5]

Robyn Kozierok, Lynette Hirschman, John Aberdeen, Cheryl Clar, Christopher Garay, Bradley Goodman, Tonia Korves, and Matthew Peterson. 2018. DARPA Communicating With Computers: Program Goals and Hallmarks. (03 2018).

Google Scholar

[6]

Nikhil Krishnaswamy, Pradyumna Narayana, Isaac Wang, Kyeongmin Rim, Rahul Bangar, Dhruva Patil, Gururaj Mulay, J. Ross Beveridge, Jaime Ruiz, Bruce A. Draper, and James Pustejovsky. 2018. Learning Interpretable Spatial Operations in a Rich 3D Blocks World. Association for the Advancement of Artificial Intelligence (AAAI).

Google Scholar

[7]

André Natal, Glen Shires, Marcos Cáceres, and Philip J"agenstedt. 2020. Web Speech API Draft Community Group Report. https://wicg.github.io/speech-api/

Google Scholar

[8]

Gabriel Skantze. 2003. Exploring human error handling strategies: Implications for spoken dialogue systems. In ISCA Tutorial and Research Workshop on Error Handling in Spoken Dialogue Systems.

Google Scholar

[9]

Rhea Sukthanker, Soujanya Poria, Erik Cambria, and Ramkumar Thirunavukarasu. 2020. Anaphora and coreference resolution: A review. Information Fusion, Vol. 59 (2020), 139--162. https://doi.org/10.1016/j.inffus.2020.01.010

Crossref

Google Scholar

Cited By

View all

Alghamdi EHalvey MNicol E(2024)System and User Strategies to Repair Conversational Breakdowns of Spoken Dialogue Systems: A Scoping ReviewProceedings of the 6th ACM Conference on Conversational User Interfaces10.1145/3640794.3665558(1-13)Online publication date: 8-Jul-2024
https://dl.acm.org/doi/10.1145/3640794.3665558
Tang HZhang CLi Q(2024)Assessing Creativity and User Experience in Immersive Virtual Reality with Cultural Heritage LearningInternational Journal of Human–Computer Interaction10.1080/10447318.2024.2405784(1-17)Online publication date: 30-Sep-2024
https://doi.org/10.1080/10447318.2024.2405784
Chuang HCheng D(2022)Conversational AI over Military Scenarios Using Intent Detection and Response GenerationApplied Sciences10.3390/app1205249412:5(2494)Online publication date: 27-Feb-2022
https://doi.org/10.3390/app12052494
Show More Cited By

Index Terms

Study on Text-based and Voice-based Dialogue Interfaces for Human-Computer Interactions in a Blocks World
1. Computing methodologies
  1. Artificial intelligence
    1. Natural language processing
      1. Discourse, dialogue and pragmatics
2. Human-centered computing
  1. Human computer interaction (HCI)
    1. Empirical studies in HCI
    2. Interaction paradigms
      1. Natural language interfaces

Recommendations

Route instructions in map-based human-human and human-computer dialogue: A comparative analysis

When conveying information about spatial situations and goals, speakers adapt flexibly to their addressee in order to reach the communicative goal efficiently and effortlessly. Our aim is to equip a dialogue system with the abilities required for such a ...
System-initiated digressive proposals in automated human-computer telephone dialogues: the use of contrasting politeness strategies

System-initiated digressive proposals may be used to introduce new and unexpected information into automated telephone services. These digressions may be viewed as particularly pronounced forms of unsolicited interruptions as they contain information ...
Registering Historical Context for Question Answering in a Blocks World Dialogue System
Text, Speech, and Dialogue
Abstract
Task-oriented dialogue-based spatial reasoning systems need to maintain history of the world/discourse states in order to convey that the dialogue agent is mentally present and engaged with the task, as well as to be able to refer to earlier ...

Comments

Information & Contributors

Information

Published In

HAI '20: Proceedings of the 8th International Conference on Human-Agent Interaction

November 2020

304 pages

ISBN:9781450380546

DOI:10.1145/3406499

General Chairs:
Mohammad Obaid
Chalmers University of Technology, Sweden
,
Omar Mubin
Western Sydney University, Australia
,
Yukie Nagai
The University of Tokyo, Japan
,
Program Chairs:
Hirotaka Osawa
University of Tsukuba, Japan
,
Yomna Abdelrahman
Bundeswehr University Munich, Germany
,
Morten Fjeld
Chalmers University of Technology, Sweden

Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for third-party components of this work must be honored. For all other uses, contact the Owner/Author.

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 10 November 2020

Check for updates

Author Tags

Qualifiers

Poster

Funding Sources

DARPA

Conference

HAI '20

Sponsor:

SIGCHI

HAI '20: 8th International Conference on Human-Agent Interaction

November 10 - 13, 2020

Virtual Event, USA

Acceptance Rates

Overall Acceptance Rate 121 of 404 submissions, 30%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

4
Total Citations
View Citations
232
Total Downloads

Downloads (Last 12 months)70
Downloads (Last 6 weeks)6

Reflects downloads up to 05 Mar 2025

Other Metrics

View Author Metrics

Citations

Cited By

View all

Alghamdi EHalvey MNicol E(2024)System and User Strategies to Repair Conversational Breakdowns of Spoken Dialogue Systems: A Scoping ReviewProceedings of the 6th ACM Conference on Conversational User Interfaces10.1145/3640794.3665558(1-13)Online publication date: 8-Jul-2024
https://dl.acm.org/doi/10.1145/3640794.3665558
Tang HZhang CLi Q(2024)Assessing Creativity and User Experience in Immersive Virtual Reality with Cultural Heritage LearningInternational Journal of Human–Computer Interaction10.1080/10447318.2024.2405784(1-17)Online publication date: 30-Sep-2024
https://doi.org/10.1080/10447318.2024.2405784
Chuang HCheng D(2022)Conversational AI over Military Scenarios Using Intent Detection and Response GenerationApplied Sciences10.3390/app1205249412:5(2494)Online publication date: 27-Feb-2022
https://doi.org/10.3390/app12052494
Kim SGarlapati ALubin JTamrakar ADivakaran A(2021)Towards Understanding Confusion and Affective States Under Communication Failures in Voice-Based Human-Machine Interaction2021 9th International Conference on Affective Computing and Intelligent Interaction Workshops and Demos (ACIIW)10.1109/ACIIW52867.2021.9666238(1-5)Online publication date: 28-Sep-2021
https://doi.org/10.1109/ACIIW52867.2021.9666238

View Options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Cited By

Index Terms

Recommendations

Route instructions in map-based human-human and human-computer dialogue: A comparative analysis

System-initiated digressive proposals in automated human-computer telephone dialogues: the use of contrasting politeness strategies

Registering Historical Context for Question Answering in a Blocks World Dialogue System