research-article

ONYX: Assisting Users in Teaching Natural Language Interfaces Through Multi-Modal Interactive Task Learning

Authors:

Alexander MaedcheAuthors Info & Claims

CHI '23: Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems

Article No.: 417, Pages 1 - 16

https://doi.org/10.1145/3544548.3580964

Published: 19 April 2023 Publication History

CHI '23: Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems

ONYX: Assisting Users in Teaching Natural Language Interfaces Through Multi-Modal Interactive Task Learning

Pages 1 - 16

Abstract
Supplementary Material
References

Abstract

Users are increasingly empowered to personalize natural language interfaces (NLIs) by teaching how to handle new natural language (NL) inputs. However, our formative study found that when teaching new NL inputs, users require assistance in clarifying ambiguities that arise and want insight into which parts of the input the NLI understands. In this paper we introduce ONYX, an intelligent agent that interactively learns new NL inputs by combining NL programming and programming-by-demonstration, also known as multi-modal interactive task learning. To address the aforementioned challenges, ONYX provides suggestions on how ONYX could handle new NL inputs based on previously learned concepts or user-defined procedures, and poses follow-up questions to clarify ambiguities in user demonstrations, using visual and textual aids to clarify the connections. Our evaluation shows that users provided with ONYX’s new features achieved significantly higher accuracy in teaching new NL inputs (median: 93.3%) in contrast to those without (median: 73.3%).

Supplementary Material

MP4 File (3544548.3580964-talk-video.mp4)

Pre-recorded Video Presentation

Download
101.08 MB

References

[1]

2019. Create commands to control online services and devices. https://support.google.com/googlenest/answer/7194656

[2]

2022. Shortcuts and Suggestions - Siri - Human Interface Guidelines - Apple Developer. https://developer.apple.com/design/human-interface-guidelines/siri/overview/shortcuts-and-suggestions/

[3]

James Allen, Nathanael Chambers, George Ferguson, Lucian Galescu, Hyuckchul Jung, Mary Swift, and William Taysom. 2007. PLOW: A collaborative task learning agent. In Proceedings of the National Conference on Artificial Intelligence, Vol. 2. AAAI Press, 1514–1519. https://doi.org/10.5555/1619797.1619888

Digital Library

[4]

Zahra Ashktorab, Mohit Jain, Q. Vera Liao, and Justin D. Weisz. 2019. Resilient chatbots: Repair strategy preferences for conversational breakdowns. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems. ACM, New York, NY, USA, 1–12. https://doi.org/10.1145/3290605.3300484

Digital Library

[5]

Amos Azaria, Shashank Srivastava, Jayant Krishnamurthy, Igor Labutov, and Tom M. Mitchell. 2020. An agent for learning new natural language commands. Autonomous Agents and Multi-Agent Systems 34, 1 (4 2020), 6. https://doi.org/10.1007/s10458-019-09425-x

Digital Library

[6]

Tracey Booth and Simone Stumpf. 2013. End-user experiences of visual and textual programming environments for Arduino. In Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), Vol. 7897 LNCS. 25–39. https://doi.org/10.1007/978-3-642-38706-7_4

[7]

M. Bostock, V. Ogievetsky, and J. Heer. 2011. D³ Data-Driven Documents. IEEE Transactions on Visualization and Computer Graphics 17, 12 (12 2011), 2301–2309. https://doi.org/10.1109/TVCG.2011.185

Digital Library

[8]

Julia Cambre, Alex C. Williams, Afsaneh Razi, Ian Bicking, Abraham Wallin, Janice Tsai, Chinmay Kulkarni, and Jofish Kaye. 2021. Firefox Voice: An Open and Extensible Voice Assistant Built Upon the Web. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems. ACM, New York, NY, USA, 1–18. https://doi.org/10.1145/3411764.3445409

Digital Library

[9]

Xin Chen, Jessica Zeitz Self, Leanna House, John Wenskovitch, Maoyuan Sun, Nathan Wycoff, Jane Robertson Evia, Scotland Leman, and Chris North. 2018. Be the Data: Embodied Visual Analytics. IEEE Transactions on Learning Technologies 11, 1 (2018), 81–95. https://doi.org/10.1109/TLT.2017.2757481

[10]

Ensheng Dong, Hongru Du, and Lauren Gardner. 2020. An interactive web-based dashboard to track COVID-19 in real time. The Lancet Infectious Diseases 20, 5 (5 2020), 533–534. https://doi.org/10.1016/S1473-3099(20)30120-1

[11]

Michael H Fischer, Giovanni Campagna, Euirim Choi, and Monica S Lam. 2021. DIY assistant: A multi-modal end-user programmable virtual assistant. In Proceedings of the ACM SIGPLAN Conference on Programming Language Design and Implementation (PLDI). ACM, 312–327. https://doi.org/10.1145/3453483.3454046

Digital Library

[12]

Tong Gao, Mira Dontcheva, Eytan Adar, Zhicheng Liu, and Karrie Karahalios. 2015. Datatone: Managing ambiguity in natural language interfaces for data visualization. In Proceedings of the 28th Annual ACM Symposium on User Interface Software and Technology. ACM Press, New York, New York, USA, 489–500. https://doi.org/10.1145/2807442.2807478

Digital Library

[13]

Jonathan Grudin and Richard Jacques. 2019. Chatbots, Humbots, and the Quest for Artificial General Intelligence. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems, Vol. 11. ACM, New York, NY, USA, 1–11. https://doi.org/10.1145/3290605.3300439

Digital Library

[14]

Piedade João, Dorotea Nuno, Sampaio Ferrentini Fábio, and Pedro Ana. 2019. A cross-analysis of block-based and visual programming apps with computer science student-teachers. Education Sciences 9, 3 (2019). https://doi.org/10.3390/educsci9030181

[15]

Young-Ho Kim, Bongshin Lee, Arjun Srinivasan, and Eun Kyoung Choe. 2021. Data@Hand: Fostering Visual Exploration of Personal Data on Smartphones Leveraging Speech and Touch Interaction. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems. ACM, New York, NY, USA, 1–17. https://doi.org/10.1145/3411764.3445421

Digital Library

[16]

Yea-Seul Kim, Mira Dontcheva, Eytan Adar, and Jessica Hullman. 2019. Vocal Shortcuts for Creative Experts. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems, Vol. 14. ACM, New York, NY, USA, 1–14. https://doi.org/10.1145/3290605.3300562

Digital Library

[17]

Rebecca Krosnick and Steve Oney. 2022. ParamMacros : Creating UI Automation Leveraging End-User Natural Language Parameterization. In 2022 IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC).

[18]

John E Laird, Kevin Gluck, John Anderson, Kenneth D Forbus, Odest Chadwicke Jenkins, Christian Lebiere, Dario Salvucci, Matthias Scheutz, Andrea Thomaz, Greg Trafton, Robert E Wray, Shiwali Mohan, and James R Kirk. 2017. Interactive Task Learning. IEEE Intelligent Systems 32, 4 (2017), 6–21. https://doi.org/10.1109/MIS.2017.3121552

Digital Library

[19]

Gilly Leshed, Eben M Haber, Tara Matthews, and Tessa Lau. 2008. CoScripter. In Proceedings of the 2008 CHI Conference on Human Factors in Computing Systems. ACM Press, New York, New York, USA, 1719. https://doi.org/10.1145/1357054.1357323

Digital Library

[20]

Chi-Hsun Li, Su-Fang Yeh, Tang-Jie Chang, Meng-Hsuan Tsai, Ken Chen, and Yung-Ju Chang. 2020. A Conversation Analysis of Non-Progress and Coping Strategies with a Banking Task-Oriented Chatbot. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems. ACM, New York, NY, USA, 1–12. https://doi.org/10.1145/3313831.3376209

Digital Library

[21]

Toby Jia-Jun Li, Amos Azaria, and Brad A. Myers. 2017. SUGILITE. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems, Vol. 2017-May. ACM, New York, NY, USA, 6038–6049. https://doi.org/10.1145/3025453.3025483

Digital Library

[22]

Toby Jia Jun Li, Jingya Chen, Haijun Xia, Tom M Mitchell, and Brad A Myers. 2020. Multi-modal repairs of conversational breakdowns in task-oriented dialogs. In Proceedings of the 33rd Annual ACM Symposium on User Interface Software and Technology. 1094–1107. https://doi.org/10.1145/3379337.3415820

Digital Library

[23]

Toby Jia-Jun Li, Igor Labutov, Xiaohan Nancy Li, Xiaoyi Zhang, Wenze Shi, Wanling Ding, Tom M. Mitchell, and Brad A. Myers. 2018. APPINITE: A Multi-Modal Interface for Specifying Data Descriptions in Programming by Demonstration Using Natural Language Instructions. In 2018 IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC), Vol. 2018-Octob. IEEE, 105–114. https://doi.org/10.1109/VLHCC.2018.8506506

[24]

Toby Jia-Jun Li, Marissa Radensky, Justin Jia, Kirielle Singarajah, Tom M. Mitchell, and Brad A. Myers. 2019. PUMICE: A Multi-Modal Agent that Learns Concepts and Conditionals from Natural Language and Demonstrations. In Proceedings of the 32nd Annual ACM Symposium on User Interface Software and Technology. ACM, New York, NY, USA, 577–589. https://doi.org/10.1145/3332165.3347899

Digital Library

[25]

Yuyu Luo, Nan Tang, Guoliang Li, Jiawei Tang, Chengliang Chai, and Xuedi Qin. 2022. Natural Language to Visualization by Neural Machine Translation. IEEE Transactions on Visualization and Computer Graphics 28, 1 (1 2022), 217–226. https://doi.org/10.1109/TVCG.2021.3114848

Digital Library

[26]

Microsoft. 2022. Teach Q&A to understand questions and terms in Power BI Q&A - Power BI | Microsoft Docs. https://docs.microsoft.com/en-us/power-bi/natural-language/q-and-a-tooling-teach-q-and-a

[27]

Brad A. Myers, Amy J. Ko, Thomas D. LaToza, and Youngseok Yoon. 2016. Programmers Are Users Too: Human-Centered Methods for Improving Programming Tools. Computer 49, 7 (7 2016), 44–52. https://doi.org/10.1109/MC.2016.200

Digital Library

[28]

Brad A. Myers, Andrew J. Ko, Chris Scaffidi, Stephen Oney, Young Seok Yoon, Kerry Chang, Mary Beth Kery, and Toby Jia Jun Li. 2017. Making end user development more natural. In New Perspectives in End-User Development. 1–22. https://doi.org/10.1007/978-3-319-60291-2_1

[29]

Brad A Myers, Richard G. McDaniel, and David S Kosbie. 1993. Marquise. In Proceedings of the 1993 CHI Conference on Human Factors in Computing Systems. ACM Press, New York, New York, USA, 293–300. https://doi.org/10.1145/169059.169225

Digital Library

[30]

Arpit Narechania, Adam Fourney, Bongshin Lee, and Gonzalo Ramos. 2021. DIY: Assessing the Correctness of Natural Language to SQL Systems. In 26th International Conference on Intelligent User Interfaces. ACM, New York, NY, USA, 597–607. https://doi.org/10.1145/3397481.3450667

Digital Library

[31]

Arpit Narechania, Arjun Srinivasan, and John Stasko. 2021. NL4DV: A Toolkit for Generating Analytic Specifications for Data Visualization from Natural Language Queries. IEEE Transactions on Visualization and Computer Graphics 27, 2 (2 2021), 369–379. https://doi.org/10.1109/TVCG.2020.3030378

[32]

Şaziye Betül Özateş, Arzucan Özgür, and Gomir R. Draradev. 2016. Sentence similarity based on dependency tree kernels for multi-document summarization. In Proceedings of the 10th International Conference on Language Resources and Evaluation, LREC 2016. 2833–2838. http://duc.nist.gov/duc2003/tasks.html

[33]

Lihang Pan, Chun Yu, JiaHui Li, Tian Huang, Xiaojun Bi, and Yuanchun Shi. 2022. Automatically Generating and Improving Voice Command Interface from Operation Sequences on Smartphones. In Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems. ACM, New York, NY, USA, 1–21. https://doi.org/10.1145/3491102.3517459

Digital Library

[34]

Marcel Ruoff and Ulrich Gnewuch. 2021. Designing Multimodal BI&A Systems for Co-Located Team Interactions. In 29th European Conference on Information Systems. Virtual. https://aisel.aisnet.org/ecis2021_rp/113

[35]

Marcel Ruoff, Brad A. Myers, and Alexander Maedche. 2022. ONYX - User Interfaces for Assisting in Interactive Task Learning for Natural Language Interfaces of Data Visualization Tools. In Proceedings of the 2022 CHI Conference Extended Abstracts on Human Factors in Computing Systems, Vol. 1. Association for Computing Machinery, 1–10. https://doi.org/10.1145/3491101.3519793

Digital Library

[36]

Vidya Setlur, Sarah E. Battersby, Melanie Tory, Rich Gossweiler, and Angel X. Chang. 2016. Eviza: A natural language interface for visual analysis. In Proceedings of the 29th Annual Symposium on User Interface Software and Technology(UIST ’16). ACM, New York, NY, USA, 365–377. https://doi.org/10.1145/2984511.2984588

Digital Library

[37]

Arjun Srinivasan, Bongshin Lee, Nathalie Henry Riche, Steven M. Drucker, and Ken Hinckley. 2020. InChorus: Designing Consistent Multimodal Interactions for Data Visualization on Tablet Devices. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems. ACM, New York, NY, USA, 1–13. https://doi.org/10.1145/3313831.3376782

Digital Library

[38]

Arjun Srinivasan, Nikhila Nyapathy, Bongshin Lee, Steven M. Drucker, and John Stasko. 2021. Collecting and Characterizing Natural Language Utterances for Specifying Data Visualizations. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems. ACM, New York, NY, USA, 1–10. https://doi.org/10.1145/3411764.3445400

Digital Library

[39]

Gavin Suddrey, Ben Talbot, and Frederic Maire. 2022. Learning and Executing Re-Usable Behaviour Trees From Natural Language Instruction. IEEE Robotics and Automation Letters 7, 4 (10 2022), 10643–10650. https://doi.org/10.1109/LRA.2022.3194681

[40]

Tableau. 2019. Optimize Data for Ask Data - Tableau. https://help.tableau.com/current/pro/desktop/en-us/ask data optimize.htm

[41]

Melanie Tory and Vidya Setlur. 2019. Do What I Mean, Not What I Say! Design Considerations for Supporting Intent and Context in Analytical Conversation. In 2019 IEEE Conference on Visual Analytics Science and Technology (VAST). IEEE, 93–103. https://doi.org/10.1109/VAST47406.2019.8986918

[42]

Priyan Vaithilingam and Philip J. Guo. 2019. Bespoke: Interactively synthesizing custom GUIs from command-line applications by demonstration. In Proceedings of the 32nd Annual ACM Symposium on User Interface Software and Technology. 563–576. https://doi.org/10.1145/3332165.3347944

Digital Library

[43]

Sida I. Wang, Samuel Ginn, Percy Liang, and Christopher D. Manning. 2017. Naturalizing a Programming Language via Interactive Learning. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, Vol. 1. Association for Computational Linguistics, Stroudsburg, PA, USA, 929–938. https://doi.org/10.18653/v1/P17-1086

[44]

Jonathan Zong, Dhiraj Barnwal, Rupayan Neogy, and Arvind Satyanarayan. 2021. Lyra 2: Designing Interactive Visualizations by Demonstration. IEEE Transactions on Visualization and Computer Graphics 27, 2 (2 2021), 304–314. https://doi.org/10.1109/TVCG.2020.3030367

Cited By

Zhou JMacLellan C(2024)Improving Interface Design in Interactive Task Learning for Hierarchical Tasks based on a Qualitative StudyAdjunct Proceedings of the 37th Annual ACM Symposium on User Interface Software and Technology10.1145/3672539.3686326(1-3)Online publication date: 13-Oct-2024
Li JLi JSu Y(2024)A Map of Exploring Human Interaction Patterns with LLM: Insights into Collaboration and CreativityArtificial Intelligence in HCI10.1007/978-3-031-60615-1_5(60-85)Online publication date: 29-Jun-2024
Myers B(2024)Pick, Click, Flick!undefinedOnline publication date: 14-Mar-2024
Show More Cited By

Index Terms

ONYX: Assisting Users in Teaching Natural Language Interfaces Through Multi-Modal Interactive Task Learning
1. Human-centered computing
  1. Human computer interaction (HCI)
    1. Interaction paradigms
      1. Natural language interfaces
    2. Interactive systems and tools
      1. User interface programming

Recommendations

MALACHITE - Enabling Users to Teach GUI-Aware Natural Language Interfaces
Users can adapt contemporary natural language interfaces (NLIs) by teaching the NLIs how to handle new natural language (NL) inputs. One promising approach is interactive task learning (ITL), which enables users to teach new NL inputs for multi-modal ...
ONYX - User Interfaces for Assisting in Interactive Task Learning for Natural Language Interfaces of Data Visualization Tools
CHI EA '22: Extended Abstracts of the 2022 CHI Conference on Human Factors in Computing Systems

While natural language interfaces (NLIs) are increasingly utilized to simplify the interaction with data visualization tools, improving and adapting the NLIs to the individual needs of users still requires the support of developers. ONYX introduces an ...
The economics of natural language interfaces: natural language processing technology as a scarce resource

This paper discusses appropriate application areas for natural language interfaces (NLIs) to databases. This requires comparing NLIs with competing approaches, including other user-friendly interfaces, and training of users with less user-friendly ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

CHI '23: Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems

April 2023

14911 pages

ISBN:9781450394215

DOI:10.1145/3544548

Editors:
Albrecht Schmidt
LMU Munich, Germany60028717
,
Kaisa Väänänen
Tampere University, Finland60011170
,
Tesh Goyal
Google Research, USA60006191
,
Per Ola Kristensson
University of Cambridge, UK60031101
,
Anicia Peters
University of Namibia, Namibia60072704
,
Stefanie Mueller
Massachusetts Institute of Technology, USA60022195
,
Julie R. Williamson
University of Glasgow, UK60001490
,
Max L. Wilson
University of Nottingham, UK60015138

Copyright © 2023 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

SIGCHI: ACM Special Interest Group on Computer-Human Interaction

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 19 April 2023

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed limited

Conference

CHI '23

Sponsor:

SIGCHI

CHI '23: CHI Conference on Human Factors in Computing Systems

April 23 - 28, 2023

Hamburg, Germany

Acceptance Rates

Overall Acceptance Rate 6,199 of 26,314 submissions, 24%

Upcoming Conference

CHI 2025

Sponsor:
sigchi

ACM CHI Conference on Human Factors in Computing Systems

April 26 - May 1, 2025

Yokohama , Japan

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

4
Total Citations
View Citations
742
Total Downloads

Downloads (Last 12 months)336
Downloads (Last 6 weeks)75

Reflects downloads up to 05 Mar 2025

Other Metrics

View Author Metrics

Citations

Cited By

Zhou JMacLellan C(2024)Improving Interface Design in Interactive Task Learning for Hierarchical Tasks based on a Qualitative StudyAdjunct Proceedings of the 37th Annual ACM Symposium on User Interface Software and Technology10.1145/3672539.3686326(1-3)Online publication date: 13-Oct-2024
Li JLi JSu Y(2024)A Map of Exploring Human Interaction Patterns with LLM: Insights into Collaboration and CreativityArtificial Intelligence in HCI10.1007/978-3-031-60615-1_5(60-85)Online publication date: 29-Jun-2024
Myers B(2024)Pick, Click, Flick!undefinedOnline publication date: 14-Mar-2024
Shen LZhang YZhang HWang Y(2023)Data Player: Automatic Generation of Data Videos with Narration-Animation InterplayIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2023.332719730:1(109-119)Online publication date: 3-Nov-2023

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Full Text

View this article in Full Text.

HTML Format

View this article in HTML Format.

Figures

Tables

Media

View full text|Download PDF

View Table of Conten