research-article

Dialogue Act Classification for Virtual Agents for Software Engineers during Debugging

Authors:

Zachary Eberhart,

Collin McMillanAuthors Info & Claims

ICSEW'20: Proceedings of the IEEE/ACM 42nd International Conference on Software Engineering Workshops

Pages 462 - 469

https://doi.org/10.1145/3387940.3391487

Published: 25 September 2020 Publication History

Abstract

A "dialogue act" is a written or spoken action during a conversation. Dialogue acts are usually only a few words long, and are often categorized by researchers into a relatively small set of dialogue act types, such as eliciting information, expressing an opinion, or making a greeting. Research interest into automatic classification of dialogue acts has grown recently due to the proliferation of Virtual Agents (VA) e.g. Siri, Cortana, Alexa. But unfortunately, the gains made into VA development in one domain are generally not applicable to other domains, since the composition of dialogue acts differs in different conversations. In this paper, we target the problem of dialogue act classification for a VA for software engineers repairing bugs. A problem in the SE domain is that very little sample data exists - the only public dataset is a recently-released Wizard of Oz study with 30 conversations. Therefore, we present a transfer-learning technique to learn on a much larger dataset for general business conversations, and apply the knowledge to the SE dataset. In an experiment, we observe between 8% and 20% improvement over two key baselines.

References

[1]

James F Allen, Lenhart K Schubert, George Ferguson, Peter Heeman, Chung Hee Hwang, Tsuneaki Kato, Marc Light, Nathaniel Martin, Bradford Miller, Massimo Poesio, et al. 1995. The TRAINS project: A case study in building a conversational planning agent. Journal of Experimental & Theoretical Artificial Intelligence 7, 1 (1995), 7--48.

[2]

Toine Andernach. [n.d.]. A Machine Learning Approach to the Classification of Dialogue Utterances. In NeMLaP '96.

[3]

Anne H Anderson, Miles Bader, Ellen Gurman Bard, Elizabeth Boyle, Gwyneth Doherty, Simon Garrod, Stephen Isard, Jacqueline Kowtko, Jan McAllister, Jim Miller, et al. 1991. The HCRC map task corpus. Language and speech 34, 4 (1991), 351--366.

[4]

Jeremy Ang, Yang Liu, and Elizabeth Shriberg. [n.d.]. Automatic dialog act segmentation and classification in multiparty meetings. In ICASSP'05.

[5]

Kent Bach and Robert Harnish. 1979. Linguistic communication and speech acts. (1979).

[6]

Srinivas Bangalore, Giuseppe Di Fabbrizio, and Amanda Stent. 2008. Learning the structure of task-driven human-human dialogs. IEEE Transactions on Audio, Speech, and Language Processing 16, 7 (2008), 1249--1259.

Digital Library

[7]

Andrew Begel and Beth Simon. 2008. Struggles of new college graduates in their first software development job. In ACM SIGCSE Bulletin, Vol. 40. ACM, 226--230.

Digital Library

[8]

Phil Blunsom, Nal Kalchbrenner, and Nal Kalchbrenner. [n.d.]. Recurrent convolutional neural networks for discourse compositionality. In CVSC'13.

[9]

Kristy Elizabeth Boyer, Eun Young Ha, Robert Phillips, Michael D Wallis, Mladen A Vouk, and James C Lester. [n.d.]. Dialogue act modeling in a complex task-oriented domain. In SIGDIAL'10.

[10]

Nicholas Bradley, Thomas Fritz, and Reid Holmes. [n.d.]. Context-Aware Conversational Developer Assistants. In ICSE'18.

[11]

Susanne Burger, Karl Weilhammer, Florian Schiel, and Hans G Tillmann. 2000. Verbmobil data collection and annotation. In Verbmobil: Foundations of speech-to-speech translation. Springer, 537--549.

[12]

Hongshen Chen, Xiaorui Liu, Dawei Yin, and Jiliang Tang. 2017. A survey on dialogue systems: Recent advances and new frontiers. ACM SIGKDD Explorations Newsletter 19, 2 (2017), 25--35.

Digital Library

[13]

Zheqian Chen, Rongqin Yang, Zhou Zhao, Deng Cai, and Xiaofei He. [n.d.]. Dialogue Act Recognition via CRF-Attentive Structured Network. In SIGIR'18. ACM, 225--234.

[14]

François Chollet et al. 2015. Keras. https://keras.io.

[15]

Javier Escobar-Avila, Esteban Parra, and Sonia Haiduc. [n.d.]. Text Retrieval-based Tagging of Software Engineering Video Tutorials. In ICSE-C '17.

[16]

Rashmi Gangadharaiah, Balakrishnan Narayanaswamy, and Charles Elkan. [n.d.]. What we need to learn if we want to do and not just talk. In NAACL-HLT'18.

[17]

Jeroen Geertzen, Volha Petukhova, and Harry Bunt. [n.d.]. A multidimensional approach to utterance segmentation and dialogue act classification. In SIGDIAL'07 Workshop on Discourse and Dialogue.

[18]

John J Godfrey, Edward C Holliman, and Jane McDaniel. [n.d.]. SWITCHBOARD: Telephone speech corpus for research and development. In ICASSP'92.

[19]

Sergio Grau, Emilio Sanchis, Maria Jose Castro, and David Vilar. [n.d.]. Dialogue act classification using a Bayesian approach. In SPECOM'04.

[20]

Yangfeng Ji, Gholamreza Haffari, and Jacob Eisenstein. [n.d.]. A latent variable recurrent neural network for discourse relation language models. NAACL-HLT'16 ([n.d.]).

[21]

Yiping Kang, Yunqi Zhang, Jonathan K Kummerfeld, Lingjia Tang, and Jason Mars. [n.d.]. Data Collection for Dialogue System: A Startup Perspective. In NAACL-HLT'18.

[22]

Hamed Khanpour, Nishitha Guntakandla, and Rodney Nielsen. [n.d.]. Dialogue act classification in domain-independent conversations using a deep recurrent neural network. In COLING'16.

[23]

Andrew J. Ko and Brad A. Myers. 2010. Extracting and Answering Why and Why Not Questions About Java Program Output. ACM Trans. Softw. Eng. Methodol. 20, 2, Article 4 (2010), 36 pages. https://doi.org/10.1145/1824760.1824761

Digital Library

[24]

Harshit Kumar, Arvind Agarwal, Riddhiman Dasgupta, Sachindra Joshi, and Arun Kumar. [n.d.]. Dialogue Act Sequence Labeling using Hierarchical encoder with CRF. ([n.d.]).

[25]

Ji Young Lee and Franck Dernoncourt. [n.d.]. Sequential short-text classification with recurrent and convolutional neural networks. NAACL-HLT'16 ([n. d.]).

[26]

Yang Liu, Kun Han, Zhao Tan, and Yun Lei. [n.d.]. Using Context Information for Dialog Act Classification in DNN Framework. In EMNLP'17.

[27]

Iain McCowan, Jean Carletta, W Kraaij, S Ashby, S Bourban, M Flynn, M Guillemot, T Hain, J Kadlec, V Karaiskos, et al. 2005. The AMI meeting corpus. In Proceedings of the 5th International Conference on Methods and Techniques in Behavioral Research, Vol. 88.

[28]

Dmitrijs Milajevs and Matthew Purver. [n.d.]. Investigating the contribution of distributional semantic information for dialogue act classification. In CVSC'14.

[29]

F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. 2011. Scikit-learn: Machine Learning in Python. Journal of Machine Learning Research 12 (2011), 2825--2830.

Digital Library

[30]

Piotr Pruski, Sugandha Lohar, William Goss, Alexander Rasin, and Jane Cleland-Huang. 2015. TiQi: answering unstructured natural language trace queries. Requirements Engineering 20, 3 (01 Sep 2015), 215--232. https://doi.org/10.1007/s00766-015-0224-4

[31]

Norbert Reithinger and Martin Klesen. [n.d.]. Dialogue act classification using language models. In EUROSPEECH'97.

[32]

Verena Rieser and Oliver Lemon. 2011. Reinforcement learning for adaptive dialogue systems: a data-driven methodology for dialogue management and natural language generation. Springer Science & Business Media.

[33]

Martin P Robillard, Andrian Marcus, Christoph Treude, Gabriele Bavota, Oscar Chaparro, Neil Ernst, Marco Aurélio Gerosa, Michael Godfrey, Michele Lanza, Mario Linares-Vásquez, et al. [n.d.]. On-Demand Developer Documentation. In ICSME'17.

[34]

Paige Rodeghero, Siyuan Jiang, Ameer Armaly, and Collin McMillan. [n.d.]. Detecting user story information in developer-client conversations to generate extractive summaries. In ICSE'17.

[35]

Tobias Roehm, Rebecca Tiarks, Rainer Koschke, and Walid Maalej. [n.d.]. How do professional developers comprehend software?. In ICSE'12.

[36]

Ruhi Sarikaya, Paul A Crook, Alex Marin, Minwoo Jeong, Jean-Philippe Robichaud, Asli Celikyilmaz, Young-Bum Kim, Alexandre Rochette, Omar Zia Khan, Xiaohu Liu, et al. [n.d.]. An overview of end-to-end language understanding and dialog management for personal digital assistants. In SLT'16.

[37]

John Searle. 1965. What is a speech act? Cambridge University Press.

[38]

Riccardo Serafin, Barbara Di Eugenio, and Michael Glass. [n.d.]. Latent Semantic Analysis for dialogue act classification. In NAACL-HLT'03.

[39]

Iulian Vlad Serban, Ryan Lowe, Peter Henderson, Laurent Charlin, and Joelle Pineau. 2018. A Survey of Available Corpora For Building Data-Driven Dialogue Systems. Dialogue & Discourse 9, 1 (2018), 1--49.

[40]

Andreas Stolcke, Klaus Ries, Noah Coccaro, Elizabeth Shriberg, Rebecca Bates, Daniel Jurafsky, Paul Taylor, Rachel Martin, Carol Van Ess-Dykema, and Marie Meteer. 2000. Dialogue act modeling for automatic tagging and recognition of conversational speech. Computational linguistics 26, 3 (2000), 339--373.

[41]

Dinoj Surendran and Gina-Anne Levow. [n.d.]. Dialog act tagging with support vector machines and hidden Markov models. In ICSLP'06.

[42]

Maryam Tavafi, Yashar Mehdad, Shafiq Joty, Giuseppe Carenini, and Raymond Ng. [n.d.]. Dialogue act recognition in synchronous and asynchronous conversations. In SIGDIAL'13.

[43]

Enn Tyugu. 1988. Knowledge-Based Programming (Turing Institute Press Knowledge Engineering Tutorial Series). Addison-Wesley Longman Publishing Co., Inc.

[44]

Andrew Wood, Paige Rodeghero, Ameer Armaly, and Collin McMillan. [n.d.]. Detecting Speech Act Types in Developer Question/Answer Conversations During Bug Repair. FSE'18 ([n. d.]).

[45]

Matthias Zimmermann. [n.d.]. Joint segmentation and classification of dialog acts using conditional random fields. In ISCA'09.

Cited By

Ribeiro TSiqueira SDe Bayser M(2024)Identifying intentions in conversational tools: a systematic mappingProceedings of the 20th Brazilian Symposium on Information Systems10.1145/3658271.3658286(1-10)Online publication date: 20-May-2024
https://dl.acm.org/doi/10.1145/3658271.3658286
Hart JAuBuchon JMason SKuttal S(2024)Navigating NLU Challenges in Pair Programming Agents: A Study on Data Size, Gender, Language, and Domain EffectsArtificial Intelligence in HCI10.1007/978-3-031-60606-9_20(356-375)Online publication date: 1-Jun-2024
https://doi.org/10.1007/978-3-031-60606-9_20
Santhanam SHecking TSchreiber AWagner S(2022)Bots in software engineering: a systematic mapping studyPeerJ Computer Science10.7717/peerj-cs.8668(e866)Online publication date: 9-Feb-2022
https://doi.org/10.7717/peerj-cs.866

Index Terms

Dialogue Act Classification for Virtual Agents for Software Engineers during Debugging
1. Computing methodologies
  1. Artificial intelligence
  2. Machine learning
    1. Learning paradigms
2. Software and its engineering
  1. Software creation and management
    1. Software verification and validation
      1. Software defect analysis
        Software testing and debugging

Index terms have been assigned to the content through auto-classification.

Recommendations

Contextual Dialogue Act Classification for Open-Domain Conversational Agents
SIGIR'19: Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval

Classifying the general intent of the user utterance in a conversation, also known as Dialogue Act (DA), e.g., open-ended question, statement of opinion, or request for an opinion, is a key step in Natural Language Understanding (NLU) for conversational ...
Dialog acts in greeting and leavetaking in social talk
ISIAA 2017: Proceedings of the 1st ACM SIGCHI International Workshop on Investigating Social Interactions with Artificial Agents

Conversation proceeds through dialogue moves or acts, and dialog act annotation can aid the design of artificial dialog. While many dialogs are task-based or instrumental, with clear goals, as in the case of a service encounter or business meeting, ...
Dialogue act annotation for consulting dialogue corpus
IUCS '09: Proceedings of the 3rd International Universal Communication Symposium

This paper introduces a new corpus of consulting dialogues, which is designed for training a dialogue manager that can handle consulting dialogues through spontaneous interactions from the tagged dialogue corpus. We have collected 130 h of consulting ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

ICSEW'20: Proceedings of the IEEE/ACM 42nd International Conference on Software Engineering Workshops

June 2020

831 pages

ISBN:9781450379632

DOI:10.1145/3387940

Copyright © 2020 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGSOFT: ACM Special Interest Group on Software Engineering
KIISE: Korean Institute of Information Scientists and Engineers
IEEE CS

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 25 September 2020

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed limited

Funding Sources

National Science Foundation

Conference

ICSE '20

Sponsor:

SIGSOFT
KIISE

ICSE '20: 42nd International Conference on Software Engineering

June 27 - July 19, 2020

Seoul, Republic of Korea

Upcoming Conference

ICSE 2025

2025 IEEE/ACM 46th International Conference on Software Engineering

April 26 - May 3, 2025

Ottawa , ON , Canada

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

3
Total Citations
View Citations
94
Total Downloads

Downloads (Last 12 months)13
Downloads (Last 6 weeks)1

Reflects downloads up to 08 Mar 2025

Other Metrics

View Author Metrics

Citations

Cited By

Ribeiro TSiqueira SDe Bayser M(2024)Identifying intentions in conversational tools: a systematic mappingProceedings of the 20th Brazilian Symposium on Information Systems10.1145/3658271.3658286(1-10)Online publication date: 20-May-2024
https://dl.acm.org/doi/10.1145/3658271.3658286
Hart JAuBuchon JMason SKuttal S(2024)Navigating NLU Challenges in Pair Programming Agents: A Study on Data Size, Gender, Language, and Domain EffectsArtificial Intelligence in HCI10.1007/978-3-031-60606-9_20(356-375)Online publication date: 1-Jun-2024
https://doi.org/10.1007/978-3-031-60606-9_20
Santhanam SHecking TSchreiber AWagner S(2022)Bots in software engineering: a systematic mapping studyPeerJ Computer Science10.7717/peerj-cs.8668(e866)Online publication date: 9-Feb-2022
https://doi.org/10.7717/peerj-cs.866

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten