Abstract
In personal assistant dialog systems, intent models are classifiers that identify the intent of a user utterance, such as to add a meeting to a calendar or get the director of a stated movie. Rapidly adding intents is one of the main bottlenecks to scaling—adding functionality to—personal assistants. In this paper we show how interactive learning can be applied to the creation of statistical intent models. Interactive learning (Simard, ICE: enabling non-experts to build models interactively for large-scale lopsided problems, 2014) combines model definition, labeling, model building, active learning, model evaluation, and feature engineering in a way that allows a domain expert—who need not be a machine learning expert—to build classifiers. We apply interactive learning to build a handful of intent models in three different domains. In controlled lab experiments, we show that intent detectors can be built using interactive learning and then improved in a novel end-to-end visualization tool. We then applied this method to a publicly deployed personal assistant—Microsoft Cortana—where a non-machine learning expert built an intent model in just over 2 h, yielding excellent performance in the commercial service.
Work of the authors “Nobal B. Niraula” and “Pradeep Dasigi” was done while at Microsoft Research.
The authors “Jason D. Williams”, “Nobal B. Niraula”, and “Pradeep Dasigi” contributed equally to this work.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
Notes
- 1.
This approach assumes that the scores are directly comparable. In this paper, the classifiers are not guaranteed to produce comparable scores, but since only a handful of classifiers are used and their calibration is similar enough, this mismatch will not be a practical problem. We’ll return to this point in the conclusion.
- 2.
- 3.
The held-out test set excluded utterances which appeared in the training set, whereas in actual deployment, utterances in the training set may reappear. Therefore, these are conservative estimates which could underestimate performance.
References
Allwein EL, Schapire RE, Singer Y (2001) Reducing multiclass to binary: a unifying approach for margin classifiers. J Mach Learn Res 1:113–141
Beygelzimer A, Langford J, Ravikumar P (2009) Error-correcting tournaments. In: Algorithmic learning theory. Springer, Heidelberg, pp 247–262
Fukubayashi Y, Komatani K, Nakano M, Funakoshi K, Tsujino H, Ogata T, Okuno HG (2008) Rapid prototyping of robust language understanding modules for spoken dialogue systems. In: The Third International Joint Conference on Natural Language Processing (IJCNLP2008)
Glass JR, Weinstein E (2001) Speechbuilder: facilitating spoken dialogue system development. In: EUROSPEECH 2001 Scandinavia, 7th European conference on speech communication and technology, 2nd INTERSPEECH Event, Aalborg, 3–7 September 2001
Haffner P, Tur G, Wright JH (2003) Optimizing SVMs for complex call classification. In: Proceedings of IEEE International Conference on Acoustics, Speech, and Signal Processing, 2003 (ICASSP ’03), April 2003
Heck LP, Hakkani-Tür D, Tür G (2013) Leveraging knowledge graphs for web-scale unsupervised semantic parsing. In: Proceedings of INTERSPEECH, Lyon, 25–29 August 2013
Jung S, Lee C, Kim S, Lee GG (2008) Dialogstudio: a workbench for data-driven spoken dialog system development and management. Speech Comm 50:697–715
Sarikaya R, Hinton G, Ramabhadran B (2011) Deep belief nets for natural language call-routing. In: 2011 IEEE International conference on acoustics, speech and signal processing (ICASSP), pp 5680–5683
Schapire R, Singer Y (2000) Boostexter: a boosting-based system for text categorization. Mach Learn 39(2–3):135–168
Simard P, Chickering D, Lakshmiratan A, Charles D, Bottou L, Suarez CGJ, Grangier D, Amershi S, Verwey J, Suh J (2014) ICE: enabling non-experts to build models interactively for large-scale lopsided problems. http://arxiv.org/ftp/arxiv/papers/1409/1409.4814.pdf
Stumpf S, Rajaram V et al. (2007) Toward harnessing user feedback for machine learning. In: Proceedings IUI
Stumpf S, Rajaram V et al. (2009) Interacting meaningfully with machine learning systems: three experiments. Int J Hum Comput Stud 67(8):639–662
Tur G, Mori RD (2011) Spoken language understanding—systems for extracting semantic information from speech. Wiley, New York
Tur G, Hakkani-Tur D, Schapire RE (2005) Combining active and semi-supervised learning for spoken language understanding. Speech Comm 45(2):171–186
Tur G, Schapire R, Hakkani-Tur D (2003) Active learning for spoken language understanding. 1:I-276–I-279
Wang YY, Deng L, Acero A (2005) Spoken language understanding. IEEE Signal Process Mag 22(5):16–31
Acknowledgements
Thanks to Puneet Agrawal for assistance with the Cortana service and to Meg Mitchell, Lihong Li, Sheeraz Ahmad, Andrey Kolobov, and Saleema Amershi for helpful discussions.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2015 Springer International Publishing Switzerland
About this chapter
Cite this chapter
Williams, J.D. et al. (2015). Rapidly Scaling Dialog Systems with Interactive Learning. In: Lee, G., Kim, H., Jeong, M., Kim, JH. (eds) Natural Language Dialog Systems and Intelligent Assistants. Springer, Cham. https://doi.org/10.1007/978-3-319-19291-8_1
Download citation
DOI: https://doi.org/10.1007/978-3-319-19291-8_1
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-19290-1
Online ISBN: 978-3-319-19291-8
eBook Packages: Computer ScienceComputer Science (R0)