Abstract
In this paper, we present the SpeechEval user simulation platform for spoken dialog system (SDS) evaluation. Our user simulation realizes a complete end-to-end user test system with spoken input and output that calls speech-based dialog systems via telephone, and is intended to be used in the development and evaluation of real-life SDSs. We give an overview of the system architecture and present the results of applying our approach to two different kinds of systems in two domains.
User simulations for SDS are developed for many different purposes, often to test or train specific system components, like dialog manager, language model or the speech recognition itself. Many user simulations are developed in the context of research systems, which differ from commercially used systems in many aspects. In contrast, our user simulation uses a speech interface, can be carried over to new domains easily by learning characteristics of user behavior from a corpus, and treats the evaluated SDS as a black box.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
Chung G (2004) Developing a flexible spoken dialog system using simulation. In: Proceedings of the 42nd Meeting of the Association for Computational Linguistics (ACL’04), Barcelona, Spain, pp 63–70
Filisko E, Seneff S (2005) Developing city name acquisition strategies in spoken dialogue systems via user simulation. In: Proceedings of SigDial 6, Lisbon, Portugal, pp 144–154
Georgila K, Henderson J, Lemon O (2005) Learning user simulations for information state update dialogue systems. In: Proceedings of the 9th European Conference on Speech Communication and Technology (Eurospeech), Lisbon, Portugal
Georgila K, Henderson J, Lemon O (2006) User simulation for spoken dialogue systems: Learning and evaluation. In: Proceedings of Interspeech
Götze J, Scheffler T, Roller R, Reithinger N (2010) User simulation for the evaluation of bus information systems. In: Proceedings of 2010 IEEE Workshop on Spoken Language Technology. IEEE Workshop on Spoken Language Technology (SLT-2010), December 12–15, Berkeley, CA, United States, pp 454–459
López-Cózar R, de la Torre A, Segura J, Rubio A (2003) Assessment of dialog systems by means of a new simulation technique. Speech Communication 40:387–407
Pfalzgraf A, Pfleger N, Schehl J, Steigner J (2008) ODP: Ontology-based dialogue platform. Tech. rep., SemVox GmbH
Raux A, Langner B, Black A, Eskenazi M (2005) Let’s go public! taking a spoken dialog system to the real world. In: Proceedings of Interspeech, Lisbon, Portugal
Schatzmann J, Weilhammer K, Stuttle M, Young S (2006) A survey of statistical user simulation techniques for reinforcement-learning of dialogue management strategies. The Knowledge Engineering Review
Scheffler T, Roller R, Reithinger N (2009) Semi-automatic creation of resources for spoken dialog systems. In: Mertsching B, Hund M, Aziz Z (eds) KI 2009: Advances in Artificial Intelligence, Springer, LNAI, vol 5803, pp 209–216
Acknowledgements
SpeechEval was carried out in collaboration with the TU Berlin Quality and Usability Labs, and was funded by the IBB through the ProFIT framework, grant #10140648, cofinanced by the European Regional Development Fund.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2011 Springer Science+Business Media, LLC
About this paper
Cite this paper
Scheffler, T., Roller, R., Reithinger, N. (2011). SpeechEval: A Domain-Independent User Simulation Platform for Spoken Dialog System Evaluation. In: Delgado, RC., Kobayashi, T. (eds) Proceedings of the Paralinguistic Information and its Integration in Spoken Dialogue Systems Workshop. Springer, New York, NY. https://doi.org/10.1007/978-1-4614-1335-6_29
Download citation
DOI: https://doi.org/10.1007/978-1-4614-1335-6_29
Published:
Publisher Name: Springer, New York, NY
Print ISBN: 978-1-4614-1334-9
Online ISBN: 978-1-4614-1335-6
eBook Packages: EngineeringEngineering (R0)