Abstract
In order to evaluate the performance of the dialogue-manager component of a developing, Slovenian and Croatian spoken dialogue system, two Wizard-of-Oz experiments were performed. The only difference between the two experiment settings was in the dialogue-management manner, i.e., while in the first experiment dialogue management was performed by a human, the wizard, in the second experiment it was performed by the newly-implemented dialogue-manager component. The data from both Wizard-of-Oz experiments was evaluated with the PARADISE evaluation framework, a potential general methodology for evaluating and comparing different versions of spoken-language dialogue systems. The study ascertains a remarkable difference in the performance functions when taking different satisfaction-measure sums or even individual scores as the target to be predicted, it proves the indispensableness of the recently introduced database parameters when evaluating information-providing dialogue systems, and it confirms the dialogue manager’s cooperativity subject to the incorporated knowledge representation.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
Walker, M.A., Litman, D., Kamm, C.A., Abella, A.: PARADISE: A General Framework for Evaluating Spoken Dialogue Agents. In: Proc. 35th Annual Meeting of the Association of Computational Linguistics, Madrid, Spain, pp. 271–280 (1997)
Hajdinjak, M., Mihelič, F.: The PARADISE Evaluation Framework: Issues and Findings. Computational Linguistics 32(2), 263–272 (2006)
Žibert, J., Martinčić-Ipšić, S., Hajdinjak, M., Ipšić, I., Mihelič, F.: Development of a Bilingual Spoken Dialog System for Weather Information Retrieval. In: Proc. 8th European Conference on Speech Communication and Technology, Geneva, Switzerland, pp. 1917–1920 (2003)
Hajdinjak, M., Mihelič, F.: Conducting the Wizard-of-Oz Experiment. Informatica 28(4), 425–430 (2004)
Hajdinjak, M., Mihelič, F.: Information-providing dialogue management. In: Sojka, P., Kopeček, I., Pala, K. (eds.) TSD 2004. LNCS (LNAI), vol. 3206, pp. 595–602. Springer, Heidelberg (2004)
Hajdinjak, M., Mihelič, F.: A Dialogue-Management Evaluation Study. Journal of Computing and Information Technology (to appear, 2007)
Hajdinjak, M.: Knowledge Representation and Performance Evaluation of Cooperative Automatic Dialogue Systems, Ph. D. Thesis, Faculty of Electrical Engineering, University of Ljubljana, Slovenia (2006)
Walker, M.A., Litman, D.J., Kamm, C.A., Abella, A.: Evaluating spoken dialogue agents with paradise: Two case studies. Computer Speech and Language 12(3), 317–347 (1998)
Di Eugenio, B., Glass, M.: The Kappa statistic: a second look. Computational Linguistics 30(1), 95–101 (2004)
Hone, K.S., Graham, R.: Towards a tool for the Subjective Assesment of Speech System Interfaces (SASSI). Natural Language Engineering: Special Issue on Best Practice in Spoken Dialogue Systems 6(3-4), 287–303 (2000)
Walker, M.A., Borland, J., Kamm, C.A.: The utility of elapsed time as a usability metric for spoken dialogue systems. In: Proc. ASRU, Keystone, USA, pp. 317–320 (1999)
Author information
Authors and Affiliations
Editor information
Rights and permissions
Copyright information
© 2007 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Hajdinjak, M., Mihelič, F. (2007). A Wizard-of-Oz System Evaluation Study. In: Matoušek, V., Mautner, P. (eds) Text, Speech and Dialogue. TSD 2007. Lecture Notes in Computer Science(), vol 4629. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-540-74628-7_69
Download citation
DOI: https://doi.org/10.1007/978-3-540-74628-7_69
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-540-74627-0
Online ISBN: 978-3-540-74628-7
eBook Packages: Computer ScienceComputer Science (R0)