Summary
In this chapter we will give a brief overview about what different methods of evaluation were applied to the SmartKom prototypes and which of them resulted in utilizable results and why others did not. Since there are no established benchmarking methods yet for multimodal HCI systems and very few debatable methods for monomodal dialogue systems our work within the SmartKom prototype evaluations was rather an exploration of new methods than the simple application of standard routines.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
N. Beringer, I. Jacobi, and S. von Tiedemann. Auswertung der ergonomischen Evaluation des SmartKom Prototypen-Szenario Home-Befragungszeitraum 10. Mrz 03 bis 20. Mrz 03. Memo SmartKom 16, Bavarian Archive for Speech Signals (BAS), 2003a.
N. Beringer, I. Jacobi, and S. von Tiedemann. Auswertung der ergonomischen Evaluation des SmartKom Prototypen-Szenario Mobil-Befragungszeitraum 04. Juli 03 bis 11. Juli 03. Memo SmartKom 18, Bavarian Archive for Speech Signals (BAS), 2003b.
N. Beringer, I. Jacobi, and S. von Tiedemann. Auswertung der ergonomischen Evaluation des SmartKom Prototypen-Szenario Public-Befragungszeitraum 11. Juni 03 bis 13. Juni 03. Memo SmartKom 17, Bavarian Archive for Speech Signals (BAS), 2003c.
N. Beringer, U. Kartal, M. Libossek, and S. Steininger. Gestaltung der End-to-End Evaluation in SmartKom 2.0. Technical Document SmartKom 19, Bavarian Archive for Speech Signals (BAS), 2002a.
N. Beringer, U. Kartal, K. Louka, F. Schiel, and U. Türk. PROMISE: A Procedure for Multimodal Interactive System Evaluation. In: Proc. Workshop “Multimodal Resources and Multimodal Systems Evaluation”, pp. 77–80, Las Palmas, Spain, 2002b.
N. Beringer, K. Louka, V. Penide-Lopez, and U. Türk. End-To-End Evaluation of Multimodal Dialogue Systems — Can We Transfer Established Methods? In: Proc. 3rd Int. Conf. on Language Resources and Evaluation (LREC 2002), pp. 558–563, Las Palmas, Spain, 2002c.
N. Beringer, H. Mögele, I. Jacobi, C. Beiras-Cunqueiro, L. Wang, U. Türk, G. Kouam-Wotchung, M. Mozul, and U. Kartal. Auswertung der End-to-End Evaluation des SmartKom Prototypen (technische Evaluation, Evaluationstool, PROMISE)-Szenario Home-, Aufnahmezeitraum 01. April 02 bis 31. Juni 02. Memo SmartKom 13, Bavarian Archive for Speech Signals (BAS), 2002d.
N. Beringer, H. Mögele, I. Jacobi, C. Beiras-Cunqueiro, L. Wang, U. Türk, G. Kouam-Wotchung, M. Mozul, and U. Kartal. Auswertung der End-to-End Evaluation des SmartKom Prototypen (technische Evaluation, Evaluationstool, PROMISE)-Szenario Mobil-, Aufnahmezeitraum 01. August 02 bis 31. August 02. Memo SmartKom 15, Bavarian Archive for Speech Signals (BAS), 2003d.
S. Burger, K. Weilhammer, F. Schiel, and H.G. Tillmann. Verbmobil Data Collection and Annotation. In: W. Wahlster (ed.), Verbmobil: Foundations of Speech-to-Speech Translation, pp. 537–549, Berlin Heidelberg Germany, 2000. Springer.
I. Jacobi, H. Mögele, S. von Tiedemann, and U. Türk. Auswertung der End-to-end Evaluation des SmartKom Prototypen (technische Evaluation, PROMISE)-Szenario Home, Public, Mobil. Memo SmartKom 19, Bavarian Archive for Speech Signals (BAS), 2004.
U. Kartal, N. Beringer, and S. Steininger. Auswertung der ergonomischen Evaluation der WOZ-Aufnahmen im Projekt SmartKom. Memo SmartKom 11, Bavarian Archive for Speech Signals (BAS), 2002a.
U. Kartal, N. Beringer, S. Steininger, and D. Sonntag. Auswertung der ergonomischen Evaluation des SmartKom Prototypen-Szenario Home-, Befragungszeitraum 01. April 02 bis 31. Juni 02. Memo SmartKom 12, Bavarian Archive for Speech Signals (BAS), 2002b.
U. Kartal, N. Beringer, S. Steininger, and D. Sonntag. Auswertung der ergonomischen Evaluation des SmartKom Prototypen-Szenario Mobil-, Befragungszeitraum 01. August 02 bis 31. August 02. Memo SmartKom 14, Bavarian Archive for Speech Signals (BAS), 2003.
F. Schiel. SmartKom — Langzeitevaluation. Memo SmartKom 23, Bavarian Archive for Speech Signals (BAS), 2004.
F. Schiel and U. Türk. Wizard-of-Oz Recordings, 2006. In this volume.
M.A. Walker, D.J. Litman, C.A. Kamm, and A. Abella. PARADISE: A Framework for Evaluating Spoken Dialogue Agents. In: Proc. 35th ACL, Madrid, Spain, 1997.
M.A. Walker, D.J. Litman, C.A. Kamm, and A. Abella. Evaluating Spoken Dialogue Agents With Paradise: Two Case Studies. Computer, Speech and Language, 12(3), 1998.
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2006 Springer-Verlag Berlin Heidelberg
About this chapter
Cite this chapter
Schiel, F. (2006). Evaluation of Multimodal Dialogue Systems. In: Wahlster, W. (eds) SmartKom: Foundations of Multimodal Dialogue Systems. Cognitive Technologies. Springer, Berlin, Heidelberg. https://doi.org/10.1007/3-540-36678-4_38
Download citation
DOI: https://doi.org/10.1007/3-540-36678-4_38
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-540-23732-7
Online ISBN: 978-3-540-36678-2
eBook Packages: Computer ScienceComputer Science (R0)