Abstract
Dialogue systems are embedded in smartphones and Artificial Intelligence (AI) speakers and are widely used through text and speech. To achieve a human-like dialogue system, one of the challenges is to have a standard automatic evaluation metric. Existing metrics like BLEU, METEOR, and ROUGE have been proposed to evaluate dialogue system. However, those methods are biased and correlate very poorly with human judgements of response quality. On the other hand, RUBER is applied to not only train the relatedness between the dialogue system generated reply and given query, but also measure the similarity between the ground truth and generated reply. It showed higher correlation with human judgements than BLEU and ROUGE. Based on RUBER, instead of static embedding, we explore using BERT contextualised word embedding to get a better evaluation metrics. The experiment shows that our evaluation metrics using BERT are more correlated to human judgement than RUBER. Experimental results show that BERT feature based evaluation metric had 0.31 and 0.26 points higher scores and BERT fine tune evaluation metric got higher 0.39 and 0.36 points in Pearson and Spearman correlation with human judgement score than RUBER, respectively.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Walker, Marilyn A., et al.: PARADISE: a framework for evaluating spoken dialogue agents. In: 35th Annual Meeting of the Association for Computational Linguistics and 8th Conference of the European Chapter of the Association for Computational Linguistics, Association for Computational Linguistics (1997)
Möller, S., et al.: MeMo: towards automatic usability evaluation of spoken dialogue services by user error simulations. In: Ninth International Conference on Spoken Language Processing (2006)
Papineni, K., Roukos, S., Ward, T., Zhu, W.-J.: BLEU: a method for automatic evaluation of machine translation. In: Proceedings of the 40th annual meeting on association for computational linguistics. Association for Computational Linguistics, pp. 311–318 (2002)
Banerjee, S., Lavie, A.: METEOR : an automatic metric for MT evaluation with improved correlation with human judgements. In: Proceedings of the Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarisation (2005)
Lin, C.-Y.: ROUGE : a package for automatic evaluation of summaries. In: Text Summarisation Branches out : Proceedings of the ACL-04 Workshop (2004)
Liu, C.-W., Lowe, R., Serban, I., Noseworthy, M., Charlin, L., Pineau, J.: How NOT to evaluate your dialogue system : an empirical study of unsupervised evaluation metrics for dialogue response generation. In: Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing (2016)
Lowe, R., Noseworthy, M., Serban, I.V., Angerlard-Gontier, N., Bengio, Y., Pineau, J.: Towards an automatic turing test : learning to evaluate dialogue responses. In: Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, Association for Computational Linguistics, pp. 1116–1126 (2017)
Tao, C., Mou, L., Zhao, D., Yan, R.: RUBER : an unsupervised method for automatic evaluation of open-domain dialog systems. In: Mcllraith and Weinberger (2018)
Devlin, J., Chang, M.-W., Lee, K., Toutanova, K.: BERT : pre-training of deep bidirectional transformers for language understanding. In: Proceedings of NAACL-HLT, pp. 4171–4186 (2018)
Li, Y., et al.: Dailydialog: A manually labelled multi-turn dialogue dataset. In: Proceedings of the Eighth International Joint Conference on Natural Language Processing (Volume 1: Long Papers) (2017)
Williams, A., Nangia, N., Bowman, S.: A broad-coverage challenge corpus for sentence understanding through inference. In: Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pp. 1112–1122 (2018)
Imamura, K., Sumita, E.: Recycling a pre-trained bert encoder for neural machine translation. In: Proceedings of the 3rd Workshop on Neural Generation and Translation (WNGT 2019), pp. 23–31 (2019)
Reimers, N., Gurevich, I.: Sentence-bert : Sentence embeddings using siamese bert-networks. In: Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, Association for Computational Linguistics (2019)
Yan, R., Song, Y., Wu, H.: Learning to respond with deep neural networks for retrieval-based human-computer conversation system. In: Proceedings of the 39th International CM SIGIR conference on Research and Development in Information Retrieval. pp. 55–64 (2016)
Luong, T., Pham, H., Manning, C.D.: Effective approaches to attention-based neural machine translation. In: Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, Association for Computational Linguistics (2015)
Danescu-Niculescu-Mizil, C., Lee, L.: Chameleons in imagined conversations : a new approach to understanding coordination of linguistic style in dialogs. In: Proceedings of the 2nd workshop on cognitive modelling and computational linguistics, Association for Computational Linguistics (2011)
Lowe, R., Pow, N., Serban, I., Pineau, J.: The Ubuntu dialogue corpus: a large dataset for research in unstructured multi-turn dialogue systems. In: Proceedings of the 16th Annual Meeting of the Special Interest Group on Discourse and Dialogue, pp. 285–294, Prague, Czech Republic. Association for Computational Linguistics (2015)
Zhao, T., Pavel, A., Eskenazi, M., Gupta, P., Mehri, S., Bigham, J.P.: Investigating evaluation of open-domain dialogue systems with human generated multiple references. In: Proceedings of the SIGDial 2019 Conference, pp. 379–391 (2019)
Ritter, A., Cherry, C., Dolan, W.B.: Data-driven response generation in social media. In: Proceedings of the 2011 Conference on Empirical Methods in Natural Language Processing, pp. 583–593, Edinburgh, Scotland, UK. Association for Computational Linguistics (2011)
Fleiss, J.L.: Measuring nominal scale agreement among many raters. Psychol. Bull. 76, 378–382 (1971)
Landis, J., Koch, G.: The measurement of observer agreement for categorical data. Biometrics 33(1), 159–74 (1977)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2021 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Htar, K.T., Wang, Y., Wu, J., Hattori, G., Thida, A. (2021). BERT-Based Dialogue Evaluation Methods with RUBER Framework. In: Yada, K., et al. Advances in Artificial Intelligence. JSAI 2020. Advances in Intelligent Systems and Computing, vol 1357. Springer, Cham. https://doi.org/10.1007/978-3-030-73113-7_12
Download citation
DOI: https://doi.org/10.1007/978-3-030-73113-7_12
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-73112-0
Online ISBN: 978-3-030-73113-7
eBook Packages: Intelligent Technologies and RoboticsIntelligent Technologies and Robotics (R0)