Abstract
Developing automatic evaluation methods that are highly correlated with human assessment is crucial in the advancement of dialogue systems. User feedback in conversation system provides a signal that represents user preferences and response quality. The user feedback prediction (UFP) task aims to predict the probabilities of likes with machine-generated responses given a user query, offering a unique perspective to facilitate dialogue evaluation. In this paper, we propose a powerful UFP system, which leverages Chinese pre-trained language models (PLMs) to understand the user queries and system replies. To improve the robustness and generalization ability of our model, we also introduce adversarial training for PLMs and design a local and global model ensemble strategy. Our system ranks first in NLPCC 2023 shared Task 9 Track 1 (User Feedback Prediction). The experimental results show the effectiveness of the method applied in our system.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Notes
- 1.
We implemented it by calling the Baidu Translation API:https://api.fanyi.baidu.com/api.
- 2.
- 3.
- 4.
- 5.
- 6.
References
Shum, H., He, X., Li, D.: From Eliza to XiaoIce: challenges and opportunities with social chatbots. Front. Inf. Technol. Electron. Eng. 19(1), 10–26 (2018). https://doi.org/10.1631/FITEE.1700826
Deriu, J., et al.: Survey on evaluation methods for dialogue systems. Artif. Intell. Rev. 54(1), 755–810 (2021). https://doi.org/10.1007/s10462-020-09866-x
Inoue, H.: Multi-sample dropout for accelerated training and better generalization. arXiv preprint arXiv:1905.09788 (2019)
Goodfellow, I.J., Shlens, J., Szegedy, C.: Explaining and harnessing adversarial examples. arXiv preprint arXiv:1412.6572 (2014)
Bengio, Y., Ducharme, R., Vincent, P.: A neural probabilistic language model. In: Advances in Neural Information Processing Systems 13 (2000)
Li, J., Galley, M., Brockett, C., Gao, J., Dolan, B.: A diversity-promoting objective function for neural conversation models. arXiv preprint arXiv:1510.03055 (2015)
Papineni, K., Roukos, S., Ward, T., Zhu, W.-J.: BLEU: a method for automatic evaluation of machine translation. In: Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pp. 311–318. (2002)
Liu, C.-W., Lowe, R., Serban, I.V., Noseworthy, M., Charlin, L., Pineau, J.: How not to evaluate your dialogue system: an empirical study of unsupervised evaluation metrics for dialogue response generation. arXiv preprint arXiv:1603.08023 (2016)
Mehri, S., Eskenazi, M.: USR: an unsupervised and reference free evaluation metric for dialog generation. arXiv preprint arXiv:2005.00456 (2020)
Ma, L., Zhuang, Z., Zhang, W., Li, M., Liu, T.: Self-Eval: self-supervised fine-grained dialogue evaluation. arXiv preprint arXiv:2208.08094 (2022)
Sun, W., et al.: Simulating user satisfaction for the evaluation of task-oriented dialogue systems. In: Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 2499–2506 (2021)
Devlin, J., Chang, M.-W., Lee, K., Toutanova, K.: BERT: pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 (2018)
Ye, Z., Lu, L., Huang, L., Lin, L., Liang, X.: Towards quantifiable dialogue coherence evaluation. arXiv preprint arXiv:2106.00507 (2021)
Cui, Y., Che, W., Liu, T., Qin, B., Yang, Z.: Pre-training with whole word masking for Chinese BERT. IEEE/ACM Trans. Audio, Speech, Lang. Process. 29, 3504–3514 (2021)
Liu, Y., et al.: RoBERTa: a robustly optimized BERT pretraining approach. arXiv preprint arXiv:1907.11692 (2019)
Cui, Y., Che, W., Liu, T., Qin, B., Wang, S., Hu, G.: Revisiting pre-trained models for Chinese natural language processing. arXiv preprint arXiv:2004.13922 (2020)
Sun, Y., et al.: ERNIE 3.0: large-scale knowledge enhanced pre-training for language understanding and generation. arXiv preprint arXiv:2107.02137 (2021)
Miyato, T., Dai, A.M., Goodfellow, I.: Adversarial training methods for semi-supervised text classification. arXiv preprint arXiv:1605.07725 (2016)
Jiang, H., He, P., Chen, W., Liu, X., Gao, J., Zhao, T.: SMART: robust and efficient fine-tuning for pre-trained natural language models through principled regularized optimization. arXiv preprint arXiv:1911.03437 (2019)
Agrawal, S., Mamidi, R.: Lastresort at semeval-2022 task 4: towards patronizing and condescending language detection using pre-trained transformer based models ensembles. In: Proceedings of the 16th International Workshop on Semantic Evaluation (SemEval-2022), pp. 352–356 (2022)
Yu, W., Boenninghoff, B., Roehrig, J., Kolossa, D.: Rubcsg at semeval-2022 task 5: ensemble learning for identifying misogynous memes. arXiv preprint arXiv:2204.03953 (2022)
Sagi, O., Rokach, L.: Ensemble learning: a survey. Wiley Interdisc. Rev.: Data Min. Knowl. Discov. 8(4), e1249 (2018)
Madry, A., Makelov, A., Schmidt, L., Tsipras, D., Vladu, A.: Towards deep learning models resistant to adversarial attacks. arXiv preprint arXiv:1706.06083 (2017)
Acknowledgements
This work was supported in part by the National Natural Science Foundation of China under Grant Grant 62006034; in part by the Natural Science Foundation of Liaoning Province under Grant 2021-BS-067; and in part by the Dalian High-level Talent Innovation Support Plan under Grant 2021RQ056.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Wang, J., Leng, Y., Zhai, X., Zong, L., Lin, H., Xu, B. (2023). Adversarial Training and Model Ensemble for User Feedback Prediciton in Conversation System. In: Liu, F., Duan, N., Xu, Q., Hong, Y. (eds) Natural Language Processing and Chinese Computing. NLPCC 2023. Lecture Notes in Computer Science(), vol 14304. Springer, Cham. https://doi.org/10.1007/978-3-031-44699-3_33
Download citation
DOI: https://doi.org/10.1007/978-3-031-44699-3_33
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-44698-6
Online ISBN: 978-3-031-44699-3
eBook Packages: Computer ScienceComputer Science (R0)
