Abstract
The demand for intelligent dialogue systems has increased rapidly in recent years. However, building such systems involves numerous complicated processes, including training data construction for language understanding. Although training data is essential in dialogue systems, it isn’t easy to find consistent guidelines that can facilitate such construction. In this paper, we propose a systematic construction process for NLU datasets with guidelines, especially considering the specificity of dialogue data. Two new datasets are constructed following the proposed procedure. We suggest using question–answering (QA) data instead of dialogue data to overcome the data shortage issue. To the best of our knowledge, this is the first attempt at using QA data for constructing a dialogue system. The process is demonstrated with a concrete example from the healthcare domain, which has rarely been considered when studying dialogue systems. The target system aims to diagnose illnesses based on user symptoms and provide healthcare, the diagnostic techniques for which were written in Korean. We present in detail the method to define intent and entity types and slots from the QA data. The effectiveness of our approach is verified by the experimental results from two crucial language understanding tasks: query intent detection and medical entity recognition. The tasks were performed using four variations of a state-of-the-art language representation model called bidirectional encoder representations from transformers. We obtained a satisfactory result on both tasks with a best f1 score of 0.84 and 0.92 for the intent detection and medical entity recognition.
Similar content being viewed by others
References
Gao J, Galley M, Li L (2014) Neural Approaches to Conversational AI, Foundations And Trends In Information Retrieval, vol. 13, issue 2–3, pp. 127–298, Feb
Gao J, Galley M, Li L Neural Approaches to Conversational AI, ACL and SIGIR tutorial, Jul. (2018) [Online]. Available: https://1drv.ms/p/s!AshEqwB44aR6k8wgo9s-Jk3-A8ZiaQ
Chen H, Liu X, Yin D, Tang J (2017) A survey on dialogue systems: recent advances and new frontiers. SIGKDD Explorations 19(2):25–35
Boyanov M, Nakov P, Moschitti A, Da San Martino G, Koychev I (2017) Building chatbots from forum data: Model selection using question answering metrics. In: Proceedings of the International Conference Recent Advances in Natural Language Processing, RANLP, 2017, pp 121–129
Akasaki S, Kaji N (2017) Chat detection in an intelligent assistant: Combining task-oriented and non-task-oriented spoken dialogue systems. In: Proc. of ACL’17, pp 1308–1319
Serban IV, Lowe R, Charlin L, Pineau J (2015) A survey of available corpora for building data-driven dialogue systems, arXiv:1512.05742
Peckham J (1993) A new generation of spoken dialogue systems: results and lessons from the sundial project. In: Proceedings of the third european conference on speech communication and technology (EUROSPEECH), pp 33–40
Seneff S, Polifroni J (2000) Dialogue Management in the Mercury Flight Reservation System. In: Proceedings of ANLP-NAACL 2000 Workshop: conversational systems
Gorin AL (1995) On automated language acquisition. J Acoust Soc Am 97(6):3441–3461
Gorin AL, Riccardi G, Wright JH (1997) How may I help you? Speech Comm 23(1):113–127
Levin E, Pieraccini R, Eckert W (2000) A stochastic model of human-machine interaction for learning dialog strategies. In: IEEE Transactions on speech and audio processing, vol 8, pp 11–23
Walker M, Kamm C, Litman D (2000) Towards developing general models of usability with PARADISE. Nat Lang Eng 6(3-4):363–377
Williams J, Young S (2007) Partially observable Markov decision processes for spoken dialog systems. Computer Speech & Language 21:393–422
Young S, Gasic M, Thomson B, Williams J (2013) POMDP-Based statistical spoken dialog systems: A review. In: proceedings of the IEEE, vol 101, pp 1160–1179
Ren H, Xu W, Zhang Y, Yan Y (2013) Dialog state tracking using conditional random fields. In: SIGDIAL Conference, pp 457–461
Zhang X, Wang H (2016) A joint model of intent determination and slot filling for spoken language understanding. In: Proceedings of the twenty-fifth international joint conference on artificial intelligence, AAAI Press, pp 2993–2999
Shah DJ, Gupta R, Fayazi AA, Hakkani-Tur D (2019) Robust zero-shot cross-domain slot filling with example values. In: Proceedings of the 57th annual meeting of the association for computational linguistics, pp 5484–5490
Bing L, Ian L (2021) Attention-Based Recurrent Neural Network Models for Joint Intent Detection and Slot Filling. In: Proceedings of INTERSPEECH, pp 685–689
Xia C, Zhang C, Yan X, Chang Y, Yu PS (2016) Zero-shot User Intent Detection via Capsule Neural Networks. In: Proceedings of EMNLP, 2018
Williams J, Raux A, Ramachandran D, Black A (2013) The dialog state tracking challenge. In: Proceedings of the SIGDIAL 2013 Conference, pp 404–413
Henderson M, Thomson B, Young SJ (2013) Deep Neural Network Approach for the Dialog State Tracking Challenge, Proceedings of the SIGDIAL, 2013, Conference
Liu F, Perez J (2017) Gated End-to-End Memory Networks. In: Proceedings of EACL
Shi H, Ushio T, Endo M, Yamagami K, Horii N (2016) Convolutional Neural Networks for Multi-topic Dialog State Tracking Inproceedings of IWSDS
Shah P, Hakkani-Tur D, Heck L (2016) Interactive reinforcement learning for task-oriented dialogue management. In: NIPS deep learning for action and interaction workshop
Mo K, Zhang Y, Li S, Li J, Yang Q (2018) Personalizing a dialogue system with transfer reinforcement learning. In: ThirtySecond AAAI conference on artificial intelligence
Hemphill CT, Godfrey JJ, Doddington GR (1990) The atis spoken language systems pilot corpus. Inproceedings of the DARPA speech and natural language workshop, pp 96–101
Henderson M, Thomson B, Williams J (2014) The second dialog state tracking challenge. In: Special Interest Group on Discourse and Dialogue (SIGDIAL)
Henderson M, Thomson B, Williams J (2014) Dialog state tracking challenge 2 & 3. In: Special Interest Group on Discourse and Dialogue (SIGDIAL)
Kim S, DHaro LF, Banchs RE, Williams J, Henderson M (2015) Dialog state tracking challenge 4
Kim S, DHaro LF, Banchs RE, Williams JD, Henderson M, Yoshino K (2016) The fifth dialog state tracking challenge. In: IEEE Spoken Language Technology Workshop (SLT)
Yan Z, Duan N, Chen P, Zhou M, Zhou J, Li Zhoujun (2017) Building task-oriented dialogue systems for online shopping. Inproceedings of the Thirty-First AAAI Conference on Artificial Intelligence, AAAI Press, pp 4618–4625
Zhang Z, Li J, Zhu P, Zhao H (2018) Modeling multi-turn conversation with deep utterance aggregation. Inproceedings of the 27th International Conference on Computational Linguistics(COLING), 2018, pp 3740–3752
Wei Z, Liu Q, Peng B, Tou H, Chen T, Huang X, Wong K-F, Dai X (2018) Task-oriented dialogue system for automatic diagnosis. Inproceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2:, Short Papers), 2, pp 201–207
Budzianowski P, Wen T-H, Tseng B-H, Casanueva I, Ultes S, Ramadan O, Gasic M (2018) MultiWOZ - A Large-Scale Multi-Domain Wizard-of-Oz Dataset for Task-Oriented Dialogue Modelling. Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp 5016–5026
Dai AM, Le QV (2015) Semi-supervised Sequence Learning. Inproceedings of the 28th International Conference on Neural Information Processing Systems - 2, NIPS’15, pp 3079–3087
Peters M, Neumann M, Iyyer M, Gardner M, Clark C, Lee K, Zettlemoyer L (2018) Deep contextualized word representations. Inproceedings of NAACL
Radford A, Narasimhan TSK, Sutskever I (2018) Improving language understanding with unsupervised learning. Technical report, OpenAI
Devlin J, Chang M-W, Lee K, Toutanova K (2019) Bert: Pre-training of deep bidirectional transformers for language understanding, Inproceedings of NAACL
Kim YM, Lee TH (2020) Korean Clinical Entity Recognition from Diagnosis Text using BERT. In: BMC medical informatics and decision making supplement
Yang Z, Dai Z, Yang Y, Carbonell J, Salakhutdinov R, Le QV (2019) XLNEt: Generalized Autoregressive Pretraining for Language Understanding. Inproceedings of the 32th International Conference on Neural Information Processing Systems (NIPS’19), pp 5753–5763
Mezza S, Cervone A, Tortoreto G, Stepanov E, Riccardi G (2018) Iso-standard domain-independent dialogue act tagging for conversational agents. Inproceedings of the 27th International Conference on Computational Linguistics, pp 3539–3551
Chen Z, Yang R, Zhao Z, Cai D, He X (2018) Dialogue act recognition via crf-attentive structured network. In: The 41st international ACM SIGIR conference on research & development in information retrieval, pp 225–234
Kumar H, Agarwal A, Dasgupta R, Joshi S (2018) Dialogue act sequence labeling using hierarchical encoder with CRF. Inproceedings of the Thirty-Second AAAI Conference on Artificial Intelligence, (AAAI-18), pp 3440–3447
Raheja V, Tetreault J (2019) Dialogue act classification with context-aware self-attention. Inproceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics:, Human Language Technologies. Long and Short Papers, 1, pp 3727–3733
Qu C, Yang L, Croft WB, Zhang Y, Trippas J, Qiu M (2019) User intent prediction in information-seeking conversations. Inproceedings of the CHIIR
Kim Y (2019) Training set expansion using word embeddings for korean medical information extraction. In: emphVLDB Workshop on Data Management and Analytics for Medicine and Healthcare (DMAH), pp 261–274
Mezza S, Cervone A, Tortoreto G, Stepanov E, Riccardi G (2018) Iso-standard domain-independent dialogue act tagging for conversational agents. Inproceedings of the 27th International Conference on Computational Linguistics, pp 3539–3551
Chen Z, Yang R, Zhao Z, Cai D, He X (2018) Dialogue act recognition via crf-attentive structured network. In: The 41st international ACM SIGIR conference on research & development in information retrieval, pp 225–234
Raheja V, Tetreault J (2019) Dialogue act classification with context-aware self-attention. Inproceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics:, Human Language Technologies, 1, Long and Short Papers, pp 3727–3733
Kumar H, Agarwal A, Dasgupta R, Joshi S (2018) Dialogue act sequence labeling using hierarchical encoder with CRF. Inproceedings of the Thirty-Second AAAI Conference on Artificial Intelligence, (AAAI-18), pp 3440–3447
Jurafsky D, Martin JH (2019) Chap 24. Dialog Systems and Chatbots, In ech and Language Processing, draft chapters in progress
Lu K, Zhang S, Chen X (2019) Goal-Oriented Dialogue policy learning from failures, inproceedings of the Thirty-Third AAAI conference on artificial intelligence, AAAI Press, pp 2596–2603
Lan Z, Chen M, Goodman S, Gimpel K, Sharma P, Soricut R (2020) ALBERT: A Lite BERT for Self-supervised Learning of Language Representations, Inproceedings of the Eighth International Conference on Learning Representations
Clark K, Luong M-T, Le QV, Manning CD (2020) ELECTRA: Pre-training Text Encoders as Discriminators Rather Than Generators, Inproceedings of the Eighth International Conference on Learning Representations
Singh S, Mahmood A (2021) The NLP cookbook: modern recipes for transformer based deep learning architectures. In: IEEE Access, pp. 68675 -, p 68702
Li L, Li C, Ji D (2021) Deep context modeling for multi-turn response selection in dialogue systems. In: Information Processing & Management Volume, vol 58
Balaraman V, Magnini B (2021) Domain-Aware Dialogue state tracker for Multi-Domain dialogue systems, in IEEE/ACM transactions on audio. Speech, and Language Processing 29:866–873
Chen L, Chen Z, Tan B, Long S, Gašić M, Yu K (2019) Agentgraph: Toward Universal Dialogue Management With Structured Deep Reinforcement Learning, In IEEE/ACM Transactions on Audio. Speech Lang Process 27(9):1378–1391
Firdaus M, Golchha H, Ekbal A, Bhattacharyya P (2021) Cognitive Computation (2021) A Deep Multi-task Model for Dialogue Act Classification, Intent Detection and Slot Filling. In: PLOS ONE
Pan B, Yang Y, Li B, Cai D (2021) Self-supervised attention flow for dialogue state tracking. In: Neurocomputing, vol 440, pp 279–286
Campillos-Llanos L, Deleger L, Grouin C, Hamon T, Ligozat A-L, Neveo A (2018) A French clinical corpus with comprehensive semantic annotations: development of the Medical Entity and Relation LIMSI Annotated Text corpus (MERLOT). In: Language resources and evaluation, vol 52
Chen Y, Zhou C, Li T, Wu H, Zhao X, Ye K, Liao J (2019) Named entity recognition from Chinese adverse drug event reports with lexical feature based BiLSTM-CRF and tri-training. In: J Biomed Inform
Patel P, Davey D, Panchal V, Pathak P (2018) Annotation of a large clinical entity corpus. Inproceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp 2033–2042
Wu J, Yılmaz E, Zhang M, Li H, Tan K (2020) Deep Spiking Neural Networks for Large Vocabulary Automatic Speech Recognition. In: Front Neurosci
Dominguez-Morales JP, Liu Q, James R, Gutierrez-Galan D (2018) Deep Spiking Neural Network model for time-variant signals classification: a real-time speech recognition approach, Inproceedings of the International Joint Conference on Neural Networks (IJCNN)
Yang S et al (2019) Real-Time Neuromorphic system for Large-Scale Conductance-Based spiking neural networks. In: IEEE transactions on cybernetics, vol 49, pp 2490–2503
Yang S, Gao T, Wang J, Deng B, Lansdell B, Linares-Barranco B (2021) Efficient Spike-Driven learning with dendritic Event-Based processing. In: Frontiers in Neuroscience, vol 15
Rastogi A, Zang X, Sunkara S, Gupta R, Khaitan P (2020) Towards scalable Multi-Domain conversational agents: the Schema-Guided dialogue dataset. AAAI 34(05):8689–8696
Xu L, Zhou Q, Gong K, Liang X, Tang J, Lin L (2019) End-to-end Knowledge-Routed Relational Dialogue System for Automatic Diagnosis. AAAI 33(01):7346–7353
Zeng G et al (2020) Meddialog: Large-scale Medical Dialogue Datasets. Inproceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pp 9241–9250
Liu W, Tang J, Qin J, Xu L, Li Z, Liang X (2020) MedDG:, A Large-scale Medical Consultation Dataset for Building Medical Dialogue System. arXiv:2010.07497
Li D, Ren Z, Ren P, Chen Z, Fan M, Ma J, de Rijke M (2021) Semi-Supervised Variational reasoning for medical dialogue generation. In: The 44th International ACM SIGIR Conference on Research & Development in Information Retrieval, pp 544–554
Funding
This work was supported by two projects, Classification of The Artists using Deep Neural Networks, funded by Hanyang University (201600000002255) and Bitstream-based Deep Multimodal Object Detection Framework in Real-time to Extend intelligent CCTV Surveillance (2020R1A2C2013687), funded by the National Research Foundation of Korea(NRF).
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Conflict of Interests
The authors declare that they have no conflict of interest.
Additional information
Publisher’s note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
About this article
Cite this article
Kim, YM., Lee, TH. & Na, SO. Constructing novel datasets for intent detection and ner in a korean healthcare advice system: guidelines and empirical results. Appl Intell 53, 941–961 (2023). https://doi.org/10.1007/s10489-022-03400-y
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10489-022-03400-y