Skip to main content

Advertisement

SpringerLink
Log in

Constructing novel datasets for intent detection and ner in a korean healthcare advice system: guidelines and empirical results

  • Published:
Applied Intelligence Aims and scope Submit manuscript

Abstract

The demand for intelligent dialogue systems has increased rapidly in recent years. However, building such systems involves numerous complicated processes, including training data construction for language understanding. Although training data is essential in dialogue systems, it isn’t easy to find consistent guidelines that can facilitate such construction. In this paper, we propose a systematic construction process for NLU datasets with guidelines, especially considering the specificity of dialogue data. Two new datasets are constructed following the proposed procedure. We suggest using question–answering (QA) data instead of dialogue data to overcome the data shortage issue. To the best of our knowledge, this is the first attempt at using QA data for constructing a dialogue system. The process is demonstrated with a concrete example from the healthcare domain, which has rarely been considered when studying dialogue systems. The target system aims to diagnose illnesses based on user symptoms and provide healthcare, the diagnostic techniques for which were written in Korean. We present in detail the method to define intent and entity types and slots from the QA data. The effectiveness of our approach is verified by the experimental results from two crucial language understanding tasks: query intent detection and medical entity recognition. The tasks were performed using four variations of a state-of-the-art language representation model called bidirectional encoder representations from transformers. We obtained a satisfactory result on both tasks with a best f1 score of 0.84 and 0.92 for the intent detection and medical entity recognition.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8

Similar content being viewed by others

Notes

  1. https://aiopen.etri.re.kr/service_dataset.php

  2. https://aihub.or.kr/

  3. https://gluebenchmark.com/leaderboard

  4. https://github.com/hymllab/Healthcare-NLU

References

  1. Gao J, Galley M, Li L (2014) Neural Approaches to Conversational AI, Foundations And Trends In Information Retrieval, vol. 13, issue 2–3, pp. 127–298, Feb

  2. Gao J, Galley M, Li L Neural Approaches to Conversational AI, ACL and SIGIR tutorial, Jul. (2018) [Online]. Available: https://1drv.ms/p/s!AshEqwB44aR6k8wgo9s-Jk3-A8ZiaQ

  3. Chen H, Liu X, Yin D, Tang J (2017) A survey on dialogue systems: recent advances and new frontiers. SIGKDD Explorations 19(2):25–35

    Article  Google Scholar 

  4. Boyanov M, Nakov P, Moschitti A, Da San Martino G, Koychev I (2017) Building chatbots from forum data: Model selection using question answering metrics. In: Proceedings of the International Conference Recent Advances in Natural Language Processing, RANLP, 2017, pp 121–129

  5. Akasaki S, Kaji N (2017) Chat detection in an intelligent assistant: Combining task-oriented and non-task-oriented spoken dialogue systems. In: Proc. of ACL’17, pp 1308–1319

  6. Serban IV, Lowe R, Charlin L, Pineau J (2015) A survey of available corpora for building data-driven dialogue systems, arXiv:1512.05742

  7. Peckham J (1993) A new generation of spoken dialogue systems: results and lessons from the sundial project. In: Proceedings of the third european conference on speech communication and technology (EUROSPEECH), pp 33–40

  8. Seneff S, Polifroni J (2000) Dialogue Management in the Mercury Flight Reservation System. In: Proceedings of ANLP-NAACL 2000 Workshop: conversational systems

  9. Gorin AL (1995) On automated language acquisition. J Acoust Soc Am 97(6):3441–3461

    Article  Google Scholar 

  10. Gorin AL, Riccardi G, Wright JH (1997) How may I help you? Speech Comm 23(1):113–127

    Article  MATH  Google Scholar 

  11. Levin E, Pieraccini R, Eckert W (2000) A stochastic model of human-machine interaction for learning dialog strategies. In: IEEE Transactions on speech and audio processing, vol 8, pp 11–23

  12. Walker M, Kamm C, Litman D (2000) Towards developing general models of usability with PARADISE. Nat Lang Eng 6(3-4):363–377

    Article  Google Scholar 

  13. Williams J, Young S (2007) Partially observable Markov decision processes for spoken dialog systems. Computer Speech & Language 21:393–422

    Article  Google Scholar 

  14. Young S, Gasic M, Thomson B, Williams J (2013) POMDP-Based statistical spoken dialog systems: A review. In: proceedings of the IEEE, vol 101, pp 1160–1179

  15. Ren H, Xu W, Zhang Y, Yan Y (2013) Dialog state tracking using conditional random fields. In: SIGDIAL Conference, pp 457–461

  16. Zhang X, Wang H (2016) A joint model of intent determination and slot filling for spoken language understanding. In: Proceedings of the twenty-fifth international joint conference on artificial intelligence, AAAI Press, pp 2993–2999

  17. Shah DJ, Gupta R, Fayazi AA, Hakkani-Tur D (2019) Robust zero-shot cross-domain slot filling with example values. In: Proceedings of the 57th annual meeting of the association for computational linguistics, pp 5484–5490

  18. Bing L, Ian L (2021) Attention-Based Recurrent Neural Network Models for Joint Intent Detection and Slot Filling. In: Proceedings of INTERSPEECH, pp 685–689

  19. Xia C, Zhang C, Yan X, Chang Y, Yu PS (2016) Zero-shot User Intent Detection via Capsule Neural Networks. In: Proceedings of EMNLP, 2018

  20. Williams J, Raux A, Ramachandran D, Black A (2013) The dialog state tracking challenge. In: Proceedings of the SIGDIAL 2013 Conference, pp 404–413

  21. Henderson M, Thomson B, Young SJ (2013) Deep Neural Network Approach for the Dialog State Tracking Challenge, Proceedings of the SIGDIAL, 2013, Conference

  22. Liu F, Perez J (2017) Gated End-to-End Memory Networks. In: Proceedings of EACL

  23. Shi H, Ushio T, Endo M, Yamagami K, Horii N (2016) Convolutional Neural Networks for Multi-topic Dialog State Tracking Inproceedings of IWSDS

  24. Shah P, Hakkani-Tur D, Heck L (2016) Interactive reinforcement learning for task-oriented dialogue management. In: NIPS deep learning for action and interaction workshop

  25. Mo K, Zhang Y, Li S, Li J, Yang Q (2018) Personalizing a dialogue system with transfer reinforcement learning. In: ThirtySecond AAAI conference on artificial intelligence

  26. Hemphill CT, Godfrey JJ, Doddington GR (1990) The atis spoken language systems pilot corpus. Inproceedings of the DARPA speech and natural language workshop, pp 96–101

  27. Henderson M, Thomson B, Williams J (2014) The second dialog state tracking challenge. In: Special Interest Group on Discourse and Dialogue (SIGDIAL)

  28. Henderson M, Thomson B, Williams J (2014) Dialog state tracking challenge 2 & 3. In: Special Interest Group on Discourse and Dialogue (SIGDIAL)

  29. Kim S, DHaro LF, Banchs RE, Williams J, Henderson M (2015) Dialog state tracking challenge 4

  30. Kim S, DHaro LF, Banchs RE, Williams JD, Henderson M, Yoshino K (2016) The fifth dialog state tracking challenge. In: IEEE Spoken Language Technology Workshop (SLT)

  31. Yan Z, Duan N, Chen P, Zhou M, Zhou J, Li Zhoujun (2017) Building task-oriented dialogue systems for online shopping. Inproceedings of the Thirty-First AAAI Conference on Artificial Intelligence, AAAI Press, pp 4618–4625

  32. Zhang Z, Li J, Zhu P, Zhao H (2018) Modeling multi-turn conversation with deep utterance aggregation. Inproceedings of the 27th International Conference on Computational Linguistics(COLING), 2018, pp 3740–3752

  33. Wei Z, Liu Q, Peng B, Tou H, Chen T, Huang X, Wong K-F, Dai X (2018) Task-oriented dialogue system for automatic diagnosis. Inproceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2:, Short Papers), 2, pp 201–207

  34. Budzianowski P, Wen T-H, Tseng B-H, Casanueva I, Ultes S, Ramadan O, Gasic M (2018) MultiWOZ - A Large-Scale Multi-Domain Wizard-of-Oz Dataset for Task-Oriented Dialogue Modelling. Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp 5016–5026

  35. Dai AM, Le QV (2015) Semi-supervised Sequence Learning. Inproceedings of the 28th International Conference on Neural Information Processing Systems - 2, NIPS’15, pp 3079–3087

  36. Peters M, Neumann M, Iyyer M, Gardner M, Clark C, Lee K, Zettlemoyer L (2018) Deep contextualized word representations. Inproceedings of NAACL

  37. Radford A, Narasimhan TSK, Sutskever I (2018) Improving language understanding with unsupervised learning. Technical report, OpenAI

  38. Devlin J, Chang M-W, Lee K, Toutanova K (2019) Bert: Pre-training of deep bidirectional transformers for language understanding, Inproceedings of NAACL

  39. Kim YM, Lee TH (2020) Korean Clinical Entity Recognition from Diagnosis Text using BERT. In: BMC medical informatics and decision making supplement

  40. Yang Z, Dai Z, Yang Y, Carbonell J, Salakhutdinov R, Le QV (2019) XLNEt: Generalized Autoregressive Pretraining for Language Understanding. Inproceedings of the 32th International Conference on Neural Information Processing Systems (NIPS’19), pp 5753–5763

  41. Mezza S, Cervone A, Tortoreto G, Stepanov E, Riccardi G (2018) Iso-standard domain-independent dialogue act tagging for conversational agents. Inproceedings of the 27th International Conference on Computational Linguistics, pp 3539–3551

  42. Chen Z, Yang R, Zhao Z, Cai D, He X (2018) Dialogue act recognition via crf-attentive structured network. In: The 41st international ACM SIGIR conference on research & development in information retrieval, pp 225–234

  43. Kumar H, Agarwal A, Dasgupta R, Joshi S (2018) Dialogue act sequence labeling using hierarchical encoder with CRF. Inproceedings of the Thirty-Second AAAI Conference on Artificial Intelligence, (AAAI-18), pp 3440–3447

  44. Raheja V, Tetreault J (2019) Dialogue act classification with context-aware self-attention. Inproceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics:, Human Language Technologies. Long and Short Papers, 1, pp 3727–3733

  45. Qu C, Yang L, Croft WB, Zhang Y, Trippas J, Qiu M (2019) User intent prediction in information-seeking conversations. Inproceedings of the CHIIR

  46. Kim Y (2019) Training set expansion using word embeddings for korean medical information extraction. In: emphVLDB Workshop on Data Management and Analytics for Medicine and Healthcare (DMAH), pp 261–274

  47. Mezza S, Cervone A, Tortoreto G, Stepanov E, Riccardi G (2018) Iso-standard domain-independent dialogue act tagging for conversational agents. Inproceedings of the 27th International Conference on Computational Linguistics, pp 3539–3551

  48. Chen Z, Yang R, Zhao Z, Cai D, He X (2018) Dialogue act recognition via crf-attentive structured network. In: The 41st international ACM SIGIR conference on research & development in information retrieval, pp 225–234

  49. Raheja V, Tetreault J (2019) Dialogue act classification with context-aware self-attention. Inproceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics:, Human Language Technologies, 1, Long and Short Papers, pp 3727–3733

  50. Kumar H, Agarwal A, Dasgupta R, Joshi S (2018) Dialogue act sequence labeling using hierarchical encoder with CRF. Inproceedings of the Thirty-Second AAAI Conference on Artificial Intelligence, (AAAI-18), pp 3440–3447

  51. Jurafsky D, Martin JH (2019) Chap 24. Dialog Systems and Chatbots, In ech and Language Processing, draft chapters in progress

  52. Lu K, Zhang S, Chen X (2019) Goal-Oriented Dialogue policy learning from failures, inproceedings of the Thirty-Third AAAI conference on artificial intelligence, AAAI Press, pp 2596–2603

  53. Lan Z, Chen M, Goodman S, Gimpel K, Sharma P, Soricut R (2020) ALBERT: A Lite BERT for Self-supervised Learning of Language Representations, Inproceedings of the Eighth International Conference on Learning Representations

  54. Clark K, Luong M-T, Le QV, Manning CD (2020) ELECTRA: Pre-training Text Encoders as Discriminators Rather Than Generators, Inproceedings of the Eighth International Conference on Learning Representations

  55. Singh S, Mahmood A (2021) The NLP cookbook: modern recipes for transformer based deep learning architectures. In: IEEE Access, pp. 68675 -, p 68702

  56. Li L, Li C, Ji D (2021) Deep context modeling for multi-turn response selection in dialogue systems. In: Information Processing & Management Volume, vol 58

  57. Balaraman V, Magnini B (2021) Domain-Aware Dialogue state tracker for Multi-Domain dialogue systems, in IEEE/ACM transactions on audio. Speech, and Language Processing 29:866–873

    Google Scholar 

  58. Chen L, Chen Z, Tan B, Long S, Gašić M, Yu K (2019) Agentgraph: Toward Universal Dialogue Management With Structured Deep Reinforcement Learning, In IEEE/ACM Transactions on Audio. Speech Lang Process 27(9):1378–1391

    Google Scholar 

  59. Firdaus M, Golchha H, Ekbal A, Bhattacharyya P (2021) Cognitive Computation (2021) A Deep Multi-task Model for Dialogue Act Classification, Intent Detection and Slot Filling. In: PLOS ONE

  60. Pan B, Yang Y, Li B, Cai D (2021) Self-supervised attention flow for dialogue state tracking. In: Neurocomputing, vol 440, pp 279–286

  61. Campillos-Llanos L, Deleger L, Grouin C, Hamon T, Ligozat A-L, Neveo A (2018) A French clinical corpus with comprehensive semantic annotations: development of the Medical Entity and Relation LIMSI Annotated Text corpus (MERLOT). In: Language resources and evaluation, vol 52

  62. Chen Y, Zhou C, Li T, Wu H, Zhao X, Ye K, Liao J (2019) Named entity recognition from Chinese adverse drug event reports with lexical feature based BiLSTM-CRF and tri-training. In: J Biomed Inform

  63. Patel P, Davey D, Panchal V, Pathak P (2018) Annotation of a large clinical entity corpus. Inproceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp 2033–2042

  64. Wu J, Yılmaz E, Zhang M, Li H, Tan K (2020) Deep Spiking Neural Networks for Large Vocabulary Automatic Speech Recognition. In: Front Neurosci

  65. Dominguez-Morales JP, Liu Q, James R, Gutierrez-Galan D (2018) Deep Spiking Neural Network model for time-variant signals classification: a real-time speech recognition approach, Inproceedings of the International Joint Conference on Neural Networks (IJCNN)

  66. Yang S et al (2019) Real-Time Neuromorphic system for Large-Scale Conductance-Based spiking neural networks. In: IEEE transactions on cybernetics, vol 49, pp 2490–2503

  67. Yang S, Gao T, Wang J, Deng B, Lansdell B, Linares-Barranco B (2021) Efficient Spike-Driven learning with dendritic Event-Based processing. In: Frontiers in Neuroscience, vol 15

  68. Rastogi A, Zang X, Sunkara S, Gupta R, Khaitan P (2020) Towards scalable Multi-Domain conversational agents: the Schema-Guided dialogue dataset. AAAI 34(05):8689–8696

    Article  Google Scholar 

  69. Xu L, Zhou Q, Gong K, Liang X, Tang J, Lin L (2019) End-to-end Knowledge-Routed Relational Dialogue System for Automatic Diagnosis. AAAI 33(01):7346–7353

    Article  Google Scholar 

  70. Zeng G et al (2020) Meddialog: Large-scale Medical Dialogue Datasets. Inproceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pp 9241–9250

  71. Liu W, Tang J, Qin J, Xu L, Li Z, Liang X (2020) MedDG:, A Large-scale Medical Consultation Dataset for Building Medical Dialogue System. arXiv:2010.07497

  72. Li D, Ren Z, Ren P, Chen Z, Fan M, Ma J, de Rijke M (2021) Semi-Supervised Variational reasoning for medical dialogue generation. In: The 44th International ACM SIGIR Conference on Research & Development in Information Retrieval, pp 544–554

Download references

Funding

This work was supported by two projects, Classification of The Artists using Deep Neural Networks, funded by Hanyang University (201600000002255) and Bitstream-based Deep Multimodal Object Detection Framework in Real-time to Extend intelligent CCTV Surveillance (2020R1A2C2013687), funded by the National Research Foundation of Korea(NRF).

Author information

Authors and Affiliations

  1. Hanyang University, 222 Wangsimni-ro, Seongdong-gu, Seoul, 04763, South Korea

    Young-Min Kim, Tae-Hoon Lee & Seon-Ok Na

Authors
  1. Young-Min Kim
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Tae-Hoon Lee
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Seon-Ok Na
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Young-Min Kim.

Ethics declarations

Conflict of Interests

The authors declare that they have no conflict of interest.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Kim, YM., Lee, TH. & Na, SO. Constructing novel datasets for intent detection and ner in a korean healthcare advice system: guidelines and empirical results. Appl Intell 53, 941–961 (2023). https://doi.org/10.1007/s10489-022-03400-y

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10489-022-03400-y

Keywords

Search

Navigation