Abstract
Slot filling and intent detection are crucial tasks of Spoken Language Understanding (SLU). However, most existing joint models establish shallow connections between intent and slot by sharing parameters, which cannot fully utilize their rich interaction information. Meanwhile, the character and word fusion methods used in the Chinese SLU simply combines the initial information without appropriate guidance, making it easy to introduce a large amount of noisy information. In this paper, we propose a deep joint model of Multi-Scale intent-slots Interaction with Second-Order Gate for Chinese SLU (MSIM-SOG). The model consists of two main modules: (1) the Multi-Scale intent-slots Interaction Module (MSIM), which enables cyclic updating the multi-scale information to achieve deep bi-directional interaction of intent and slots; (2) the Second-Order Gate Module (SOG), which controls the propagation of valuable information through the gate with second-order weights, reduces the noise information of fusion, accelerates model convergence, and alleviates model overfitting. Experiments on two public datasets demonstrate that our model outperforms the baseline and achieves state-of-the-art performance compared to previous models.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Goo, C.W., Gao, G., Hsu, Y.K., Huo, C.L., Chen, T.C.: Slot-gated modeling for joint slot filling and intent prediction. In: NAACL, pp. 753–757 (2018)
Haffner, P., Tür, G., Wright, J.H.: Optimizing SVMs for complex call classification. In: ICASSP (2003)
Hochreiter, S., Schmidhuber, J.: Long short-term memory. Neural Comput. 9, 1735–1780 (1997)
Kim, S., D’Haro, L.F., Banchs, R.E., Williams, J.D., Henderson, M.: The fourth dialog state tracking challenge. In: Jokinen, K., Wilcock, G. (eds.) Dialogues with Social Robots. LNEE, vol. 999, pp. 435–449. Springer, Singapore (2017). https://doi.org/10.1007/978-981-10-2585-3_36
Kingma, D.P., Ba, J.: Adam: a method for stochastic optimization. CoRR, pp. 1–11 (2014)
Lai, S., Xu, L., Liu, K., Zhao, J.: Recurrent convolutional neural networks for text classification. In: AAAI 2015, pp. 2267–2273 (2015)
Li, C., Zhou, Y., Chao, G., Chu, D.: Understanding users’ requirements precisely: a double bi-LSTM-CRF joint model for detecting user’s intentions and slot tags. Neural Comput. Appl. 34, 13639–13648 (2022)
Liu, L., et al.: On the variance of the adaptive learning rate and beyond. ArXiv, pp. 1–13 (2019)
Liu, Y., Meng, F., Zhang, J., Zhou, J.: CM-net: a novel collaborative memory network for spoken language understanding. In: EMNLP-ICJNLP, pp. 1051–1060
Ma, Z., Sun, B., Li, S.: A two-stage selective fusion framework for joint intent detection and slot filling. IEEE Trans. Neural Netw. Learn. 1–12 (2022)
Ni, P., Li, Y., Li, G., Chang, V.I.: Natural language understanding approaches based on joint task of intent detection and slot filling for IoT voice interaction. Neural Comput. Appl. 1–18 (2020)
Qin, L., Che, W., Li, Y., Wen, H., Liu, T.: A stack-propagation framework with token-level intent detection for spoken language understanding. In: EMNLP-IJCNLP, pp. 2078–2087 (2019)
Sun, C., Lv, L., Liu, T., Li, T.: A joint model based on interactive gate mechanism for spoken language understanding. Appl. Intell. 52, 6057–6064 (2021)
Tang, H., Ji, D.H., Zhou, Q.: End-to-end masked graph-based CRF for joint slot filling and intent detection. Neurocomputing 413, 348–359 (2020)
Teng, D., Qin, L., Che, W., Liu, T.: Injecting word information with multi-level word adapter for Chinese spoken language understanding. In: ICASSP, pp. 8188–8192 (2021)
Wei, P., Zeng, B., Liao, W.: Joint intent detection and slot filling with wheel-graph attention networks. J. Intell. Fuzzy Syst. 42, 2409–2420 (2021)
Weld, H., Huang, X., Long, S., Poon, J., Han, S.C.: A survey of joint intent detection and slot filling models in natural language understanding. ACM Comput. Surv. 55, 1–38 (2021)
Xu, C., Li, Q., Zhang, D., Cui, J.: A model with length-variable attention for spoken language understanding. Neurocomputing 379, 197–202 (2020)
Xu, P., Sarikaya, R.: Convolutional neural network based triangular CRF for joint intent detection and slot filling. In: 2013 IEEE Workshop on Automatic Speech Recognition and Understanding, pp. 78–83 (2013)
Yao, K., Peng, B., Zhang, Y., Yu, D., Zweig, G.: Spoken language understanding using long short-term memory neural networks. In: IEEE-SLT, pp. 189–194 (2014)
Zhu, Z., Huang, P., Huang, H., Liu, S., Lao, L.: A graph attention interactive refine framework with contextual regularization for jointing intent detection and slot filling. In: ICASSP, pp. 7617–7621 (2022)
Acknowledgements
This work was supported in part by the National Science Foundation of China under Grant 62172111, in part by the Natural Science Foundation of Guangdong Province under Grant 2019A1515011056, in part by the Key technology project of Shunde District under Grant 2130218003002.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2024 The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd.
About this paper
Cite this paper
Wen, Q., Zeng, B., Wei, P., Hu, H. (2024). A Deep Joint Model of Multi-scale Intent-Slots Interaction with Second-Order Gate for SLU. In: Luo, B., Cheng, L., Wu, ZG., Li, H., Li, C. (eds) Neural Information Processing. ICONIP 2023. Communications in Computer and Information Science, vol 1966. Springer, Singapore. https://doi.org/10.1007/978-981-99-8148-9_4
Download citation
DOI: https://doi.org/10.1007/978-981-99-8148-9_4
Published:
Publisher Name: Springer, Singapore
Print ISBN: 978-981-99-8147-2
Online ISBN: 978-981-99-8148-9
eBook Packages: Computer ScienceComputer Science (R0)