Abstract
New Intent Discovery (NID) strives to identify known and reasonably deduce novel intent groups in the open-world scenario. But current methods face issues with inaccurate pseudo-labels and poor representation learning, creating a negative feedback loop that degrades overall model performance, including accuracy and the adjusted rand index. To address the aforementioned challenges, we propose a Robust New Intent Discovery (RoNID) framework optimized by an EM-style method, which focuses on constructing reliable pseudo-labels and obtaining cluster-friendly discriminative representations. RoNID comprises two main modules: reliable pseudo-label generation module and cluster-friendly representation learning module. Specifically, the pseudo-label generation module assigns reliable synthetic labels by solving an optimal transport problem in the E-step, which effectively provides high-quality supervised signals for the input of the cluster-friendly representation learning module. To learn cluster-friendly representation with strong intra-cluster compactness and large inter-cluster separation, the representation learning module combines intra-cluster and inter-cluster contrastive learning in the M-step to feed more discriminative features into the generation module. RoNID can be performed iteratively to ultimately yield a robust model with reliable pseudo-labels and cluster-friendly representations. Experimental results on multiple benchmarks demonstrate our method brings substantial improvements over previous state-of-the-art methods by a large margin of +1\(\sim \)+4 points.
S. Zhang and C. Yan—Equal contribution.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
Notes
- 1.
We estimate the number of K based on prior works [42] to ensure a fair comparison.
- 2.
- 3.
References
An, W., Tian, F., Zheng, Q., Ding, W., Wang, Q., Chen, P.: Generalized category discovery with decoupled prototypical network. In: Proceedings of AAAI (2023)
Asano, Y.M., Rupprecht, C., Vedaldi, A.: Self-labelling via simultaneous clustering and representation learning. In: Proceedings of ICLR (2020)
Caron, M., Bojanowski, P., Joulin, A., Douze, M.: Deep clustering for unsupervised learning of visual features. In: Proceedings of ECCV (2018)
Caron, M., Misra, I., Mairal, J., Goyal, P., Bojanowski, P., Joulin, A.: Unsupervised learning of visual features by contrasting cluster assignments. In: Proceedings of NeurIPS (2020)
Casanueva, I., Temčinas, T., Gerz, D., Henderson, M., Vulić, I.: Efficient intent detection with dual sentence encoders. In: Proceedings of the 2nd Workshop on Natural Language Processing for Conversational AI (2020)
Chang, J., Wang, L., Meng, G., Xiang, S., Pan, C.: Deep adaptive image clustering. In: 2017 IEEE International Conference on Computer Vision (ICCV) (2017)
Chen, Q., Zhuo, Z., Wang, W.: Bert for joint intent classification and slot filling. arXiv preprint arXiv:1902.10909 (2019)
Chen, T., Kornblith, S., Norouzi, M., Hinton, G.E.: A simple framework for contrastive learning of visual representations. In: Proceedings of ICML (2020)
Cuturi, M.: Sinkhorn distances: lightspeed computation of optimal transport. In: Proceedings of NeurIPS (2013)
Devlin, J., Chang, M.W., Lee, K., Toutanova, K.: Bert: pre-training of deep bidirectional transformers for language understanding. In: Proceedings of AACL (2019)
Gowda, K.C., Krishna, G.: Agglomerative clustering using the concept of mutual nearest neighbourhood. Pattern recognition (1978)
Gulrajani, I., Ahmed, F., Arjovsky, M., Dumoulin, V., Courville, A.C.: Improved training of wasserstein GANs. In: Proceedings of NeurIPS (2017)
Hsu, Y.C., Lv, Z., Kira, Z.: Learning to cluster in order to transfer across domains and tasks. arXiv preprint arXiv:1711.10125 (2017)
Khosla, P., et al.: Supervised contrastive learning. In: Proceedings of NeurIPS (2020)
Kumar, R., Patidar, M., Varshney, V., Vig, L., Shroff, G.: Intent detection and discovery from user logs via deep semi-supervised contrastive clustering. In: Proceedings of NAACL (2022)
Larson, S., et al.: An evaluation dataset for intent classification and out-of-scope prediction. arXiv preprint arXiv:1909.02027 (2019)
Li, J., Zhou, P., Xiong, C., Hoi, S.C.H.: Prototypical contrastive learning of unsupervised representations. In: Proceedings of ICLR (2021)
Lin, T.E., Xu, H., Zhang, H.: Discovering new intents via constrained deep adaptive clustering with cluster refinement. In: Proceedings of AAAI (2020)
Liu, B., Mazumder, S.: Lifelong and continual learning dialogue systems: learning during conversation. In: Proceedings of AAAI (2021)
MacQueen, J., et al.: Some methods for classification and analysis of multivariate observations. In: Proceedings of the fifth Berkeley Symposium on Mathematical Statistics and Probability (1967)
Min, Q., Qin, L., Teng, Z., Liu, X., Zhang, Y.: Dialogue state induction using neural latent variable models. arXiv preprint arXiv:2008.05666 (2020)
Mo, Y., et al.: MCL-NER: cross-lingual named entity recognition via multi-view contrastive learning. In: Proceedings of AAAI (2024)
Padmasundari, S.B.: Intent discovery through unsupervised semantic text clustering. Proc. Interspeech 2018 (2018)
Peyré, G., Cuturi, M., et al.: Computational optimal transport: With applications to data science. Foundations and Trends® in Machine Learning (2019)
Shi, W., An, W., Tian, F., Zheng, Q., Wang, Q., Chen, P.: A diffusion weighted graph framework for new intent discovery. arXiv preprint arXiv:2310.15836 (2023)
Taherkhani, F., Dabouei, A., Soleymani, S., Dawson, J.M., Nasrabadi, N.M.: Transporting labels via hierarchical optimal transport for semi-supervised learning. In: Proc. of ECCV (2020)
Tai, K.S., Bailis, P., Valiant, G.: Sinkhorn label allocation: Semi-supervised classification via annealed self-training. In: Proc. of ICML (2021)
Vaze, S., Han, K., Vedaldi, A., Zisserman, A.: Generalized category discovery. In: Proc. of CVPR (2022)
Wang, Z., Chai, L., Yang, J., Bai, J., Yin, Y., Liu, J., Guo, H., Li, T., Yang, L., Li, Z., et al.: Mt4crossoie: Multi-stage tuning for cross-lingual open information extraction. arXiv preprint arXiv:2308.06552 (2023)
Wu, C.S., Madotto, A., Hosseini-Asl, E., Xiong, C., Socher, R., Fung, P.: Transferable multi-domain state generator for task-oriented dialogue systems. In: Proc. of ACL (2019)
Xie, J., Girshick, R., Farhadi, A.: Unsupervised deep embedding for clustering analysis. In: Proc. of ICML (2016)
Xu, J., Wang, P., Tian, G., Xu, B., Zhao, J., Wang, F., Hao, H.: Short text clustering via convolutional neural networks. In: Proceedings of the 1st Workshop on Vector Space Modeling for Natural Language Processing (2015)
Yang, B., Fu, X., Sidiropoulos, N.D., Hong, M.: Towards k-means-friendly spaces: Simultaneous deep learning and clustering. In: Proc. of ICML (2017)
Yang, J., Guo, H., Yin, Y., Bai, J., Wang, B., Liu, J., Liang, X., Cahi, L., Yang, L., Li, Z.: m3p: Towards multimodal multilingual translation with multimodal prompt. arXiv preprint arXiv:2403.17556 (2024)
Yang, J., Huang, S., Ma, S., Yin, Y., Dong, L., Zhang, D., Guo, H., Li, Z., Wei, F.: CROP: zero-shot cross-lingual named entity recognition with multilingual labeled sequence translation. In: Proc. of EMNLP Findings (2022)
Yang, J., Ma, S., Dong, L., Huang, S., Huang, H., Yin, Y., Zhang, D., Yang, L., Wei, F., Li, Z.: Ganlm: Encoder-decoder pre-training with an auxiliary discriminator. In: Proc. of ACL (2023)
Yang, J., Ma, S., Huang, H., Zhang, D., Dong, L., Huang, S., Muzio, A., Singhal, S., Hassan, H., Song, X., Wei, F.: Multilingual machine translation systems from microsoft for WMT21 shared task. In: Proceedings of the Sixth Conference on Machine Translation, WMT@EMNLP 2021, Online Event, November 10-11, 2021 (2021)
Yang, J., Ma, S., Zhang, D., Wu, S., Li, Z., Zhou, M.: Alternating language modeling for cross-lingual pre-training. In: Proc. of AAAI (2020)
Yang, J., Yin, Y., Ma, S., Huang, H., Zhang, D., Li, Z., Wei, F.: Multilingual agreement for multilingual neural machine translation. In: Proc. of ACL (2021)
Yang, J., Yin, Y., Ma, S., Zhang, D., Wu, S., Guo, H., Li, Z., Wei, F.: UM4: unified multilingual multiple teacher-student model for zero-resource neural machine translation. In: Proc. of IJCAI (2022)
Zhang, C., Xu, R., He, X.: Novel class discovery for long-tailed recognition. CoRR (2023)
Zhang, H., Xu, H., Lin, T.E., Lyu, R.: Discovering new intents with deep aligned clustering. In: Proc. of AAAI (2021)
Zhang, H., Xu, H., Wang, X., Long, F., Gao, K.: USNID: A framework for unsupervised and semi-supervised new intent discovery. CoRR (2023)
Zhang, S., Bai, J., Li, T., Yan, Z., Li, Z.: Modeling intra-class and inter-class constraints for out-of-domain detection. In: Proc. of DASFAA (2023)
Zhang, S., Li, T., Bai, J., Li, Z.: Label-guided contrastive learning for out-of-domain detection. In: Proc. of ICASSP (2023)
Zhang, S., Yang, J., Bai, J., Yan, C., Li, T., Yan, Z., Li, Z.: New intent discovery with attracting and dispersing prototype. arXiv preprint arXiv:2403.16913 (2024)
Zhang, Y., Zhang, H., Zhan, L.M., Wu, X.M., Lam, A.: New intent discovery with pre-training and contrastive learning. In: Proc. of ACL (2022)
Zhou, Y., Quan, G., Qiu, X.: A probabilistic framework for discovering new intents. In: Proc. of ACL (2023)
Acknowledgement
This work was supported in part by the National Natural Science Foundation of China (Grant Nos. U1636211, U2333205, 61672081, 62302025, 62276017), a fund project: State Grid Co., Ltd. Technology R&D Project (ProjectName: Research on Key Technologies of Data Scenario-based Security Governance and Emergency Blocking in Power Monitoring System, Proiect No.: 5108-202303439A-3-2-ZN), the 2022 CCF-NSFOCUS Kun-Peng Scientific Research Fund and the Opening Project of Shanghai Trusted Industrial Control Platform and the State Key Laboratory of Complex & Critical Software
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2024 The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd.
About this paper
Cite this paper
Zhang, S. et al. (2024). RoNID: New Intent Discovery with Generated-Reliable Labels and Cluster-friendly Representations. In: Onizuka, M., et al. Database Systems for Advanced Applications. DASFAA 2024. Lecture Notes in Computer Science, vol 14854. Springer, Singapore. https://doi.org/10.1007/978-981-97-5569-1_7
Download citation
DOI: https://doi.org/10.1007/978-981-97-5569-1_7
Published:
Publisher Name: Springer, Singapore
Print ISBN: 978-981-97-5568-4
Online ISBN: 978-981-97-5569-1
eBook Packages: Computer ScienceComputer Science (R0)