Abstract
Recently, many studies have combined contrastive learning with clustering to address this issue and achieved excellent clustering results. However, traditional contrastive learning methods suffer from class conflict. We propose a new framework called Matrix Contrastive Learning (MCL) for text clustering to address this issue. Firstly, data augmentation techniques are utilized to generate pairs of positive and negative instances for all anchor examples. These pairs are mapped into a feature space, where the rows of the matrix represent soft labels for individual instances, and the columns represent cluster representations. We perform contrastive learning at both the instance and cluster levels using these rows and columns. To further improve the cluster allocation in unsupervised clustering tasks and alleviate the class conflict problem caused by instance-level contrastive learning in unsupervised conditions, the K-Nearest Neighbors algorithm is used to filter out negative instances. We conducted extensive experiments on eight challenging text datasets and compared MCL with six existing clustering methods. The results show that MCL significantly outperforms the competing methods. The code is available at https://github.com/2251821381/MCL.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Abati, D., Tomczak, J., Blankevoort, T., et al.: Conditional channel gated networks for task-aware continual learning. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3931–3940 (2020)
Boyd, D.M., Ellison, N.B.: Social network sites: definition, history, and scholarship. J. Comput.-Mediat. Commun. 13(1), 210–230 (2007)
Chen, T., Kornblith, S., Norouzi, M., et al.: A simple framework for contrastive learning of visual representations. In: International Conference on Machine Learning, pp. 1597–1607. PMLR (2020)
Dang, Z., Deng, C., Yang, X., et al.: Doubly contrastive deep clustering. arXiv preprint arXiv:2103.05484 (2021)
Gao, T., Yao, X., Chen, D.: SimCSE: simple contrastive learning of sentence embeddings. In: 2021 Conference on Empirical Methods in Natural Language Processing, EMNLP 2021, pp. 6894–6910. Association for Computational Linguistics (ACL) (2021)
Hadifar, A., Sterckx, L., Demeester, T., et al.: A self-training approach for short text clustering. In: Proceedings of the 4th Workshop on Representation Learning for NLP (RepL4NLP-2019), pp. 194–199 (2019)
Hu, W., Miyato, T., Tokui, S., et al.: Learning discrete representations via information maximizing self-augmented training. In: International Conference on Machine Learning, pp. 1558–1567. PMLR (2017)
Huang, Z., Chen, J., Zhang, J., et al.: Learning representation for clustering via prototype scattering and positive sampling. IEEE Trans. Pattern Anal. Mach. Intell. 45(6), 7509–7524 (2022)
Ji, X., Henriques, J.F., Vedaldi, A.: Invariant information clustering for unsupervised image classification and segmentation. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 9865–9874 (2019)
Khosla, P., Teterwak, P., Wang, C., et al.: Supervised contrastive learning. Adv. Neural. Inf. Process. Syst. 33, 18661–18673 (2020)
Li, C., Yu, X., Song, S., et al.: Simctc: A simple contrast learning method of text clustering (student abstract). In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 36, pp. 12997–12998 (2022)
Li, J., Zhou, P., Xiong, C., Hoi, S.: Prototypical contrastive learning of unsupervised representations. In: International Conference on Learning Representations (2020)
Li, R., Wang, H.: Clustering of short texts based on dynamic adjustment for contrastive learning. IEEE Access 10, 76069–76078 (2022)
Ma, Y., Zhang, X., Gao, C., et al.: Enhancing recommendations with contrastive learning from collaborative knowledge graph. Neurocomputing 523, 103–115 (2023)
Van der Maaten, L., Hinton, G.: Visualizing data using t-SNE. J. Mach. Learn. Res. 9, 2579–2605 (2008)
Mikolov, T., Chen, K., Corrado, G.S., et al.: Efficient estimation of word representations in vector space (2013)
Moukafih, Y., Sbihi, N., Ghogho, M., et al.: SuperConText: supervised contrastive learning framework for textual representations. IEEE Access 11, 16820–16830 (2023)
Niu, C., Shan, H., Wang, G.: Spice: Semantic pseudo-labeling for image clustering. IEEE Trans. Image Process. 31, 7264–7278 (2022)
Phan, X.H., Nguyen, L.M., Horiguchi, S.: Learning to classify short and sparse text & web with hidden topics from large-scale data collections. In: Proceedings of the 17th International Conference on World Wide Web, pp. 91–100 (2008)
Rakib, M.R.H., Zeh, N., Jankowska, M., Milios, E.: Enhancement of short text clustering by iterative classification. In: Métais, E., Meziane, F., Horacek, H., Cimiano, P. (eds.) NLDB 2020. LNCS, vol. 12089, pp. 105–117. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-51310-8_10
Sanh, V., Debut, L., Chaumond, J., et al.: Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter (2019)
Sarzynska-Wawer, J., Wawer, A., Pawlak, A., et al.: Detecting formal thought disorder by deep contextualized word representations. Psychiatry Res. 304, 114135 (2021)
Sun, K., Yao, T., Chen, S., et al.: Dual contrastive learning for general face forgery detection. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 36, pp. 2316–2324 (2022)
Tejankar, A., Koohpayegani, S.A., Pillai, V., et al.: ISD: self-supervised learning by iterative similarity distillation. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 9609–9618 (2021)
Tian, R., Shi, H.: Momentum memory contrastive learning for transfer-based few-shot classification. Appl. Intell. 53(1), 864–878 (2023)
Xie, J., Girshick, R., Farhadi, A.: Unsupervised deep embedding for clustering analysis. In: International Conference on Machine Learning, pp. 478–487. PMLR (2016)
Xu, J., Xu, B., Wang, P., et al.: Self-taught convolutional neural networks for short text clustering. Neural Netw. 88, 22–31 (2017)
Zhang, D., Nan, F., Wei, X., et al.: Supporting clustering with contrastive learning. In: Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 5419–5430 (2021)
Zhang, Y., Zhang, H., Zhan, L.M., Wu, X.M., Lam, A.: New intent discovery with pre-training and contrastive learning. In: Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 256–269 (2022)
Acknowledgement
This work was supported by the Natural Science Foundation of China under Grants No. 61966038 and No. 62266051.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2024 The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd.
About this paper
Cite this paper
Zhu, Z., Li, J., Zhang, X., Wang, J., Zhou, X. (2024). Matrix Contrastive Learning for Short Text Clustering. In: Luo, B., Cheng, L., Wu, ZG., Li, H., Li, C. (eds) Neural Information Processing. ICONIP 2023. Communications in Computer and Information Science, vol 1961. Springer, Singapore. https://doi.org/10.1007/978-981-99-8126-7_42
Download citation
DOI: https://doi.org/10.1007/978-981-99-8126-7_42
Published:
Publisher Name: Springer, Singapore
Print ISBN: 978-981-99-8125-0
Online ISBN: 978-981-99-8126-7
eBook Packages: Computer ScienceComputer Science (R0)