Skip to main content
SpringerLink
Log in

Consistency regularization for deep semi-supervised clustering with pairwise constraints

  • Original Article
  • Published:
International Journal of Machine Learning and Cybernetics Aims and scope Submit manuscript

Abstract

Due to its powerful learning capabilities for high-dimensional and complex data, deep semi-supervised clustering algorithms often outperform traditional semi-supervised clustering methods. However, most deep semi-supervised clustering methods cannot fully utilize prior knowledge and unlabeled data. Deep semi-supervised classification algorithms have recently made significant progress in using unlabeled data during training by combining a consistency regularization method. Consistency training encourages network predictions to remain consistent when the input is perturbed. Motivated by the success of consistency regularization methods, we proposed a new semi-supervised clustering framework based on Siamese networks. To leverage the additional structure of unlabeled data and to uncover more information hidden by pairwise constraints, we add a consistency regularization loss, calculated on unlabeled data and pairwise constraints, to our objective function. After consistency training, the connected data can be closer in the learned feature space, while the disconnected data can be far away. To verify the effectiveness of the proposed method, we conducted extensive experiments on several real-world data sets. Experimental results show that the proposed method is more effective than other state-of-the-art methods in clustering performance.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7

Similar content being viewed by others

References

  1. Kang Z, Pan H, Hoi SCH, Xu Z (2019) Robust graph learning from noisy data. IEEE Trans Cybern 50:1833–1843

    Article  Google Scholar 

  2. Chao G (2019) Discriminative k-means laplacian clustering. Neural Process Lett 49:393–405

    Article  Google Scholar 

  3. Kang Z, Peng C, Cheng Q, Liu X, Peng X, Xu Z, Tian L (2021) Structured graph learning for clustering and semi-supervised classification. Pattern Recogn 110:107627

    Article  Google Scholar 

  4. Nie F, Zhang H, Wang R, Li X (2020) Semi-supervised clustering via pairwise constrained optimal graph. In: Proceedings of the 29th international joint conference on artificial intelligence (IJCAI), pp 3160–3166

  5. Peng J, Pedersoli M, Desrosiers C (2020) Mutual information deep regularization for semi-supervised segmentation. Med Imaging Deep Learn 20:601–613

    Google Scholar 

  6. Shi Y, Otto C, Jain AK (2018) Face clustering: representation and pairwise constraints. IEEE Trans Inf Forensics Secur 13:1626–1640

    Article  Google Scholar 

  7. Li X, Wu Y, Ester M, Kao B, Wang X, Zheng Y (2017) Semi-supervised clustering in attributed heterogeneous information networks. In: Proceedings of the 26th international conference on world wide web, pp 1621–1629

  8. Wagstaff K, Cardie C (2000) Clustering with instance-level constraints. In: Proceedings of the 17th national conference on artificial intelligence, vol 1097, pp 577–584

  9. Qian P, Jiang Y, Wang S, Su KH, Wang J, Hu L, Muzic RF (2016) Affinity and penalty jointly constrained spectral clustering with all-compatibility, flexibility, and robustness. IEEE Trans Neural Netw Learn Syst 28:1123–1138

    Article  Google Scholar 

  10. Safari S, Afsari F (2020) Ensemble p-spectral semi-supervised clustering. In: Proceedings of the 11th Iranian and the first international conference on machine vision and image processing (MVIP), pp 1–5

  11. Wu W, Jia Y, Kwong S, Hou J (2018) Pairwise constraint propagation-induced symmetric nonnegative matrix factorization. IEEE Trans Neural Netw Learn Syst 29:6348–6361

    Article  MathSciNet  Google Scholar 

  12. Jia Y, Wu W, Wang R, Hou J, Kwong S (2020) Joint optimization for pairwise constraint propagation. IEEE Trans Neural Netw Learn Syst 32:3168–3180

    Article  MathSciNet  Google Scholar 

  13. Liu H, Jia Y, Hou J, Zhang Q (2019) Imbalance-aware pairwise constraint propagation. In: Proceedings of the 27th ACM international conference on multimedia, pp 1605–1613

  14. Jia Y, Liu H, Hou J, Kwong S (2020) Pairwise constraint propagation with dual adversarial manifold regularization. IEEE Trans Neural Netw Learn Syst 31:5575–5587

    Article  MathSciNet  Google Scholar 

  15. Xie J, Girshick R, Farhadi A (2016) Unsupervised deep embedding for clustering analysis. In: Proceedings of the 33rd international conference on machine learning (ICML), pp 478–487

  16. Peng Z, Jia Y, Liu H, Hou J, Zhang Q (2021) Maximum entropy subspace clustering network. IEEE Trans Circ Syst Video Technol 20:1–1

    Google Scholar 

  17. Hsu YC, Kira Z (2015) Neural network-based clustering using pairwise constraints. arXiv:1511.06321 (arXiv preprint)

  18. Ren Y, Hu K, Dai X, Pan L, Hoi SC, Xu Z (2019) Semi-supervised deep embedded clustering. Neurocomputing 325:121–130

    Article  Google Scholar 

  19. Śmieja M, Struski L, Figueiredo MA (2020) A classification-based approach to semi-supervised clustering with pairwise constraints. Neural Netw 127:193–203

    Article  Google Scholar 

  20. Ohi AQ, Mridha M, Safir FB, Hamid MA, Monowar MM (2020) Autoembedder: a semi-supervised DNN embedding system for clustering. Knowl-Based Syst 204:106190

    Article  Google Scholar 

  21. Verma V, Kawaguchi K, Lamb A, Kannala J, Bengio Y, Lopez-Paz D (2019) Interpolation consistency training for semi-supervised learning. In: Proceedings of the 28th international joint conference on artificial intelligence (IJCAI), pp 3635–3641

  22. Wagstaff K, Cardie C, Rogers S, Schroedl S, et al (2001) Constrained k-means clustering with background knowledge. In: Proceedings of the 18th international conference on machine learning (ICML), pp 277–584

  23. Xiong S, Azimi J, Fern XZ (2013) Active learning of constraints for semi-supervised clustering. IEEE Trans Knowl Data Eng 26:43–54

    Article  Google Scholar 

  24. Smieja M, Myronov O, Tabor J 2018) Semi-supervised discriminative clustering with graph regularization. Knowl Based Syst 151:24–36

    Article  Google Scholar 

  25. Pei Y, Fern XZ, Tjahja TV, Rosales R (2016) Comparing clustering with pairwise and relative constraints: a unified framework. ACM Trans Knowl Discov Data 11:1–26

    Article  Google Scholar 

  26. Shukla A, Cheema GS, Anand S (2020) Semi-supervised clustering with neural networks. In: Proceedings of 2020 IEEE 16th international conference on multimedia big data (BigMM), pp 152–161

  27. Zhang H, Basu S, Davidson I (2019) A framework for deep constrained clustering-algorithms and advances. In: Proceedings of the joint European conference on machine learning and knowledge discovery in databases (ECML PKDD), pp 27–72

  28. Pezeshki M, Fan L, Brakel P, Courville A, Bengio Y (2016) Deconstructing the ladder network architecture. In: Proceedings of the 33rd international conference on machine learning (ICML), pp 2368–2376

  29. Laine S, Aila T (2016) Temporal ensembling for semi-supervised learning. arXiv:1610.02242 (arXiv preprint)

  30. Miyato T, Si Maeda, Koyama M, Ishii S (2018) Virtual adversarial training: a regularization method for supervised and semi-supervised learning. IEEE Trans Pattern Anal Mach Intell 41:1979–1993

    Article  Google Scholar 

  31. Xie Q, Dai Z, Hovy E, Luong MT (2020) Unsupervised data augmentation for consistency training. In: Proceedings of the 34th conference on neural information processing systems (NeurIPS), vol 33, pp 6256–6268

  32. Luo Y, Zhu J, Li M, Ren Y, Zhang B (2018) Smooth neighbors on teacher graphs for semi-supervised learning. In: Proceedings of 2018 IEEE/CVF conference on computer vision and pattern recognition (CVPR), pp 8896–8905

  33. Bromley J, Bentz JW, Bottou L, Guyon I, LeCun Y, Moore C, Säckinger E, Shah R (1993) Signature verification using a “siamese’’ time delay neural network. Int J Pattern Recogn Artif Intell 7:669–688

    Article  Google Scholar 

  34. Mueller J, Thyagarajan A (2016) Siamese recurrent architectures for learning sentence similarity. In: Proceedings of the 16th AAAI conference on artificial intelligence, vol 30, pp 2786–2792

  35. Chopra S, Hadsell R, LeCun Y (2005) Learning a similarity metric discriminatively, with application to face verification. In: Proceedings of the 2005 IEEE computer society conference on computer vision and pattern recognition (CVPR), vol 1, pp 539–546

  36. Bertinetto L, Valmadre J, Henriques JF, Vedaldi A (2016) Torr PHS, Fully-convolutional siamese networks for object tracking. In: Proceedings of the 14th European conference on computer vision workshop, pp 850–865

  37. Chen K, Salman A (2011) Extracting speaker-specific information with a regularized siamese deep network. Adv Neural Inf Process Syst 2011:298–306

    Google Scholar 

  38. Nandy A, Haldar S, Banerjee S, Mitra S (2020) A survey on applications of siamese neural networks in computer vision. In: Proceedings of 2020 international conference for emerging technology (INCET), pp 1–5

  39. Chicco D (2021) Siamese neural networks: an overview. Artif Neural Netw 20:79–94

    Google Scholar 

  40. LeCun Y, Bottou L, Bengio Y, Haffner P (1998) Gradient-based learning applied to document recognition. Proc IEEE 86:2278–2324

    Article  Google Scholar 

  41. Xiao H, Rasul K, Vollgraf R (2017) Fashion-Mnist: a novel image dataset for benchmarking machine learning algorithms. arXiv:1708.07747(arXiv preprint)

  42. Krizhevsky A, Hinton G, et al (2009) Learning multiple layers of features from tiny images. Technical report

  43. Hull JJ (1994) A database for handwritten text recognition research. IEEE Trans Pattern Anal Mach Intell 16:550–554

    Article  Google Scholar 

  44. Strehl A, Ghosh J (2002) Cluster ensembles-a knowledge reuse framework for combining multiple partitions. J Mach Learn Res 3:583–617

    MathSciNet  MATH  Google Scholar 

  45. Hubert L, Arabie P (1985) Comparing partitions. J Classif 2:293–218

    Article  Google Scholar 

  46. MacQueen J, et al (1967) Some methods for classification and analysis of multivariate observations. In: Proceedings of the 5th Berkeley symposium on mathematical statistics and probability, pp 281–297

  47. Kang Z, Lin Z, Zhu X, Xu W (2021) Structured graph learning for scalable subspace clustering: from single view to multiview. IEEE Trans Cybern 20:1–11

    Google Scholar 

  48. Lv J, Kang Z, Lu X, Xu Z (2021) Pseudo-supervised deep subspace clustering. IEEE Trans Image Process 30:5252–5263

    Article  Google Scholar 

  49. Guo X, Zhu E, Liu X, Yin J (2018) Deep embedded clustering with data augmentation. In: Proceedings of the 10th Asian conference on machine learning, pp 550–565

Download references

Acknowledgements

This work is supported by the National Natural Science Foundation of China (no. 61976182) and Sichuan Key R &D project (nos. 2020YFG0035, 2021YFG0312).

Author information

Authors and Affiliations

  1. Institute of Artificial Intelligence, Southwest Jiaotong University, Chengdu, 611756, China

    Dan Huang, Jie Hu, Tianrui Li, Shengdong Du & Hongmei Chen

Authors
  1. Dan Huang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Jie Hu
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Tianrui Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Shengdong Du
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Hongmei Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Jie Hu.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Huang, D., Hu, J., Li, T. et al. Consistency regularization for deep semi-supervised clustering with pairwise constraints. Int. J. Mach. Learn. & Cyber. 13, 3359–3372 (2022). https://doi.org/10.1007/s13042-022-01599-3

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s13042-022-01599-3

Keywords

Search

Navigation