skip to main content
research-article

MICCF: A Mutual Information Constrained Clustering Framework for Learning Clustering-Oriented Feature Representations

Published: 16 August 2024 Publication History

Abstract

Deep clustering is a crucial task in machine learning and data mining that focuses on acquiring feature representations conducive to clustering. Previous research relies on self-supervised representation learning for general feature representations, such features may not be optimally suited for downstream clustering tasks. In this article, we introduce MICCF, a framework designed to bridge this gap and enhance clustering performance. MICCF enhances feature representations by combining mutual information constraints at different levels and employs an auxiliary alignment mutual information module for learning clustering-oriented features. To be specific, we propose a dual mutual information constraints module, incorporating minimal mutual information constraints at the feature level and maximal mutual information constraints at the instance level. This reduction in feature redundancy encourages the neural network to extract more discriminative features, while maximization ensures more unbiased and robust representations. To obtain clustering-oriented representations, the auxiliary alignment mutual information module utilizes pseudo-labels to maximize mutual information through a multi-classifier network, aligning features with the clustering task. The main network and the auxiliary module work in synergy to jointly optimize feature representations that are well-suited for the clustering task. We validate the effectiveness of our method through extensive experiments on six benchmark datasets. The results indicate that our method performs well in most scenarios, particularly on fine-grained datasets, where our approach effectively distinguishes subtle differences between closely related categories. Notably, our approach achieved a remarkable accuracy of 96.4% on the ImageNet-10 dataset, surpassing other comparison methods. The code is available at https://github.com/Li-Hyn/MICCF.git.

References

[1]
Jinyu Cai, Jicong Fan, Wenzhong Guo, Shiping Wang, Yunhe Zhang, and Zhao Zhang. 2022. Efficient Deep Embedded Subspace Clustering. In CVPR. 1–10.
[2]
Mathilde Caron, Piotr Bojanowski, Armand Joulin, and Matthijs Douze. 2018. Deep Clustering for Unsupervised Learning of Visual Features. In ECCV, Vol. 11218. 139–156.
[3]
Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. 2020. Unsupervised Learning of Visual Features by Contrasting Cluster Assignments. In NeurIPS. 9912–9924.
[4]
Jianlong Chang, Lingfeng Wang, Gaofeng Meng, Shiming Xiang, and Chunhong Pan. 2017. Deep Adaptive Image Clustering. In ICCV. 5880–5888.
[5]
Mansheng Chen, Ling Huang, Chang-Dong Wang, Dong Huang, and Philip S. Yu. 2022. Multiview Subspace Clustering with Grouping Effect. IEEE Trans. Cybern. 52, 8 (2022), 7655–7668.
[6]
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. 2020a. A Simple Framework for Contrastive Learning of Visual Representations. In ICML. 1597–1607.
[7]
Adam Coates, Andrew Y. Ng, and Honglak Lee. 2011. An Analysis of Single-Layer Networks in Unsupervised Feature Learning. In AISTATS, Vol. 15. 215–223.
[8]
Zhiyuan Dang, Cheng Deng, Xu Yang, and Heng Huang. 2021. Doubly Contrastive Deep Clustering. arXiv:2103.05484. Retrieved from https://arxiv.org/abs/2103.05484
[9]
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. 2009. ImageNet: A Large-Scale Hierarchical Image Database. In CVPR. 248–255.
[10]
Kien Do, Truyen Tran, and Svetha Venkatesh. 2021. Clustering by Maximizing Mutual Information Across Views. In ICCV. 9908–9918.
[11]
Carl Doersch, Abhinav Gupta, and Alexei A. Efros. 2015. Unsupervised Visual Representation Learning by Context Prediction. In ICCV. 1422–1430.
[12]
Jeff Donahue, Philipp Krähenbühl, and Trevor Darrell. 2017. Adversarial Feature Learning. In ICLR. 1–18.
[13]
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross B. Girshick. 2020. Momentum Contrast for Unsupervised Visual Representation Learning. In CVPR. 9726–9735.
[14]
Geoffrey E. Hinton, Oriol Vinyals, and Jeffrey Dean. 2015. Distilling the Knowledge in a Neural Network. arXiv:1503.02531. Retrieved from https://arxiv.org/abs/1503.02531
[15]
Weihua Hu, Takeru Miyato, Seiya Tokui, Eiichi Matsumoto, and Masashi Sugiyama. 2017. Learning Discrete Representations via Information Maximizing Self-Augmented Training. In ICML, Vol. 70. 1558–1567.
[16]
Jiabo Huang, Shaogang Gong, and Xiatian Zhu. 2020. Deep Semantic Clustering by Partition Confidence Maximisation. In CVPR. 8849–8858.
[17]
Lawrence Hubert and Phipps Arabie. 1985. Comparing Partitions. J Classif. 2, 1 (1985), 193–218.
[18]
Xu Ji, Andrea Vedaldi, and João F. Henriques. 2019. Invariant Information Clustering for Unsupervised Image Classification and Segmentation. In ICCV. 9865–9874.
[19]
Barakeel Fanseu Kamhoua, Lin Zhang, Kaili Ma, James Cheng, Bo Li, and Bo Han. 2023. GRACE: A General Graph Convolution Framework for Attributed Graph Clustering. ACM Trans. Knowl. Discov. Data 17, 3 (2023), 31:1–31.
[20]
Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan. 2020. Supervised Contrastive Learning. In NeurIPS. 18661–18673.
[21]
Diederik P. Kingma and Max Welling. 2013. Auto-Encoding Variational Bayes. Retrieved from https://doi.org/10.48550/arXiv.1312.6114 arXiv:1312.6114.
[22]
Lingpeng Kong, Cyprien de Masson d’Autume, Lei Yu, Wang Ling, Zihang Dai, and Dani Yogatama. 2020. A Mutual Information Maximization Perspective of Language Representation Learning. In ICLR. 1–12.
[23]
Alex Krizhevsky and Geoffrey Hinton. 2009. Learning Multiple Layers of Features From Tiny Images. Technical Report. CIFAR.
[24]
H. Li, L. Zhang, and K. Su. 2023. Dual Mutual Information Constraints for Discriminative Clustering. In AAAI. 2539–2545.
[25]
Tao Li and Chris Ding. 2006. The Relationships Among Various Nonnegative Matrix Factorization Methods for Clustering. In ICDM. 362–371.
[26]
Yunfan Li, Peng Hu, Jerry Zitao Liu, Dezhong Peng, Joey Tianyi Zhou, and Xi Peng. 2021. Contrastive Clustering. In AAAI. 8547–8555.
[27]
Jiaqi Ma, Yipeng Zhang, Lefei Zhang, Bo Du, and Dapeng Tao. 2019. Pseudo Supervised Matrix Factorization in Discriminative Subspace. In IJCAI. 4554–4560.
[28]
Sihan Ma, Lefei Zhang, Wenbin Hu, Yipeng Zhang, Jia Wu, and Xuelong Li. 2018. Self-Representative Manifold Concept Factorization with Adaptive Neighbors for Clustering. In IJCAI. 2539–2545.
[29]
J. MacQueen. 1967. Some Methods for Classification and Analysis of Multivariate Observations. In BSMSP, Vol. 1. 281–297.
[30]
Ioannis Maniadis Metaxas, Georgios Tzimiropoulos, and Ioannis Patras. 2023. DivClust: Controlling Diversity in Deep Clustering. In CVPR. 3418–3428.
[31]
Feiping Nie, Xiaoqian Wang, Michael I. Jordan, and Heng Huang. 2016. The Constrained Laplacian Rank Algorithm for Graph-Based Clustering. In AAAI. 1969–1976.
[32]
Mehdi Noroozi and Paolo Favaro. 2016. Unsupervised Learning of Visual Representations by Solving Jigsaw Puzzles. In ECCV. 69–84.
[33]
Yunsheng Pang, Feiyu Chen, Sheng Huang, Yongxin Ge, Wei Wang, and Taiping Zhang. 2020. Deep Fuzzy Clustering with Weighted Intra-Class Variance and Extended Mutual Information Regularization. In ICDM. 464–471.
[34]
Deepak Pathak, Ross Girshick, Piotr Dollár, Trevor Darrell, and Bharath Hariharan. 2017. Learning Features by Watching Objects Move. In CVPR. 2701–2710.
[35]
Puloma Roy, Sadia Sharmin, Amin Ahsan Ali, and Mohammad Shoyaib. 2020. Discretization and Feature Selection Based on Bias Corrected Mutual Information Considering High-Order Dependencies. In PAKDD, Vol. 12084. 830–842.
[36]
Krishna Kumar Singh, Utkarsh Ojha, and Yong Jae Lee. 2019. Finegan: Unsupervised Hierarchical Disentanglement for Fine-Grained Object Generation and Discovery. In CVPR. 6490–6499.
[37]
Alexander Strehl and Joydeep Ghosh. 2002. Cluster Ensembles—A Knowledge Reuse Framework for Combining Multiple Partitions. In JMLR. 583–617.
[38]
Yaling Tao, Kentaro Takagi, and Kouta Nakata. 2021. Clustering-Friendly Representation Learning via Instance Discrimination and Feature Decorrelation. In ICLR. 1–15.
[39]
Wouter Van Gansbeke, Simon Vandenhende, Stamatios Georgoulis, Marc Proesmans, and Luc Van Gool. 2020. Scan: Learning to Classify Images Without Labels. In ECCV. 268–285.
[40]
Dexian Wang, Tianrui Li, Ping Deng, Fan Zhang, Wei Huang, Pengfei Zhang, and Jia Liu. 2023. A Generalized Deep Learning Clustering Algorithm Based on Non-Negative Matrix Factorization. ACM Trans. Knowl. Discov. Data 17, 7 (2023), 99:1–99:20.
[41]
Xiang Wang, Liping Jing, Huafeng Liu, and Jian Yu. 2024. Structure-Driven Representation Learning for Deep Clustering. ACM Trans. Knowl. Discov. Data 18, 1 (2024), 31:1–31:25.
[42]
Jie Wen, Zheng Zhang, Zhao Zhang, Lei Zhu, Lunke Fei, Bob Zhang, and Yong Xu. 2021. Unified Tensor Framework for Incomplete Multi-View Clustering and Missing-View Inferring. In AAAI. 10273–10281.
[43]
Jianlong Wu, Keyu Long, Fei Wang, Chen Qian, Cheng Li, Zhouchen Lin, and Hongbin Zha. 2019. Deep Comprehensive Correlation Mining for Image Clustering. In ICCV. 8149–8158.
[44]
Zhirong Wu, Yuanjun Xiong, Stella X. Yu, and Dahua Lin. 2018. Unsupervised Feature Learning via Non-Parametric Instance Discrimination. In CVPR. 3733–3742.
[45]
Yuan Xie, Jinyan Liu, Yanyun Qu, Dacheng Tao, Wensheng Zhang, Longquan Dai, and Lizhuang Ma. 2020. Robust Kernelized Multiview Self-Representation for Subspace Clustering. IEEE Trans. Neural. Netw. Learn. Syst. 32, 2 (2020), 868–881.
[46]
Yuankun Xu, Dong Huang, Chang-Dong Wang, and Jian-Huang Lai. 2024. Deep Image Clustering with Contrastive Learning and Multi-Scale Graph Convolutional Networks. Pattern Recognit. 146, 110065.
[47]
Jianwei Yang, Devi Parikh, and Dhruv Batra. 2016. Joint Unsupervised Learning of Deep Representations and Image Clusters. In CVPR. 5147–5156.
[48]
Jun Yin, Runcheng Cai, and Shiliang Sun. 2022. Anchor-Based Incomplete Multi-View Spectral Clustering. Neurocomputing 514, 526–538.
[49]
Xuanqi Zhang, Qiangqiang Shen, Yongyong Chen, Guokai Zhang, Zhongyun Hua, and Jingyong Su. 2023. Multi-View Ensemble Clustering via Low-rank and Sparse Decomposition: From Matrix to Tensor. ACM Trans. Knowl. Discov. Data 17, 7 (2023), 103:1–103:19.
[50]
Peisen Zhao, Lingxi Xie, Ya Zhang, and Qi Tian. 2020. Universal-to-Specific Framework for Complex Action Recognition. IEEE Trans. Multimed. 23, 3441–3453.
[51]
Yunxiao Zhao and Liang Bai. 2024. Contrastive Clustering with a Graph Consistency Constraint. Pattern Recognit. 146, 110032.
[52]
Huasong Zhong, Chong Chen, Zhongming Jin, and Xian-Sheng Hua. 2020. Deep Robust Clustering by Contrastive Learning. arXiv:2008.03030. Retrieved from https://doi.org/10.48550/arXiv.2008.03030
[53]
Michał Znalezniak, Przemysław Rola, Patryk Kaszuba, Jacek Tabor, and Marek śmieja. 2023. Contrastive Hierarchical Clustering. In ECML-PKDD. 627–643.

Cited By

View all
  • (2024)Explainability for Property Violations in Cyberphysical Systems: An Immune-Inspired ApproachIEEE Software10.1109/MS.2024.338728941:5(43-51)Online publication date: 16-Apr-2024

Index Terms

  1. MICCF: A Mutual Information Constrained Clustering Framework for Learning Clustering-Oriented Feature Representations

    Recommendations

    Comments

    Information & Contributors

    Information

    Published In

    cover image ACM Transactions on Knowledge Discovery from Data
    ACM Transactions on Knowledge Discovery from Data  Volume 18, Issue 8
    September 2024
    700 pages
    EISSN:1556-472X
    DOI:10.1145/3613713
    Issue’s Table of Contents

    Publisher

    Association for Computing Machinery

    New York, NY, United States

    Publication History

    Published: 16 August 2024
    Online AM: 07 July 2024
    Accepted: 30 May 2024
    Revised: 07 April 2024
    Received: 15 December 2023
    Published in TKDD Volume 18, Issue 8

    Check for updates

    Author Tags

    1. Clustering
    2. discriminative learning
    3. contrastive learning
    4. mutual information
    5. auxiliary alignment module

    Qualifiers

    • Research-article

    Funding Sources

    • National Natural Science Foundation of China

    Contributors

    Other Metrics

    Bibliometrics & Citations

    Bibliometrics

    Article Metrics

    • Downloads (Last 12 months)255
    • Downloads (Last 6 weeks)28
    Reflects downloads up to 28 Feb 2025

    Other Metrics

    Citations

    Cited By

    View all
    • (2024)Explainability for Property Violations in Cyberphysical Systems: An Immune-Inspired ApproachIEEE Software10.1109/MS.2024.338728941:5(43-51)Online publication date: 16-Apr-2024

    View Options

    Login options

    Full Access

    View options

    PDF

    View or Download as a PDF file.

    PDF

    eReader

    View online with eReader.

    eReader

    Full Text

    View this article in Full Text.

    Full Text

    Figures

    Tables

    Media

    Share

    Share

    Share this Publication link

    Share on social media