Abstract
Deep clustering has recently emerged as a promising direction in clustering analysis, which aims to leverage the representation learning power of deep neural networks to enhance the clustering of highly-complex data. However, most of the existing deep clustering algorithms tend to utilize a single layer (typically the last fully-connected layer) of representation to build the clustering, yet cannot well exploit the rich and diverse information hidden in multiple layers. In view of this, this paper proposes a deep triply-aligned clustering (D-TRACE) approach, which is able to jointly explore three types of representations from multiple layers in the neural network. Specifically, we incorporate the contrastive learning into the first-stage network training, where three modules (i.e., the backbone network, the instance contrastive head, and the cluster contrastive head) are simultaneously optimized. By fusing the three types of representations from the three modules, we further propose the concept of the triply-aligned representation, based on which a unified neural network with a reconstruction loss and a Kullback-Leibler (KL) divergence based clustering loss is trained and thus the final clustering can be achieved in an unsupervised manner. Experiments on multiple image datasets demonstrate the superiority of our D-TRACE approach over the state-of-the-art deep clustering approaches.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Bengio, Y., Lamblin, P., Popovici, D., Larochelle, H.: Greedy layer-wise training of deep networks. In: Advances in Neural Information Processing Systems, vol. 19 (2006)
Cai, D., He, X., Wang, X., Bao, H., Han, J.: Locality preserving nonnegative matrix factorization. In: Proceedings of International Joint Conference on Artificial Intelligence (2009)
Caron, M., Bojanowski, P., Joulin, A., Douze, M.: Deep clustering for unsupervised learning of visual features. In: Proceedings of European Conference on Computer Vision, pp. 132–149 (2018)
Chang, J., Guo, Y., Wang, L., Meng, G., Xiang, S., Pan, C.: Deep discriminative clustering analysis. arXiv preprint arXiv:1905.01681 (2019)
Chang, J., Wang, L., Meng, G., Xiang, S., Pan, C.: Deep adaptive image clustering. In: Proceedings of IEEE International Conference on Computer Vision, pp. 5879–5887 (2017)
Chen, T., Kornblith, S., Norouzi, M., Hinton, G.: A simple framework for contrastive learning of visual representations. In: Proceedings of International Conference on Machine Learning, pp. 1597–1607 (2020)
Coates, A., Ng, A., Lee, H.: An analysis of single-layer networks in unsupervised feature learning. In: Proceedings of International Conference on Artificial Intelligence and Statistics, pp. 215–223 (2011)
Dang, Z., Deng, C., Yang, X., Wei, K., Huang, H.: Nearest neighbor matching for deep clustering. In: Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 13693–13702 (2021)
Dizaji, K.G., Herandi, A., Deng, C., Cai, W., Huang, H.: Deep clustering via joint convolutional autoencoder embedding and relative entropy minimization. In: Proceedings of IEEE International Conference on Computer Vision, pp. 5736–5745 (2017)
Guo, X., Gao, L., Liu, X., Yin, J.: Improved deep embedded clustering with local structure preservation. In: Proceedings of International Joint Conference on Artificial Intelligence, pp. 1753–1759 (2017)
He, K., Fan, H., Wu, Y., Xie, S., Girshick, R.: Momentum contrast for unsupervised visual representation learning. In: Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9729–9738 (2020)
He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, pp. 770–778 (2016)
Huang, D., Wang, C., Peng, H., Lai, J., Kwoh, C.: Enhanced ensemble clustering via fast propagation of cluster-wise similarities. IEEE Trans. Syst. Man Cybern. Syst. 51(1), 508–520 (2021)
Huang, D., Wang, C.D., Wu, J.S., Lai, J.H., Kwoh, C.K.: Ultra-scalable spectral clustering and ensemble clustering. IEEE Trans. Knowl. Data Eng. 32(6), 1212–1226 (2020)
Huang, D., Wang, C.D., Lai, J.H., Kwoh, C.K.: Toward multidiversified ensemble clustering of high-dimensional data: from subspaces to metrics and beyond. IEEE Trans. Cybern. (2021, in press)
Huang, J., Gong, S., Zhu, X.: Deep semantic clustering by partition confidence maximisation. In: Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8849–8858 (2020)
Jain, A.K.: Data clustering: 50 years beyond k-means. Pattern Recogn. Lett. 31(8), 651–666 (2010)
Ji, X., Henriques, J.F., Vedaldi, A.: Invariant information clustering for unsupervised image classification and segmentation. In: Proceedings of IEEE/CVF International Conference on Computer Vision, pp. 9865–9874 (2019)
Kingma, D.P., Ba, J.: Adam: a method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014)
Kingma, D.P., Welling, M.: Auto-encoding variational Bayes. arXiv preprint arXiv:1312.6114 (2013)
Krizhevsky, A., Hinton, G., et al.: Learning multiple layers of features from tiny images (2009)
Li, Y., Hu, P., Liu, Z., Peng, D., Zhou, J.T., Peng, X.: Contrastive clustering. In: Proceedings of AAAI Conference on Artificial Intelligence (2021)
Van der Maaten, L., Hinton, G.: Visualizing data using t-SNE. J. Mach. Learn. Res. 9(11) (2008)
Ng, A.Y., Jordan, M.I., Weiss, Y.: On spectral clustering: analysis and an algorithm. In: Advances in Neural Information Processing Systems, pp. 849–856 (2002)
Nigam, K., Ghani, R.: Analyzing the effectiveness and applicability of co-training. In: Proceedings of International Conference on Information and Knowledge Management, pp. 86–93 (2000)
Radford, A., Metz, L., Chintala, S.: Unsupervised representation learning with deep convolutional generative adversarial networks. arXiv preprint arXiv:1511.06434 (2015)
Van Gansbeke, W., Vandenhende, S., Georgoulis, S., Proesmans, M., Van Gool, L.: Scan: learning to classify images without labels. In: Proceedings of European Conference on Computer Vision, pp. 268–285 (2020)
Vincent, P., Larochelle, H., Lajoie, I., Bengio, Y., Manzagol, P.A., Bottou, L.: Stacked denoising autoencoders: learning useful representations in a deep network with a local denoising criterion. J. Mach. Learn. Res. 11(12) (2010)
Wu, J., et al.: Deep comprehensive correlation mining for image clustering. In: Proceedings of IEEE/CVF International Conference on Computer Vision, pp. 8150–8159 (2019)
Xie, J., Girshick, R., Farhadi, A.: Unsupervised deep embedding for clustering analysis. In: Proceedings of International Conference on Machine Learning, pp. 478–487 (2016)
Yang, J., Parikh, D., Batra, D.: Joint unsupervised learning of deep representations and image clusters. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, pp. 5147–5156 (2016)
Zeiler, M.D., Krishnan, D., Taylor, G.W., Fergus, R.: Deconvolutional networks. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, pp. 2528–2535 (2010)
Acknowledgments
This work was supported by the Science and Technology Program of Guangzhou, China (202201010314), the NSFC (61976097 & 61876193), and the Natural Science Foundation of Guangdong Province (2021A1515012203).
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Chen, DH., Huang, D., Cheng, H., Wang, CD. (2022). D-TRACE: Deep Triply-Aligned Clustering. In: Pimenidis, E., Angelov, P., Jayne, C., Papaleonidas, A., Aydin, M. (eds) Artificial Neural Networks and Machine Learning – ICANN 2022. ICANN 2022. Lecture Notes in Computer Science, vol 13529. Springer, Cham. https://doi.org/10.1007/978-3-031-15919-0_44
Download citation
DOI: https://doi.org/10.1007/978-3-031-15919-0_44
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-15918-3
Online ISBN: 978-3-031-15919-0
eBook Packages: Computer ScienceComputer Science (R0)