CNN to GNN: Unsupervised Multi-level Knowledge Learning
Pages 4313 - 4317
Abstract
Although graph neural networks (GNNs) can extract the latent relationship-level knowledge among the graph nodes and have achieved excellent performance in unsupervised scenarios, it is weak in learning the instance-level knowledge in contrast to the convolution neural networks (CNNs). Besides, lacking of the graph structure limits the extension of GNNs on non-graph datasets. To solve these problems, we propose a novel unsupervised multi-level knowledge fusion network. It successfully unifies the instance-level and relationship-level knowledge on the non-graph data by distillation from a pre-trained CNN teacher to a GNN student. Meanwhile, a sparse weighted strategy is designed to adaptively extract the sparse graph topology and extend the GNN on non-graph datasets. By optimization of distillation loss, the "boosted'' GNN student can learn the multi-level knowledge and extract more discriminative deep embeddings for clustering. Finally, extensive experiments show it has achieved excellent performance compared with the current methods.
References
[1]
Arthur Asuncion and David Newman. 2007. UCI machine learning repository.
[2]
Deng Cai, Xiaofei He, and Jiawei Han. 2005. Document Clustering Using Locality Preserving Indexing. IEEE Transactions on Knowledge and Data Engineering 17, 12 (2005), 1624--1637.
[3]
Sohan Gunawardena, Khanh Luong, Thirunavukarasu Balasubramaniam, and Richi Nayak. 2024. DCCNMF: Deep Complementary and Consensus Non-negative Matrix Factorization for multi-view clustering. Knowledge-Based Systems 285 (2024), 111330.
[4]
Lars Hagen and Andrew B Kahng. 1992. New spectral methods for ratio cut partitioning and clustering. IEEE transactions on computer-aided design of integrated circuits and systems 11, 9 (1992), 1074--1085.
[5]
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition. 770--778.
[6]
Chenping Hou, Feiping Nie, Xuelong Li, Dongyun Yi, and Yi Wu. 2013. Joint embedding learning and sparse regression: A framework for unsupervised feature selection. IEEE Transactions on Cybernetics 44, 6 (2013), 793--804.
[7]
Ziheng Jiao and Xuelong Li. 2024. An End-to-End Deep Graph Clustering via Online Mutual Learning. IEEE Transactions on Neural Networks and Learning Systems (2024).
[8]
Ziheng Jiao and Xuelong Li. 2024. An End-to-End Deep Graph Clustering via Online Mutual Learning. IEEE Transactions on Neural Networks and Learning Systems (2024), 1--8. https://doi.org/10.1109/TNNLS.2024.3353217
[9]
Thomas N Kipf and MaxWelling. 2016. Semi-supervised classification with graph convolutional networks. arXiv preprint arXiv:1609.02907 (2016).
[10]
Thomas N Kipf and Max Welling. 2016. Variational graph auto-encoders. arXiv preprint arXiv:1611.07308 (2016).
[11]
Van Der Maaten Laurens and Geoffrey Hinton. 2008. Visualizing Data using t-SNE. Journal of Machine Learning Research 9, 2605 (2008), 2579--2605.
[12]
Xuelong Li, Hongyuan Zhang, and Rui Zhang. 2021. Adaptive Graph Auto-Encoder for General Data Clustering. IEEE Transactions on Pattern Analysis and Machine Intelligence (2021), 1--1. https://doi.org/10.1109/TPAMI.2021.3125687
[13]
Suyuan Liu, Qing Liao, Siwei Wang, Xinwang Liu, and En Zhu. 2024. Robust and Consistent Anchor Graph Learning for Multi-View Clustering. IEEE Transactions on Knowledge and Data Engineering (2024).
[14]
Si Lu and Ruisi Li. 2022. DAC--deep autoencoder-based clustering: A general deep learning framework of representation learning. In Intelligent Systems and Applications: Proceedings of the 2021 Intelligent Systems Conference (IntelliSys) Volume 1. Springer, 205--216.
[15]
Michael Lyons, Shigeru Akamatsu, Miyuki Kamachi, and Jiro Gyoba. 1998. Coding facial expressions with gabor wavelets. In Proceedings Third IEEE international conference on automatic face and gesture recognition. IEEE, 200--205.
[16]
Sameer A Nene, Shree K Nayar, Hiroshi Murase, et al. 1996. Columbia object image library (coil-100). (1996).
[17]
Andrew Y Ng, Michael I Jordan, and Yair Weiss. 2002. On spectral clustering: Analysis and an algorithm. In Advances in neural information processing systems. 849--856.
[18]
Feiping Nie, Wei Chang, Rong Wang, and Xuelong Li. 2023. Learning an Optimal Bipartite Graph for Subspace Clustering via Constrained Laplacian Rank. IEEE Transactions on Cybernetics 53, 2 (2023), 1235--1247. https://doi.org/10.1109/ TCYB.2021.3113520
[19]
Feiping Nie, Xiaoqian Wang, and Heng Huang. 2014. Clustering and projected clustering with adaptive neighbors. In Proceedings of the 20th ACM SIGKDD international conference on Knowledge discovery and data mining. 977--986.
[20]
Hae-Sang Park and Chi-Hyuck Jun. 2009. A simple and fast algorithm for Kmedoids clustering. Expert systems with applications 36, 2 (2009), 3336--3341.
[21]
Carl Rasmussen. 1999. The infinite Gaussian mixture model. Advances in neural information processing systems 12 (1999).
[22]
Aruni RoyChowdhury, Xiang Yu, Kihyuk Sohn, Erik Learned-Miller, and Manmohan Chandraker. 2020. Improving Face Recognition by Clustering Unlabeled Faces in the Wild. arXiv (2020). arXiv:2007.06995
[23]
Chenyang Si, Wentao Chen, Wei Wang, Liang Wang, and Tieniu Tan. 2019. An attention enhanced graph convolutional lstm network for skeleton-based action recognition. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 1227--1236.
[24]
Chun Wang, Shirui Pan, Guodong Long, Xingquan Zhu, and Jing Jiang. 2017. Mgae: Marginalized graph autoencoder for graph clustering. In Proceedings of the 2017 ACM on Conference on Information and Knowledge Management. 889--898.
[25]
Han Xiao, Kashif Rasul, and Roland Vollgraf. 2017. Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms. arXiv preprint arXiv:1708.07747 (2017).
[26]
Junyuan Xie, Ross Girshick, and Ali Farhadi. 2016. Unsupervised deep embedding for clustering analysis. In International conference on machine learning. PMLR, 478--487.
[27]
Hongyuan Zhang, Yanan Zhu, and Xuelong Li. 2024. Decouple Graph Neural Networks: Train Multiple Simple GNNs Simultaneously Instead of One. IEEE Transactions on Pattern Analysis and Machine Intelligence (2024), 1--12. https://doi.org/10.1109/TPAMI.2024.3392782
[28]
Rui Zhang, Xuelong Li, Hongyuan Zhang, and Feiping Nie. 2019. Deep fuzzy K-means with adaptive loss and entropy regularization. IEEE Transactions on Fuzzy Systems 28, 11 (2019), 2814--2824.
Index Terms
- CNN to GNN: Unsupervised Multi-level Knowledge Learning
Recommendations
Learning Precoding Policy: CNN or GNN?
2022 IEEE Wireless Communications and Networking Conference (WCNC)Optimizing precoding with deep learning enables its real-time implementation. In addition to the learning perfor-mance such as sum rate, training complexity is also important since neural networks (NNs) have to be re-trained in time-varying channels. By ...
Multi-level wavelet network based on CNN-Transformer hybrid attention for single image deraining
AbstractRemoving rain streaks from rainy images can improve the accuracy of computer vision applications such as object detection. In order to make full use of the frequency domain analysis characteristics of wavelet and combine the advantages of ...
Quantum multi-level wavelet transforms
AbstractThe classical wavelet transform has been widely applied in the information processing field. It implies that quantum wavelet transform (QWT) may play an important role in quantum information processing. In this paper, we present the ...
Comments
Information & Contributors
Information
Published In
October 2024
5705 pages
Copyright © 2024 ACM.
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].
Sponsors
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 21 October 2024
Check for updates
Author Tags
Qualifiers
- Short-paper
Conference
CIKM '24
Sponsor:
CIKM '24: The 33rd ACM International Conference on Information and Knowledge Management
October 21 - 25, 2024
ID, Boise, USA
Acceptance Rates
Overall Acceptance Rate 1,861 of 8,427 submissions, 22%
Upcoming Conference
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 85Total Downloads
- Downloads (Last 12 months)85
- Downloads (Last 6 weeks)14
Reflects downloads up to 28 Feb 2025
Other Metrics
Citations
View Options
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in