research-article

RicciNet: Deep Clustering via A Riemannian Generative Model

Authors:

Philip YuAuthors Info & Claims

WWW '24: Proceedings of the ACM Web Conference 2024

Pages 4071 - 4082

https://doi.org/10.1145/3589334.3645428

Published: 13 May 2024 Publication History

Abstract

In recent years, deep clustering has achieved encouraging results. However, existing deep clustering methods work with the traditional Euclidean space and thus present deficiency on clustering complex structures. On the contrary, Riemannian geometry provides an elegant framework to model complex structures as well as a powerful tool for clustering, i.e., the Ricci flow. In this paper, we rethink the problem of deep clustering, and introduce the Riemannian geometry to deep clustering for the first time. Deep clustering in Riemannian manifold still faces significant challenges: (1) Ricci flow itself is unaware of cluster membership, (2) Ricci curvature prevents the gradient backpropagation, and (3) learning the flow largely remains open in the manifold. To bridge these gaps, we propose a novel Riemannian generative model (RicciNet), a neural Ricci flow with several theoretical guarantees. The novelty is that we model the dynamic self-clustering process of Ricci flow: data points move to the respective clusters in the manifold, influenced by Ricci curvatures. The point's trajectory is characterized by a parametric velocity, taking the form of Ordinary Differential Equation (ODE). Specifically, we encode data points as samples of Gaussian mixture in the manifold where we propose two types of reparameterization approaches: Gumbel reparameterization, and geometric trick. We formulate a differentiable Ricci curvature parameterized by a Riemannian graph convolution. Thereafter, we propose a geometric learning approach in which we study the geometric regularity of the point's trajectory, and learn the flow via distance matching and velocity matching. Consequently, data points go along the shortest Ricci flow to complete clustering. Extensive empirical results show RicciNet outperforms Euclidean deep methods.

Supplemental Material

MP4 File

Supplemental video

Download
75.04 MB

References

[1]

Gregor Bachmann, Gary Bécigneul, and Octavian Ganea. 2020. Constant Curvature Graph Convolutional Networks. In Proceedings of the 37th ICML, Vol. 119. 486--496.

[2]

Gary Bécigneul and Octavian-Eugen Ganea. 2019. Riemannian Adaptive Optimization Methods. In Proceedings of 7th ICLR.

[3]

Deyu Bo, Xiao Wang, Chuan Shi, Meiqi Zhu, Emiao Lu, and Peng Cui. 2020. Structural Deep Clustering Network. In Proceedings of The Web Conference 2020. 1400--1410.

Digital Library

[4]

Silvère Bonnabel. 2013. Stochastic Gradient Descent on Riemannian Manifolds. IEEE Trans. on Autom. Control. 58, 9 (2013), 2217--2229.

[5]

Jinyu Cai, Jicong Fan, Wenzhong Guo, Shiping Wang, Yunhe Zhang, and Zhao Zhang. 2022. Efficient Deep Embedded Subspace Clustering. In Proceedings of CVPR 2022. IEEE, 21--30.

[6]

Ines Chami, Zhitao Ying, Christopher Ré, and Jure Leskovec. 2019. Hyperbolic Graph Convolutional Neural Networks. In Advances in the 32nd NeurIPS. 4869--4880.

[7]

Ricky T. Q. Chen and Yaron Lipman. 2023. Riemannian Flow Matching on General Geometries. CoRR abs/2302.03660 (2023).

[8]

Tian Qi Chen, Yulia Rubanova, Jesse Bettencourt, and David Duvenaud. 2018. Neural Ordinary Differential Equations. In Advances in the 31st NeurIPS. 6572--6583.

[9]

Xiaohui Chen and Yun Yang. 2019. Diffusion K-means clustering on manifolds: provable exact recovery via semidefinite relaxations. CoRR abs/1903.04416 (2019).

[10]

Robin Forman. 2003. Bochner's Method for Cell Complexes and Combinatorial Ricci Curvature. Discret. Comput. Geom. 29, 3 (2003), 323--374.

[11]

Albert Gu, Frederic Sala, Beliz Gunel, and Christopher Ré. 2019. Learning Mixed-Curvature Representations in Product Spaces. In Proceedings of the 7th ICLR. [12] Caglar Gulcehre, Misha Denil, Mateusz Malinowski, Ali Razavi, Razvan Pascanu, Karl Moritz Hermann, Peter Battaglia, Victor Bapst, David Raposo, Adam Santoro, and Nando de Freitas. 2019. Hyperbolic Attention Networks. In Proceedings of ICLR 2019.

[12]

Ishaan Gulrajani, Faruk Ahmed, Martín Arjovsky, Vincent Dumoulin, and Aaron C. Courville. 2017. Improved Training of Wasserstein GANs. In Advances in the 30th NeurIPS. 5767--5777.

Digital Library

[13]

Wengang Guo, Kaiyan Lin, and Wei Ye. 2021. Deep Embedded K-Means Clustering. In Proceedings of ICDM 2021 (Workshops). 686--694.

[14]

Christopher Hopper and Ben Andrews. 2010. The Ricci flow in Riemannian geometry. Springer.

[15]

Pavel Izmailov, Polina Kirichenko, Marc Finzi, and Andrew Gordon Wilson. 2020. Semi-Supervised Learning with Normalizing Flows. In Proceedings of the 37th ICML, Vol. 119. 4615--4630.

[16]

Eric Jang, Shixiang Gu, and Ben Poole. 2017. Categorical Reparameterization with Gumbel-Softmax. In Proceedings of the 5th ICLR.

[17]

Zhuxi Jiang, Yin Zheng, Huachun Tan, Bangsheng Tang, and Hanning Zhou. 2017. Variational Deep Embedding: An Unsupervised and Generative Approach to Clustering. In Proceedings of the 26th IJCAI. 1965--1972.

[18]

Diederik P. Kingma and Jimmy Ba. 2015. Adam: A Method for Stochastic Optimization. In Proceedings of the 3rd ICLR.

[19]

Ivan Kobyzev, Simon J. D. Prince, and Marcus A. Brubaker. 2021. Normalizing Flows: An Introduction and Review of Current Methods. IEEE Trans. Pattern Anal. Mach. Intell. 43, 11 (2021), 3964--3979.

[20]

Marc Law. 2021. Ultrahyperbolic Neural Networks. In Advances in the 34th NeurIPS. 22058--22069.

[21]

Xiaopeng Li, Zhourong Chen, Leonard K. M. Poon, and Nevin L. Zhang. 2019. Learning Latent Superstructures in Variational Autoencoders for Deep Multidimensional Clustering. In Proceedings of ICLR.

[22]

Yunfan Li, Peng Hu, Jerry Zitao Liu, Dezhong Peng, Joey Tianyi Zhou, and Xi Peng. 2021. Contrastive Clustering. In Proceedings of the 35th AAAI. 8547--8555.

[23]

Tong Lin and Hongbin Zha. 2008. Riemannian Manifold Learning. IEEE Trans. Pattern Anal. Mach. Intell. 30, 5 (2008), 796--809.

Digital Library

[24]

Yaron Lipman, Ricky T. Q. Chen, Heli Ben-Hamu, Maximilian Nickel, and Matthew Le. 2023. Flow Matching for Generative Modeling. In Proceedings of the 11th ICLR.

[25]

Xingchao Liu, Chengyue Gong, and Qiang Liu. 2023. Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flow. In Proceedings of the 11th ICLR.

[26]

Yang Liu, Chuan Zhou, Shirui Pan, Jia Wu, Zhao Li, Hongyang Chen, and Peng Zhang. 2023. CurvDrop: A Ricci Curvature Based Approach to Prevent Graph Neural Networks from Over-Smoothing and Over-Squashing. In Proceedings of the ACM Web Conference 2023. 221--230.

Digital Library

[27]

Aaron Lou, Derek Lim, Isay Katsman, Leo Huang, Qingxuan Jiang, Ser-Nam Lim, and Christopher De Sa. 2020. Neural Manifold Ordinary Differential Equations. In Advances in the 33rd NeurIPS.

[28]

Emile Mathieu and Maximilian Nickel. 2020. Riemannian Continuous Normalizing Flows. In Advances in the 33rd NeurIPS.

[29]

Sudipto Mukherjee, Himanshu Asnani, Eugene Lin, and Sreeram Kannan. 2019. ClusterGAN: Latent Space Clustering in Generative Adversarial Networks. In Proceedings of the 33th AAAI. 4610--4617.

Digital Library

[30]

Khang Nguyen, Hieu Nong, Vinh Nguyen, Nhat Ho, Stanley Osher, and Tan Nguyen. 2023. Revisiting Over-smoothing and Over-squashing Using Ollivier-Ricci Curvature. In Proceedings of the 40th ICML.

[31]

Yann Ollivier. 2010. A survey of Ricci curvature for metric spaces and Markov chains. Advanced Studies in Pure Mathematics 57 (2010), 343--381.

[32]

Namyong Park, Ryan A. Rossi, Eunyee Koh, Iftikhar Ahamath Burhanuddin, Sungchul Kim, Fan Du, Nesreen K. Ahmed, and Christos Faloutsos. 2022. CGC: Contrastive Graph Clustering for Community Detection and Tracking. In Proceedings of the Web Conference. 1115--1126.

Digital Library

[33]

Peter Petersen. 2016. Riemannian Geometry. Graduate Texts in Mathematics, Vol. 171. Springer International Publishing.

[34]

Janis Postels, Mengya Liu, Riccardo Spezialetti, Luc Van Gool, and Federico Tombari. 2021. Go with the Flows: Mixtures of Normalizing Flows for Point Cloud Generation and Reconstruction. In Proceedings of 3DV. IEEE, 1249--1258.

[35]

Danilo Jimenez Rezende and Shakir Mohamed. 2015. Variational Inference with Normalizing Flows. In Proceedings of the 32nd ICML, Vol. 37. 1530--1538.

[36]

Noam Rozen, Aditya Grover, Maximilian Nickel, and Yaron Lipman. 2021. Moser Flow: Divergence-based Generative Modeling on Manifolds. In Advances in the 34th NeurIPS. 17669--17680.

[37]

Ryohei Shimizu, Yusuke Mukuta, and Tatsuya Harada. 2021. Hyperbolic Neural Networks. In Proceedings of 9th ICLR. OpenReview.net.

[38]

Li Sun, Zhenhao Huang, Zixi Wang, Feiyang Wang, Hao Peng, and Philip S. Yu. 2024. Motif-aware Riemannian Graph Neural Network with Generative- Contrastive Learning. In Proceedings of the 38th AAAI.

[39]

Li Sun, Zhenhao Huang, Hua Wu, Junda Ye, Hao Peng, Zhengtao Yu, and Philip S. Yu. 2023. DeepRicci: Self-supervised Graph Structure-Feature Co-Refinement for Alleviating Over-squashing. In Proceedings of the 23rd ICDM. 558--567.

[40]

Li Sun, Feiyang Wang, Junda Ye, Hao Peng, and Philip S. Yu. 2023. Congregate: Contrastive Graph Clustering in Curvature Spaces. In Proceedings of the 32nd IJCAI. 2296--2305.

[41]

Li Sun, Junda Ye, Hao Peng, Feiyang Wang, and Philip S. Yu. 2023. Self-Supervised Continual Graph Learning in Adaptive Riemannian Spaces. In Proceedings of the 37th AAAI. 4633--4642.

[42]

Li Sun, Junda Ye, Hao Peng, and Philip S. Yu. 2022. A Self-supervised Riemannian GNN with Time Varying Curvature for Temporal Graph Learning. In Proceedings of the 31st ACM CIKM. 1827--1836.

[43]

Li Sun, Zhongbao Zhang, Junda Ye, Hao Peng, Jiawei Zhang, Sen Su, and Philip S. Yu. 2022. A Self-Supervised Mixed-Curvature Graph Neural Network. In Proceedings of the 36th AAAI. 4146--4155.

[44]

Li Sun, Zhongbao Zhang, Jiawei Zhang, Feiyang Wang, Hao Peng, Sen Su, and Philip S. Yu. 2021. Hyperbolic Variational Graph Neural Network for Modeling Dynamic Graphs. In Proceedings of the 35th AAAI. 4375--4383.

[45]

Jake Topping, Francesco Di Giovanni, Benjamin Paul Chamberlain, Xiaowen Dong, and Michael M. Bronstein. 2022. Understanding over-squashing and bottlenecks on graphs via curvature. In Proceedings of the 10th ICLR. 1--30.

[46]

Lucas Vinh Tran, Yi Tay, Shuai Zhang, Gao Cong, and Xiaoli Li. 2020. HyperML: A Boosting Metric Learning Approach in Hyperbolic Space for Recommender Systems. In Proceedings of the 13th WSDM. ACM, 609--617.

[47]

Wenxuan Tu, Sihang Zhou, Xinwang Liu, Xifeng Guo, Zhiping Cai, En Zhu, and Jieren Cheng. 2021. Deep Fusion Clustering Network. In Proceedings of the 35th AAAI. 9978--9987.

[48]

Tianchun Wang, Farzaneh Mirzazadeh, Xiang Zhang, and Jie Chen. 2023. GC-Flow: A Graph-Based Flow Network for Effective Clustering. In Proceedings of the 40th ICML, Vol. 202. 36157--36173.

[49]

Junyuan Xie, Ross B. Girshick, and Ali Farhadi. 2016. Unsupervised Deep Embedding for Clustering Analysis. In Proceedings of the 33rd ICML, Vol. 48. 478--487.

[50]

Bo Xiong, Shichao Zhu, Mojtaba Nayyeri, Chengjin Xu, Shirui Pan, Chuan Zhou, and Steffen Staab. 2022. Ultrahyperbolic Knowledge Graph Embeddings. In Proceedings of the 28th SIGKDD. ACM, 2130--2139.

Digital Library

[51]

Menglin Yang, Min Zhou, Rex Ying, Yankai Chen, and Irwin King. 2023. Hyperbolic Representation Learning: Revisiting and Advancing. In Proceedings of the 40th ICML.

[52]

Ruitong Zhang, Hao Peng, Yingtong Dou, Jia Wu, Qingyun Sun, Yangyang Li, Jingyi Zhang, and Philip S. Yu. 2022. Automating DBSCAN via Deep Reinforcement Learning. In Proceedings of the 31st CIKM. 2620--2630.

[53]

Xi Zhao, Yao Tian, Kai Huang, Bolong Zheng, and Xiaofang Zhou. 2023. Towards Efficient Index Construction and Approximate Nearest Neighbor Search in High- Dimensional Spaces. Proceedings of VLDB 16, 8 (2023), 1979--1991.

Digital Library

[54]

Sheng Zhou, Hongjia Xu, Zhuonan Zheng, Jiawei Chen, Zhao Li, Jiajun Bu, Jia Wu, Xin Wang, Wenwu Zhu, and Martin Ester. 2022. A Comprehensive Survey on Deep Clustering: Taxonomy, Challenges, and Future Directions. CoRR abs/2206.07579 (2022).

[55]

Yubo Zhuang, Xiaohui Chen, and Yun Yang. 2022. Wasserstein K-means for clustering probability distributions. In Advances in the 36th NeurIPS. 1--14.

Cited By

Wang HLu NWang ZYan YCarneiro GWang Z(2025)Self-Correcting ClusteringIEEE Transactions on Knowledge and Data Engineering10.1109/TKDE.2024.352302137:3(1439-1454)Online publication date: Mar-2025
https://doi.org/10.1109/TKDE.2024.3523021
Rabbani SMedri ISamad M(2025)Deep clustering of tabular data by weighted Gaussian distribution learningNeurocomputing10.1016/j.neucom.2025.129359623(129359)Online publication date: Mar-2025
https://doi.org/10.1016/j.neucom.2025.129359
Sun LHuang ZPeng HWang YLiu CYu PSalakhutdinov RKolter ZHeller KWeller AOliver NScarlett JBerkenkamp F(2024)LSEnetProceedings of the 41st International Conference on Machine Learning10.5555/3692070.3693986(47078-47104)Online publication date: 21-Jul-2024
https://dl.acm.org/doi/10.5555/3692070.3693986

Index Terms

RicciNet: Deep Clustering via A Riemannian Generative Model
1. Computing methodologies
  1. Machine learning
    1. Learning paradigms
      1. Unsupervised learning
    2. Machine learning approaches
      1. Neural networks
2. Information systems
  1. Information systems applications
    1. Data mining
      1. Clustering

Recommendations

Geometry of Deep Generative Models for Disentangled Representations
ICVGIP '18: Proceedings of the 11th Indian Conference on Computer Vision, Graphics and Image Processing

Deep generative models like variational autoencoders approximate the intrinsic geometry of high dimensional data manifolds by learning a set of low-dimensional latent-space variables and an embedding function. The geometrical properties of these latent ...
A Riemannian approach to graph embedding

In this paper, we make use of the relationship between the Laplace-Beltrami operator and the graph Laplacian, for the purposes of embedding a graph onto a Riemannian manifold. To embark on this study, we review some of the basics of Riemannian geometry ...
An overview on deep clustering
Abstract
In recent years, with the great success of deep learning and especially deep unsupervised learning, many deep architectural clustering methods, collectively known as deep clustering, have emerged. Deep clustering shows the potential to outperform ...
Highlights
- Analyze the theory of deep clustering and its performance advantages and disadvantages.
- Summarizes the improvement of deep clustering in recent years, analyzes the improvement effect, and shows it with experimental data.
- Finally, ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

WWW '24: Proceedings of the ACM Web Conference 2024

May 2024

4826 pages

ISBN:9798400701719

DOI:10.1145/3589334

General Chairs:
Tat-Seng Chua
National University of Singapore
,
Chong-Wah Ngo
Singapore Management University
,
Proceedings Chair:
Roy Ka-Wei Lee
Singapore University of Technology and Design
,
Program Chairs:
Ravi Kumar
Google
,
Hady W. Lauw
Singapore Management University

Copyright © 2024 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

SIGWEB: ACM Special Interest Group on Hypertext, Hypermedia, and Web

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 13 May 2024

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

Conference

WWW '24

Sponsor:

SIGWEB

WWW '24: The ACM Web Conference 2024

May 13 - 17, 2024

Singapore, Singapore

Acceptance Rates

Overall Acceptance Rate 1,899 of 8,196 submissions, 23%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

3
Total Citations
View Citations
382
Total Downloads

Downloads (Last 12 months)382
Downloads (Last 6 weeks)30

Reflects downloads up to 05 Mar 2025

Other Metrics

View Author Metrics

Citations

Cited By

Wang HLu NWang ZYan YCarneiro GWang Z(2025)Self-Correcting ClusteringIEEE Transactions on Knowledge and Data Engineering10.1109/TKDE.2024.352302137:3(1439-1454)Online publication date: Mar-2025
https://doi.org/10.1109/TKDE.2024.3523021
Rabbani SMedri ISamad M(2025)Deep clustering of tabular data by weighted Gaussian distribution learningNeurocomputing10.1016/j.neucom.2025.129359623(129359)Online publication date: Mar-2025
https://doi.org/10.1016/j.neucom.2025.129359
Sun LHuang ZPeng HWang YLiu CYu PSalakhutdinov RKolter ZHeller KWeller AOliver NScarlett JBerkenkamp F(2024)LSEnetProceedings of the 41st International Conference on Machine Learning10.5555/3692070.3693986(47078-47104)Online publication date: 21-Jul-2024
https://dl.acm.org/doi/10.5555/3692070.3693986

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten