An Information Theoretic Perspective for Heterogeneous Subgraph Federated Learning

Abstract

Mining graph data has gained wide attention in modern applications. With the explosive growth of graph data, it is common to see many of them collected and stored in different distinction systems. These local graphs can not be directly shared due to privacy and bandwidth concerns. Thus, Federated Learning approach needs to be considered to collaboratively train a powerful generalizable model. However, these local subgraphs are usually heterogeneously distributed. Such heterogeneity brings challenges for subgraph federated learning. In this work, we analyze subgraph federated learning and find that sub-optimal objectives under the FedAVG training setting influence the performance of GNN. To this end, we propose InfoFedSage, a federated subgraph learning framework guided by Information bottleneck to alleviate the non-iid issue. Experiments on public datasets demonstrate the effectiveness of InfoFedSage against heterogeneous subgraph federated learning.

J. Guo and S. Li—Equal contribution.

A Appendix

1.1 A.1 Proof for Proposition 4.1

We first state a lemma.

Lemma A.1. Given X have n states ($x_1,x_2, \cdots x_n$) and $x_1$ can be divided into k sub states ($x_{11},x_{12}, \cdots x_{1n}$), Y has m states ($y_1,y_2, \cdots y_n$), we have

$$\begin{aligned} \begin{aligned}&I\left( x_{11}, x_{12}, \cdots , x_{1 k}, x_{2}, \cdots , x_{n} ; Y\right) \\&=\,p\left( x_{1}\right) \cdot I\left( x_{11}, x_{12}, \cdots , x_{1 k} ; Y\right) +I\left( x_{1}, x_{2}, \cdots , x_{n} ; Y\right) \end{aligned} \end{aligned}$$

(15)

Proof. The mutual information between ($x_{11},x_{12}, \cdots x_{1n}$) and Y:

$$\begin{aligned} \begin{aligned}&I\left( x_{11}, x_{12}, \cdots , x_{1 k} ; Y\right) \\&=\,H\left( x_{11}, x_{12}, \cdots , x_{1 k}\right) -H\left( x_{11}, x_{12}, \cdots , x_{1 k} / Y\right) \\ \end{aligned} \end{aligned}$$

(16)

Then, we have

$$\begin{aligned} \begin{aligned}&I\left( x_{11}, x_{12}, \cdots , x_{1 k}, x_{2}, \cdots , x_{n} ; Y\right) \\&= -\,\sum _{t=1}^{k} p\left( x_{1 t}\right) \log \frac{p\left( x_{1 t}\right) }{p\left( x_{1}\right) }+\sum _{t=1}^{k} \sum _{j=1}^{m} p\left( x_{1 t} y_{j}\right) \log \frac{p\left( x_{1 t} y_{j}\right) }{p\left( x_{1} y_{j}\right) }\\&+\,I\left( x_{1}, x_{2}, \cdots , x_{n} ; Y\right) \\&=\, p\left( x_{1}\right) \cdot I\left( x_{11}, x_{12}, \cdots , x_{1 k} ; Y\right) +I\left( x_{1}, x_{2}, \cdots , x_{n} ; Y\right) \end{aligned} \end{aligned}$$

(17)

Then, we can get Corollary A.1:

$$\begin{aligned} I\left( x_{11}, x_{12}, \cdots , x_{1 k}, x_{2}, \cdots , x_{n} ; Y\right) \ge I\left( x_{1}, x_{2}, \cdots , x_{n} ; Y\right) \end{aligned}$$

(18)

We restate Proposition 4.1: For $Z_X' = Z_{X1} \cup \cdots \cup Z_{Xm}$ and $Y' = Y_{1} \cup \cdots \cup Y_{m}$, we have

$$\begin{aligned} \begin{aligned} {\mathcal {L}}_{\text {FedAVG}} \ge -\frac{1}{m}\sum _{i=1}^{m} I(Z_{X_i}, Y_{i}) \ge -I(Z_X', Y'). \end{aligned} \end{aligned}$$

(19)

Proof. We first consider the first inequality, since the definition of mutual information has the form

$$\begin{aligned} \begin{aligned} I(Z_{X_i}, Y_i)=\sum p(y_i, z_{X_i}) \log p(y_i \mid z_{X_i})+H(Y_i) \end{aligned} \end{aligned}$$

(20)

Notice that the entropy of labels $H(Y_i)$ is independent of our optimization procedure and can be ignored. Therefore, we have

$$\begin{aligned} \begin{aligned} {\mathcal {L}}_{\textrm{FedAVG}} =&-\frac{1}{m}\sum _{i=1}^{m} I(Z_{X_i} ; Y_i) + \frac{1}{m} H(Y) \ge -\frac{1}{m} \sum _{i=1}^{m} I(Z_{X_i}, Y_{i}). \end{aligned} \end{aligned}$$

(21)

Then, we consider the second inequality. Directly, based on our Corollary A.1, we can get

$$\begin{aligned} \begin{aligned} {\mathcal {L}}_{\textrm{FedAVG}} \ge&-\frac{1}{m}\sum _{i=1}^{m} I(Z_{X_i}, Y_{i}) \\ \ge&-max (I(Z_{X_i}, Y_{i})) \ge -I(Z', Y_{i} ) \ge -I(Z', Y') \end{aligned} \end{aligned}$$

(22)