Boosting for graph classification with universum

Abstract

Recent years have witnessed extensive studies of graph classification due to the rapid increase in applications involving structural data and complex relationships. To support graph classification, all existing methods require that training graphs should be relevant (or belong) to the target class, but cannot integrate graphs irrelevant to the class of interest into the learning process. In this paper, we study a new universum graph classification framework which leverages additional “non-example” graphs to help improve the graph classification accuracy. We argue that although universum graphs do not belong to the target class, they may contain meaningful structure patterns to help enrich the feature space for graph representation and classification. To support universum graph classification, we propose a mathematical programming algorithm, ugBoost, which integrates discriminative subgraph selection and margin maximization into a unified framework to fully exploit the universum. Because informative subgraph exploration in a universum setting requires the search of a large space, we derive an upper bound discriminative score for each subgraph and employ a branch-and-bound scheme to prune the search space. By using the explored subgraphs, our graph classification model intends to maximize the margin between positive and negative graphs and minimize the loss on the universum graph examples simultaneously. The subgraph exploration and the learning are integrated and performed iteratively so that each can be beneficial to the other. Experimental results and comparisons on real-world dataset demonstrate the performance of our algorithm.

Appendix

Lagrangian Dual of Eq. (7). The Lagrangian function of Eq. (7) can be written as:

$$\begin{aligned} L(\varvec{\xi },\varvec{\psi },\varvec{\eta }, \varvec{w})= & {} \Vert \mathbf {w}\Vert + C_l\sum \limits _{i=1}^{l} \xi _i + C_u\sum \limits _{j=l+1}^n{(\psi _j+\eta _j)}\nonumber \\&-\, \sum _{i=1}^{l}\alpha _i\{y_i \sum \limits _{k=1}^{m}w_k\cdot \hbar _{g_k}(G_i) + \xi _i -1\} \nonumber \\&+\, \sum _{j=l+1}^{n}\beta _j\{\sum \limits _{k=1}^{m}w_k \cdot \hbar _{g_k}(G_j)-\varepsilon - \psi _j\} \nonumber \\&-\, \sum _{j=l+1}^{n}p_j\{\sum \limits _{k=1}^{m}w_k \cdot \hbar _{g_k}(G_j)+\varepsilon + \eta _j\} \nonumber \\&-\,\varvec{r}^T\varvec{w}-\varvec{s}^T\varvec{\xi }-\varvec{q}^T\varvec{\psi }-\varvec{z}^T\varvec{\eta } \end{aligned}$$

(14)

where, we have $\alpha _i \ge 0, \beta _i \ge 0, p_i \ge 0, r_i \ge 0, s_i \ge 0, q_i \ge 0, z_i \ge 0$.

At optimum, the first derivative of the Lagrangian with respect to the primal variables ($\varvec{\xi },\varvec{w}$, $\varvec{\psi }$ and $\varvec{\eta }$) must vanish,

$$\begin{aligned} \frac{\partial L}{\partial \xi _i}= & {} C_l - \alpha _i -s_i = 0 ~~\Rightarrow 0 \le \alpha _i \le C_l\\ \frac{\partial L}{\partial \psi _i}= & {} C_u - \beta _i -q_i = 0 ~~\Rightarrow 0 \le \beta _i \le C_u\\ \frac{\partial L}{\partial \eta _i}= & {} C_u - p_i -s_i = 0 ~~\Rightarrow 0 \le p_i \le C_u\\ \frac{\partial L}{\partial w_k}= & {} 1 - \sum \limits _{i=1}^l\alpha _i y_i \hbar _{g_k}(G_i) + \sum \limits _{j=l+1}^n \beta _j \hbar _{g_k}(G_j) \\&-\, \sum \limits _{j=l+1}^n p_j \hbar _{g_k}(G_j) -r_k = 0 \\\Rightarrow & {} \sum \limits _{i=1}^l\alpha _i y_i \hbar _{g_k}(G_j) + \sum \limits _{j=l+1}^n (p_j-\beta _j) \hbar _{g_k}(G_j) \le 1, \quad \forall k \end{aligned}$$

Substituting these variables in Eq. (14), we obtain the its dual problem as Eq. (8).