Low-Rank and Sparse Multi-task Learning

Abstract

Multi-task learning (MTL) aims to improve the overall generalization performance by learning multiple related tasks simultaneously. Specifically, MTL exploits the intrinsic task relatedness, based on which the informative domain knowledge from each task can be shared across multiple tasks and thus facilitate the individual task learning. Modeling the relationship of multiple tasks is critical to the practical performance of MTL. We propose to correlate multiple tasks using a low-rank representation and formulate our MTL approaches as mathematical optimization problems of minimizing the empirical loss regularized by the aforementioned low-rank structure and a separate sparse structure. For the proposed MTL approaches, we develop gradient based optimization algorithms to efficiently find their globally optimal solutions. We also conduct theoretical analysis on our MTL approaches, i.e., deriving performance bounds to evaluate how well the integration of low-rank and sparse representations can estimate multiple related tasks.

Appendix

Lemma 1

Let $\delta _1, \delta _2, \ldots , \delta _n$ be a random sample of size $n$ from the Gaussian distribution $\fancyscript{N} (0, \sigma )$. Let $x_1, x_2, \ldots , x_n$ satisfy $x_1^2 + x_2^2 + \cdots + x_n^2 = 1$. Denote a random variable $v$ as

$$\begin{aligned} v = \frac{1}{\sigma } \sum _{i=1}^n x_i \delta _{i}. \end{aligned}$$

Then $v$ obeys the Gaussian distribution $\fancyscript{N} (0, 1)$.

Proof

Since $\{ \delta _i \}$ are mutually independent, the mean of the random variable $v$ can be computed as

$$\begin{aligned} \mathbb {E} (v) = \mathbb {E} \left( \frac{1}{\sigma } \sum _{i=1}^n x_i \delta _{i} \right) = \frac{1}{\sigma } \sum _{i=1}^n x_i \mathbb {E} \left( \delta _{i} \right) = 0. \end{aligned}$$

Similarly, the variance of $v$ can be computed

$$\begin{aligned} \mathbb {E} \left( v - \mathbb {E} (v) \right) ^2 = \mathbb {E} \left( \frac{1}{\sigma ^2} \sum _{i=1}^n x_{i}^2 \delta _{i}^2 \right) = \frac{1}{\sigma ^2} \sum _{i=1}^n x_i^2 \mathbb {E} \left( \delta _{i}^2 \right) = 1, \end{aligned}$$

where the first equality follows from $\mathbb {E} \left( \delta _i \delta _j \right) = 0~(i \ne j)$. Using the fact that the sum of Gaussian random variables is Gaussian distributed, we complete the proof of this lemma.

Lemma 2

Let $\fancyscript{X}_p^2$ be a chi-squared random variable with $p$ degrees of freedom. Then

$$\begin{aligned} \Pr \left( \fancyscript{X}_p^2 \ge p + \pi \right) \le \exp \left( - \frac{1}{2} \left( \pi - p \log \left( 1 + \frac{\pi }{p} \right) \right) \right) , \pi > 0. \end{aligned}$$

Proof

From Theorem $4.1$ in [42], we approximate the chi-square distribution using a normal distribution as

$$\begin{aligned} \Pr \left( \fancyscript{X}_p^2 \ge q \right) \le \Pr \left( \fancyscript{N}_{0,1} \ge z_p(q) \right) , \, q > p, \end{aligned}$$

where $\fancyscript{N}_{0,1} \sim \fancyscript{N} (0, 1)$ and $z_p(q) = \sqrt{q - p - p \log \left( \frac{q}{p} \right) }$. It is known that for $x \sim \fancyscript{N} (0, 1)$, the inequality $\Pr \left( x \ge t \right) \le \exp (- \frac{t^2}{2})$ holds. Therefore we have

$$\begin{aligned} \Pr \left( \fancyscript{X}_p^2 \ge q \right) \le \exp \left( - \frac{1}{2} z_p^2(q) \right) . \end{aligned}$$

By substituting $q = p + \pi ~(\pi > 0)$ into the inequality above, we complete the proof of this lemma.