Abstract
This paper addresses the problem of extracting the largest possible number of columns from a given matrix \(X\in \mathbb R^{n\times p}\) in such a way that the resulting submatrix has an coherence smaller than a given threshold \(\eta \). This problem can clearly be expressed as the one of finding a maximum cardinality stable set in the graph whose adjacency matrix is obtained by taking the componentwise absolute value of \(X^tX\) and setting entries less than \(\eta \) to 0 and the other entries to 1. We propose a spectral-type relaxation which boils down to optimising a quadratic function on a sphere. We prove a theoretical approximation bound for the solution of the resulting relaxed problem.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
Notes
- 1.
Here, positivity is trivial.
References
Adcock, B., Hansen, A.C., Poon, C., Roman, B.: Breaking the coherence barrier: a new theory for compressed sensing. Forum Math. Sigma 5, 84 (2017)
Adcock, B., Hansen, A.C., Poon, C., Roman, B., et al.: Breaking the coherence barrier: asymptotic incoherence and asymptotic sparsity in compressed sensing. Preprint (2013)
Arora, S., Ge, R., Moitra, A.:. New algorithms for learning incoherent and overcomplete dictionaries. In: Conference on Learning Theory, pp. 779–806 (2014)
Baraniuk, R.G.: Compressive sensing [lecture notes]. IEEE Signal Process. Mag. 24(4), 118–121 (2007)
Bellec, P.C.: Localized Gaussian width of \(m\)-convex hulls with applications to lasso and convex aggregation. arXiv preprint arXiv:1705.10696 (2017)
Bühlmann, P., Van De Geer, S.: Statistics for High-dimensional Data: Methods, Theory and Applications. Springer, Heidelberg (2011). https://doi.org/10.1007/978-3-642-20192-9
Candes, E., Romberg, J.: Sparsity and incoherence in compressive sampling. Inverse Prob. 23(3), 969 (2007)
Candès, E.J.: Mathematics of sparsity (and a few other things). In: Proceedings of the International Congress of Mathematicians, Seoul, South Korea, vol. 123. Citeseer (2014)
Candes, E.J., Eldar, Y.C., Needell, D., Randall, P.: Compressed sensing with coherent and redundant dictionaries. Appl. Comput. Harmonic Anal. 31(1), 59–73 (2011)
Candès, E.J., Plan, Y.: Near-ideal model selection by l1 minimization. Ann. Stat. 37(5A), 2145–2177 (2009)
Candès, E.J., Wakin, M.B.: An introduction to compressive sampling. IEEE Signal Process. Mag. 25(2), 21–30 (2008)
Cevher, V., Boufounos, P., Baraniuk, R.G., Gilbert, A.C., Strauss, M.J.: Near-optimal Bayesian localization via incoherence and sparsity. In: International Conference on Information Processing in Sensor Networks, IPSN 2009, pp. 205–216. IEEE (2009)
Chrétien, S., Darses, S.: Invertibility of random submatrices via tail-decoupling and a matrix Chernoff inequality. Stat. Probab. Lett. 82(7), 1479–1487 (2012)
Chrétien, S., Darses, S.: Sparse recovery with unknown variance: a Lasso-type approach. IEEE Trans. Inf. Theory 60(7), 3970–3988 (2014)
Foucart, S., Rauhut, H.: A Mathematical Introduction to Compressive Sensing, vol. 1. Birkhäuser, Basel (2013)
Hager, W.W.: Minimizing a quadratic over a sphere. SIAM J. Optim. 12(1), 188–208 (2001)
Mallat, S.: A Wavelet Tour of Signal Processing: The Sparse Way. Academic Press, Cambridge (2008)
Nelson, J.L., Temlyakov, V.N.: On the size of incoherent systems. J. Approximation Theory 163(9), 1238–1245 (2011)
Romberg, J.: Imaging via compressive sampling. IEEE Signal Process. Mag. 25(2), 14–20 (2008)
Van De Geer, S.A., Bühlmann, P., et al.: On the conditions used to prove oracle results for the Lasso. Electron. J. Stat. 3, 1360–1392 (2009)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
A Minimizing Quadratic Functionals on the Sphere
A Minimizing Quadratic Functionals on the Sphere
1.1 A.1 A Semi-explicit Solution
The following result can be found in [16].
Lemma 1
For \(Q\in \mathbb S_p\) and \(q\in \mathbb R^p\), consider the following quadratic programming problem over the sphere:
Let \(\lambda _1 \le \ldots \le \lambda _p\) be the eigenvalues of Q and \(\phi _1\),...,\( \phi _p\) be associated pairwise orthogonal, unit-norm eigenvectors. Let \(\gamma _{k,i}=q^t \phi _i\), \(i=1,\ldots ,p\). Let \(\mathcal E_1=\{i \text { s.t. } \lambda _i=\lambda _1 \}\) and \(\mathcal E_+=\{i \text { s.t. } \lambda _i>\lambda _1 \}\). Then, \(x^*\) is a solution if and only if
and
-
1.
degenerate case: If \(\gamma _i=0\) for all \(i \in \mathcal E_1\) and
$$\begin{aligned} \sum _{i\in \mathcal E_+} \, \frac{\gamma _i^2}{(\lambda _i-\lambda _1)^2} \le 1. \end{aligned}$$then \(c_i^*=\gamma _i/(\lambda _i-\lambda _1)\), \(i\in \mathcal E_1\) and \(c_i^*\), \(i\in \mathcal E_1\) are arbitrary under the constraint that \(\sum _{i\in \mathcal E_1} \quad c^{*^2}_i = 1-\sum _{i\in \mathcal E_+} \quad c^{*^2}_i\).
-
2.
nondegenerate case: If not in the degenerate case, \(c_i^*=\gamma _i/(\lambda _i-\mu )\), \(i=1,\ldots ,n\) for \(\mu > -\lambda _1\) which is a solution of
$$\begin{aligned} \sum _{i=1,\ldots ,n} \, \frac{\gamma _i^2}{(\lambda _i-\mu )^2}&= 1. \end{aligned}$$(11)
Moreover, we have the following useful result.
Corollary 1
If Q is positive definite, and \(\sum _{i=1,\ldots ,p} \ \gamma _i^2/\lambda _i^2 <1\), then \(0<\mu <\lambda _1\).
Proof
This follows immediately from the intermediate value theorem.
1.2 A.2 Bounds on \(\mu \)
From (11), we can get the following easy bounds on \(\mu \).
Lemma 2
Let \(\gamma _{\min }= \min _{i=1}^p \gamma _i\) and \(\gamma _{\max }= \max _{i=1}^p \gamma _i\). Then, we have
and
Proof
The proof is divided into three parts, corresponding to each (double) inequality.
Proof of (12): We have
This immediately gives \(p \gamma _{\max } \ge \max _{i=1}^p \ \{(\lambda _i-\mu )^2\}\). On the one hand, we have
Therefore, we get \(\max _{i=1}^p \{(\lambda _i-\mu )^2\}\ge p \ \gamma _{\min }^2\). On the other hand, we have
Proof of (13):
which gives
for \(i=1,\ldots ,p\). Thus, the lower bound follows. For the other bound, since
we get
and the proof in completed.
1.3 A.3 \(\ell _\infty \) Perturbation of the Linear Term
We now consider the problem of controlling the solution under perturbation of q.
Lemma 3
Consider the two quadratic programming problems over the sphere:
for \(k=1,2\). Assume that the solution to (15) is non-degenerate in both cases \(k=1,2\) and let \(x^*_1\) and \(x^*_2\) be the corresponding solutions. Assume further that \(\sum _{i=1,\ldots ,n} \ \gamma _{k,i}^2/\lambda _i^2 <1\), \(k=1,2\). Let \(\phi \) denote the inverse function of \(x\mapsto x/(1+x)^3\). Then, we have
with \(r^*\) given by
Proof
Let \(\varPhi \) denote the matrix whose columns are the eigenvectors of A. More precisely, \(\lambda _1\le \cdots \le \lambda _p\) and let \(\phi _i\) be an eigenvector associated with \(\lambda _i\), \(i=1,\ldots ,p\). Let \(\gamma _i=q^t \phi _i\), \(i=1,\ldots ,p\). Let \(c_1^*\) (resp. \(c_2^*\)) be the vector of coefficients of \(x_1^*\) (resp. \(x_2^*\)) in the eigenbasis of A. For each \(k=1,2\), there exists a real \(\mu _k\) such that
\(i=1,\ldots ,p\) for \(\mu _k > -\lambda _1\) which is a solution of
Now, apply Neuberger’s Theorem 2 to obtain an estimation of \(\vert \mu _1-\mu _2\vert \) as a function of \(\gamma _1\) and \(\gamma _2\). For this purpose, set
Now, we need to find the smallest value of \(\nu \) such that, for all \(\mu \in B(\mu _1,\nu )\), we need to find a number \(h \in \bar{B}(0,\nu )\) such that
We therefore have that
and since
we have
where \(\cdot ^2\) is to be understood componentwise. Moreover, since \(\sum _{i=1,\ldots ,p}\)\(\gamma _{k,i}^2/\lambda _i^2 <1\), \(k=1,2\),
Thus, for \(\nu >0\) such that
we get from Theorem 2 that there exists a solution to the equation \(F(u)=0\) inside the ball \(\bar{B}(\mu _1,\nu )\). Make the change of variable
and obtain that we need to find \(\alpha \in (0,1)\) such that
Lemma 2 now gives
from which we get that the value \(\nu ^*\) of \(\nu \) given by
is admissible, for \(\Vert \gamma _1^2-\gamma _2^2\Vert _1\) such that the term involving \(\phi \) is less than one.
Therefore,
Finally, using that \(\vert \mu _1-\mu _2\vert \le \nu ^*\), we get
which gives
as announced.
1.4 A.4 Neuberger’s Theorem
In this subsection, we recall Neuberger’s theorem.
Theorem 2
Suppose that \(r > 0\), that \(x \in R^p\), and that F is a continuous function from \(\bar{B}(x,r)\) to \(R^m\) with the property that for each y in B(x, r), there is an h in \(\bar{B}(0,r)\) such that
Then, there exists u in \(\bar{B}(x,r)\) such that \(F(u) = 0\).
Rights and permissions
Copyright information
© 2019 Springer Nature Switzerland AG
About this paper
Cite this paper
Chrétien, S., Ho, Z.W.O. (2019). Incoherent Submatrix Selection via Approximate Independence Sets in Scalar Product Graphs. In: Nicosia, G., Pardalos, P., Umeton, R., Giuffrida, G., Sciacca, V. (eds) Machine Learning, Optimization, and Data Science. LOD 2019. Lecture Notes in Computer Science(), vol 11943. Springer, Cham. https://doi.org/10.1007/978-3-030-37599-7_9
Download citation
DOI: https://doi.org/10.1007/978-3-030-37599-7_9
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-37598-0
Online ISBN: 978-3-030-37599-7
eBook Packages: Computer ScienceComputer Science (R0)