Abstract
The paper considers the sparse envelope function, defined as the biconjugate of the sum of a squared \(\ell _2\)-norm function and the indicator of the set of k-sparse vectors. It is shown that both function and proximal values of the sparse envelope function can be reduced into a one-dimensional search that can be efficiently performed in linear time complexity in expectation. The sparse envelope function naturally serves as a regularizer that can handle both sparsity and grouping information in inverse problems, and can also be utilized in sparse support vector machine problems.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
Notes
Obviously, the \(\ell _0\)-“norm" is not actually a norm.
n being the underlying dimension and k being the sparsity level.
The lemma is written in a general way that will allow us to compute the prox operator of \({{\mathcal {S}}}_k\) later on.
If \(x_i=0\), then the formula (3.6) implies that \(u_i(\eta )=0\) for all \(\eta \ge 0.\)
References
Argyriou, A., Foygel, R., Srebro, N.: Sparse prediction with the k-support norm. In: Pereira, F., Burges, C.J.C., Bottou, L., Weinberger, K.Q. (eds.) Advances in Neural Information Processing Systems 25, pp. 1457–1465. Curran Associates Inc., New York (2012)
Beck, A.: First-Order Methods in Optimization, Volume 25 of MOS-SIAM Series on Optimization. Society for Industrial and Applied Mathematics (SIAM). Mathematical Optimization Society, Philadelphia (2017)
Beck, A., Teboulle, M.: A fast iterative shrinkage-thresholding algorithm for linear inverse problems. SIAM J. Imag. Sci. 2(1), 183–202 (2009)
Brucker, P.: An \(O(n)\) algorithm for quadratic knapsack problems. Oper. Res. Lett. 3(3), 163–166 (1984)
Bruckstein, A.M., Donoho, D.L., Elad, M.: From sparse solutions of systems of equations to sparse modeling of signals and images. SIAM Rev. 51(1), 34–81 (2009)
Cormen, T.H., Leiserson, C.E., Rivest, R.L., Stein, C.: Introduction to Algorithms, 3rd edn. MIT Press, Cambridge (2009)
Cortes, C., Vapnik, V.: Support-vector networks. Mach. Learn. 20(3), 273–297 (1995)
Moreau, J.J.: Proximité et dualité dans un espace hilbertien. Bull. Soc. Math. France 93, 273–299 (1965)
Rockafellar, R.T.: Convex Analysis. Princeton Mathematical Series, No. 28. Princeton University Press, Princeton (1970)
Singer,Y., Duchi, J., Shalev-Shwartz, S., Chandra, T.: Efficient projections onto the l1-ball for learning in high dimensions. In: Proceedings of the International Conference on Machine Learning (ICML) (2008)
Tibshirani, R.: Regression shrinkage and selection via the lasso. J. R. Stat. Soc. Ser. B 58(1), 267–288 (1996)
Weston, J., Mukherjee, S., Chapelle, O., Pontil, M., Poggio, T., Vapnik, V.: Feature selection for svms. In: Advances in Neural Information Processing Systems 13, pp. 668–674. MIT Press, Cambridge (2001)
Zou, H., Hastie, T.: Regularization and variable selection via the elastic net. J. R. Stat. Soc. Ser. B Stat. Methodol. 67(2), 301–320 (2005)
Funding
Funding was provided by Israel Science Foundation (Grant Number 92621).
Author information
Authors and Affiliations
Corresponding author
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
About this article
Cite this article
Beck, A., Refael, Y. Sparse regularization via bidualization. J Glob Optim 82, 463–482 (2022). https://doi.org/10.1007/s10898-021-01089-w
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10898-021-01089-w