Abstract
In this paper we propose randomized first-order algorithms for solving bilinear saddle points problems. Our developments are motivated by the need for sublinear time algorithms to solve large-scale parametric bilinear saddle point problems where cheap online assessment of the solution quality is crucial. We present the theoretical efficiency estimates of our algorithms and discuss a number of applications, primarily to the problem of ℓ 1 minimization arising in sparsity-oriented signal processing. We demonstrate, both theoretically and by numerical examples, that when seeking for medium-accuracy solutions of large-scale ℓ 1 minimization problems, our randomized algorithms outperform significantly (and progressively as the sizes of the problem grow) the state-of-the art deterministic methods.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Azuma K.: Weighted sums of certain dependent random variables. Tökuku Math. J. 19, 357–367 (1967)
Candès E.J., Tao T.: Decoding by linear programming. IEEE Trans. Inform. Theory 51, 4203–4215 (2006)
Candès E.J.: Compressive sampling. In: Sanz-Solé, M., Soria, J., Varona, J.L., Verdera, J. (eds) International Congress of Mathematicians, Madrid 2006, vol. III, pp. 1437–1452. European Mathematical Society Publishing House, Zurich (2006)
Dalalyan A.S., Juditsky A., Spokoiny V.: A new algorithm for estimating the effective dimension-reduction subspace. J. Mach. Learn. Res. 9, 1647–1678 (2008)
Donoho D., Huo X.: Uncertainty principles and ideal atomic decomposition. IEEE Trans. Inform. Theory 47(7), 2845–2862 (2001)
Dümbgen L., van de Geer S., Veraar M., Wellner J.: Nemirovski’s inequalities revisited. Am. Math. Mon. 117(2), 138–160 (2010)
Grigoriadis M.D., Khachiyan l.G.: A sublinear-time randomized approximation algorithm for matrix games. Oper. Res. Lett. 18, 53–58 (1995)
Hoeffding W.: Probability inequalities for sums of bounded random variables. J. Am. Stat. Assoc. 58(301), 13–30 (1963)
Judistky, A., Nemirovski, A.: Large deviations of vector-valued martingales in 2-smooth normed spaces (2008) E-print: http://www2.isye.gatech.edu/~nemirovs/LargeDevSubmitted.pdf
Juditsky, A., Nemirovski, A., Tauvel, C.: Solving variational inequalities with stochastic mirror prox algorithm. Stoch. Syst. 1(1), 17–58 (2011)
Lemaréchal C., Nemirovski A., Nesterov Y.: New variants of bundle methods. Math. Program. 69(1), 111–148 (1995)
Nemirovskii, A., Yudin, D.: Efficient methods for large-scale convex problems. Ekonomika i Matematicheskie Metody (in Russian), 15(1), 135–152 (1979)
Nemirovski A.: Prox-method with rate of convergence O(1/t) for variational inequalities with Lipschitz continuous monotone operators and smooth convex-concave saddle point problems. SIAM J. Optim. 15, 229–251 (2004)
Nemirovski A., Juditsky A., Lan G., Shapiro A.: Stochastic approximation approach to stochastic programming. SIAM J. Optim. 19(4), 1574–1609 (2009)
Nesterov Y.: Smooth minimization of non-smooth functions—CORE Discussion Paper 2003/12, February 2003. Math. Progr. 103, 127–152 (2005)
Pinelis I.: Optimum bounds for the distributions of martingales in banach spaces. Ann. Probab. 22(4), 1679–1706 (1994)
Author information
Authors and Affiliations
Corresponding author
Additional information
Research was partly supported by the ONR grant N000140811104 and the NSF grant DMS-0914785 (second and third authors) and the BSF grant 2008302 (third author).
Rights and permissions
About this article
Cite this article
Juditsky, A., Kılınç Karzan, F. & Nemirovski, A. Randomized first order algorithms with applications to ℓ 1-minimization. Math. Program. 142, 269–310 (2013). https://doi.org/10.1007/s10107-012-0575-2
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10107-012-0575-2