Abstract
In this paper, we incorporate importance sampling strategy into accelerated framework of stochastic alternating direction method of multipliers for solving a class of stochastic composite problems with linear equality constraint. The rates of convergence for primal residual and feasibility violation are established. Moreover, the estimation of variance of stochastic gradient is improved due to the use of important sampling. The proposed algorithm is capable of dealing with the situation, where the feasible set is unbounded. The experimental results indicate the effectiveness of the proposed method.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Boyd, S., Parikh, N., Chu, E., Peleato, B., Eckstein, J.: Distributed optimization and statistical learning via the alternating direction method of multipliers. Found. Trends Mach. Learn. 3(1), 1–122 (2011)
Azadi, S., Sra, S.: Towards an optimal stochastic alternating direction method of multipliers. In: Proceedings of the 31st ICML, pp. 620–628 (2014)
Ouyang, H., He, N., Tran, L., Gray, A.: Stochastic alternating direction method of multipliers. In: Proceedings of the 30th ICML, pp. 80–88 (2013)
Suzuki, T.: Dual averaging and proximal gradient descent for online alternating direction multiplier method. In: Proceedings of the 30th ICML, pp. 392–400 (2013)
Wang, H., Banerjee, A.: Online alternating direction method. arXiv preprint arXiv:1306.3721 (2013)
Strohmer, T., Vershynin, R.: A randomized Kaczmarz algorithm with exponential convergence. J. Fourier Anal. Appl. 15(2), 262–278 (2009)
Nesterov, Y.: Efficiency of coordinate descent methods on huge-scale optimization problems. SIAM J. Optim. 22(2), 341–362 (2012)
Needell, D., Srebro, N., Ward, R.: Stochastic gradient descent and the randomized Kaczmarz algorithm. arXiv preprint arXiv:1310.5715 (2013)
Schmidt, M., Le Roux, N., Bach, F.: Minimizing finite sums with the stochastic average gradient. Math. Program. 162, 83–112 (2013)
Zhao, P., Zhang, T.: Stochastic optimization with importance sampling for regularized loss minimization. In: International Conference on Machine Learning, pp. 1–9 (2015)
Chen, Y., Lan, G., Ouyang, Y.: Optimal primal-dual methods for a class of saddle point problems. SIAM J. Optim. 24(4), 1779–1814 (2014)
Jacob, L., Obozinski, G., Vert, J.P.: Group lasso with overlap and graph lasso. In: Proceedings of the 26th ICML, pp. 433–440. ACM (2009)
Tomioka, R., Hayashi, K., Kashima, H.: Estimation of low-rank tensors via convex optimization. arXiv preprint arXiv:1010.0789 (2010)
Goldstein, T., Osher, S.: The split bregman method for l1-regularized problems. SIAM J. Imaging Sci. 2(2), 323–343 (2009)
Goldfarb, D., Ma, S., Scheinberg, K.: Fast alternating linearization methods for minimizing the sum of two convex functions. Math. Program. 141, 349–382 (2013)
Touzi, N.: Stochastic control and application to finance. Pisa. Special Research Semester on Financial Mathematics, Scuola Normale Superiore (2002)
Ziemba, W.T., Vickson, R.G.: Stochastic Optimization Models in Finance. World Scientific, Singapore (1975)
Nesterov, Y.: A method of solving a convex programming problem with convergence rate o (1/k2). Sov. Math. Doklady 27(2), 372–376 (1983)
Nesterov, Y.: Introductory Lectures on Convex Optimization, vol. 87. Springer, Berlin (2004)
Zhao, S.Y., Li, W.J., Zhou, Z.H.: Scalable stochastic alternating direction method of multipliers. arXiv preprint arXiv:1502.03529 (2015)
Zhang, C., Shen, Z., Qian, H., Zhou, T.: Accelerated stochastic ADMM with variance reduction. arXiv preprint arXiv:1611.04074 (2016)
Zheng, S., Kwok, J.T.: Stochastic variance-reduced ADMM. arXiv preprint arXiv:1604.07070 (2016)
Liu, Y., Shang, F., Cheng, J.: Accelerated variance reduced stochastic ADMM. In: AAAI, pp. 2287–2293 (2017)
Shapiro, A.: Monte carlo sampling methods. Handb. Oper. Res. Manag. Sci. 10, 353–425 (2003)
Shapiro, A., Nemirovski, A.: On complexity of stochastic programming problems. In: Continuous Optimization, pp. 111–146. Springer (2005)
Nemirovski, A., Juditsky, A., Lan, G., Shapiro, A.: Robust stochastic approximation approach to stochastic programming. SIAM J. Optim. 19(4), 1574–1609 (2009)
Rockafellar, R.T.: Convex Analysis. Princeton University Press, Princeton (1970)
Lan, G., Nemirovski, A., Shapiro, A.: Validation analysis of mirror descent stochastic approximation method. Math. Program. 134(2), 425–458 (2012)
Esser, E., Zhang, X., Chan, T.F.: A general framework for a class of first order primal-dual algorithms for convex optimization in imaging science. SIAM J. Imaging Sci. 3(4), 1015–1046 (2010)
Moreau, J.J.: Décomposition orthogonale dun espace hilbertien selon deux cônes mutuellement polaires. CR Acad. Sci. Paris 255, 238–240 (1962)
Ouyang, Y., Chen, Y., Lan, G., Pasiliao Jr, E.: An accelerated linearized alternating direction method of multipliers. arXiv preprint arXiv:1401.6607 (2014)
Banerjee, O., Ghaoui, L.E., dAspremont, A.: Model selection through sparse maximum likelihood estimation for multivariate gaussian or binary data. J. Mach. Learn. Res. 9(Mar), 485–516 (2008)
Acknowledgements
Funding was provided by National Science Foundation (Grant No. DMS 1719932).
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Chen, C., Chen, Y., Ouyang, Y. et al. Stochastic Accelerated Alternating Direction Method of Multipliers with Importance Sampling. J Optim Theory Appl 179, 676–695 (2018). https://doi.org/10.1007/s10957-018-1270-0
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10957-018-1270-0