Block Bregman Majorization Minimization with Extrapolation
Abstract
In this paper, we consider a class of nonsmooth nonconvex optimization problems whose objective is the sum of a block relative smooth function and a proper and lower semicontinuous block separable function. Although the analysis of block proximal gradient (BPG) methods for the class of block $L$-smooth functions has been successfully extended to Bregman BPG methods that deal with the class of block relative smooth functions, accelerated Bregman BPG methods are scarce and challenging to design. Taking our inspiration from Nesterov-type acceleration and the majorization-minimization scheme, we propose a block alternating Bregman majorization minimization framework with extrapolation (BMME). We prove subsequential convergence of BMME to a first-order stationary point under mild assumptions and study its global convergence under stronger conditions. We illustrate the effectiveness of BMME on the penalized orthogonal nonnegative matrix factorization problem.
Keywords
MSC codes
Get full access to this article
View all available purchase options and get full access to this article.
Supplementary Material
PLEASE NOTE: These supplementary files have not been peer-reviewed.
Index of Supplementary Materials
Title of paper: Block Bregman Majorization Minimization with Extrapolation
Authors: Le Thi Khanh Hien, Duy Nhat Phan, Nicolas Gillis, Masoud Ahookhosh, and Panagiotis Patrinos
File: supplement.pdf
Type: PDF
Contents: This supplementary material provides additional numerical experiments on using BMME (our proposed algorithmic framework) to solve orthogonal NMF for facial features extraction and document classification.
It also contains a comparison between BMME and CoCaIn on the matrix completion problem, and the pseudo-code for two competing algorithms, namely A-BPALM and BIBPA.
References
1.
M. Aharon, M. Elad, and A. Bruckstein, K-SVD: An algorithm for designing overcomplete dictionaries for sparse representation, IEEE Trans. Signal Process., 54 (2006), pp. 4311--4322.
2.
M. Ahookhosh, L. T. K. Hien, N. Gillis, and P. Patrinos, A block inertial Bregman proximal algorithm for nonsmooth nonconvex problems with application to symmetric nonnegative matrix tri-factorization, J. Optim. Theory Appl., 190 (2021), pp. 234--258.
3.
M. Ahookhosh, L. T. K. Hien, N. Gillis, and P. Patrinos, Multi-block Bregman proximal alternating linearized minimization and its application to sparse orthogonal nonnegative matrix factorization, Comput. Optim. Appl., 79 (2021), 681–715.
4.
M. Ahookhosh, A. Themelis, and P. Patrinos, Bregman forward-backward splitting for nonconvex composite optimization: Superlinear convergence to nonisolated critical points, SIAM J. Optim., 31 (2021), pp. 653--685.
5.
U. Araújo, B. Saldanha, R. Galva͂o, T. Yoneyama, H. Chame, and V. Visani, The successive projections algorithm for variable selection in spectroscopic multicomponent analysis, Chemometrics and Intelligent Laboratory Systems, 57 (2001), pp. 65--73.
6.
H. Attouch and J. Bolte, On the convergence of the proximal algorithm for nonsmooth functions involving analytic features, Math. Program., 116 (2009), pp. 5--16, https://doi.org/10.1007/s10107-007-0133-5.
7.
H. Attouch, J. Bolte, P. Redont, and A. Soubeyran, Proximal alternating minimization and projection methods for nonconvex problems: An approach based on the Kurdyka-Łojasiewicz inequality, Math. Oper. Res., 35 (2010), pp. 438--457, https://doi.org/10.1287/moor.1100.0449.
8.
H. Attouch, J. Bolte, and B. F. Svaiter, Convergence of descent methods for semi-algebraic and tame problems: Proximal algorithms, forward--backward splitting, and regularized gauss--seidel methods, Math. Program., 137 (2013), pp. 91--129.
9.
H. H. Bauschke, J. Bolte, and M. Teboulle, A descent lemma beyond Lipschitz gradient continuity: First-order methods revisited and applications, Math. Oper. Res., 42 (2017), pp. 330--348, https://doi.org/10.1287/moor.2016.0817.
10.
A. Beck and L. Tetruashvili, On the convergence of block coordinate descent type methods, SIAM J. Optim., 23 (2013), pp. 2037--2060.
11.
T. Blumensath and M. E. Davies, Iterative hard thresholding for compressed sensing, Appl. Comput. Harmon. Anal., 27 (2009), pp. 265--274, https://doi.org/10.1016/j.acha.2009.04.002.
12.
J. Bochnak, M. Coste, and M.-F. Roy, Real Algebraic Geometry, Springer, New York, 1998.
13.
J. Bolte, S. Sabach, and M. Teboulle, Proximal alternating linearized minimization for nonconvex and nonsmooth problems, Math. Program., 146 (2014), pp. 459--494.
14.
J. Bolte, S. Sabach, M. Teboulle, and Y. Vaisbourd, First order methods beyond convexity and Lipschitz gradient continuity with applications to quadratic inverse problems, SIAM J. Optim., 28 (2018), pp. 2131--2151, https://doi.org/10.1137/17M1138558.
15.
N. Gillis, Nonnegative Matrix Factorization, SIAM, Philadelphia, 2020.
16.
N. Gillis, D. Kuang, and H. Park, Hierarchical clustering of hyperspectral images using rank-two nonnegative matrix factorization, IEEE Trans. Geosci. Remote Sensing, 53 (2015), pp. 2066--2078.
17.
N. Gillis and S. A. Vavasis, Fast and robust recursive algorithms for separable nonnegative matrix factorization, IEEE Tran. Pattern Anal. Machine Intelligence, 36 (2014), pp. 698--714, https://doi.org/10.1109/TPAMI.2013.226.
18.
L. T. K. Hien and N. Gillis, Algorithms for nonnegative matrix factorization with the Kullback-Leibler divergence, J. Sci. Comput., 87 (2021), 93, https://doi.org/10.1007/s10915-021-01504-0.
19.
L. T. K. Hien, N. Gillis, and P. Patrinos, Inertial block proximal method for non-convex non-smooth optimization, in Proceedings of the 37th International Conference on Machine Learning, 2020.
20.
L. T. K. Hien, D. N. Phan, and N. Gillis, Inertial Block Majorization Minimization Framework for Nonconvex Nonsmooth Optimization. arXiv:2010.12133, 2020.
21.
K. Kurdyka, On gradients of functions definable in o-minimal structures, Ann. Inst. Fourier, 48 (1998), pp. 769--783, https://doi.org/10.5802/aif.1638.
22.
H. Lu, R. M. Freund, and Y. Nesterov, Relatively smooth convex optimization by first-order methods, and applications, SIAM J. Optim., 28 (2018), pp. 333--354.
23.
J. Mairal, Optimization with first-order surrogate functions, in Proceedings of the 30th International Conference on International Conference on Machine Learning, Volume 28, JMLR, 2013, pp. 783--791.
24.
M. C. Mukkamala, P. Ochs, T. Pock, and S. Sabach, Convex-concave backtracking for inertial Bregman proximal gradient algorithms in nonconvex optimization, SIAM J. Math. Data Sci., 2 (2020), pp. 658--682, https://doi.org/10.1137/19M1298007.
25.
B. Natarajan, Sparse approximate solutions to linear systems, SIAM J. Comput., 24 (1995), pp. 227--234, https://doi.org/10.1137/S0097539792240406.
26.
Y. Nesterov, A method of solving a convex programming problem with convergence rate O$(1/k^2)$, Soviet Math. Dokl., 27 (1983).
27.
Y. Nesterov, Lectures on Convex Optimization, 2nd ed., Springer, New York, 2018.
28.
P. Ochs, Unifying abstract inexact convergence theorems and block coordinate variable metric ipiano, SIAM J. Optim., 29 (2019), pp. 541--570, https://doi.org/10.1137/17M1124085.
29.
T. Pock and S. Sabach, Inertial proximal alternating linearized minimization (iPALM) for nonconvex and nonsmooth problems, SIAM J. Imaging Sci., 9 (2016), pp. 1756--1787, https://doi.org/10.1137/16M1064064.
30.
F. Pompili, N. Gillis, P.-A. Absil, and F. Glineur, Two algorithms for orthogonal nonnegative matrix factorization with application to clustering, Neurocomputing, 141 (2014), pp. 15--25.
31.
M. Razaviyayn, M. Hong, and Z. Luo, A unified convergence analysis of block successive minimization methods for nonsmooth optimization, SIAM J. Optim., 23 (2013), pp. 1126--1153, https://doi.org/10.1137/120891009.
32.
M. Teboulle and Y. Vaisbourd, Novel proximal gradient methods for nonnegative matrix factorization with sparsity constraints, SIAM J. Imaging Sci., 13 (2020), pp. 381--421, https://doi.org/10.1137/19M1271750.
33.
P. Tseng and S. Yun, A coordinate gradient descent method for nonsmooth separable minimization, Math. Program., 117 (2009), pp. 387--423.
34.
Y. Xu and W. Yin, A block coordinate descent method for regularized multiconvex optimization with applications to nonnegative tensor factorization and completion, SIAM J. Imaging Sci., 6 (2013), pp. 1758--1789, https://doi.org/10.1137/120887795.
35.
Y. Xu and W. Yin, A fast patch-dictionary method for whole image recovery, Inverse Problems Imaging, 10 (2016), pp. 563--583, https://doi.org/10.3934/ipi.2016012.
36.
Y. Xu and W. Yin, A globally convergent algorithm for nonconvex optimization based on block coordinate update, J. Sci. Comput., 72 (2017), pp. 700--734.
37.
S. Zhong and J. Ghosh, Generative model-based document clustering: a comparative study, Knowledge Inform. Syst., 8 (2005), pp. 374--384.
Information & Authors
Information
Published In
SIAM Journal on Mathematics of Data Science
Pages: 1 - 25
DOI: 10.1137/21M1432661
ISSN (online): 2577-0187
Copyright
© 2022, Society for Industrial and Applied Mathematics.
History
Submitted: 9 July 2021
Accepted: 21 September 2021
Published online: 13 January 2022
Keywords
MSC codes
Authors
Funding Information
European Research Council : 679515
Fonds De La Recherche Scientifique - FNRS https://doi.org/10.13039/501100002661
Fonds Wetenschappelijk Onderzoek https://doi.org/10.13039/501100003130 : O005318F-RG47
Metrics & Citations
Metrics
Citations
If you have the appropriate software installed, you can download article citation data to the citation manager of your choice. Simply select your manager software from the list below and click Download.
Cited By
There are no citations for this item
View Options
Get Access
- Access via your Institution
- Questions about how to access this content? Contact SIAM at [email protected].