Abstract
The variable projection method is a classical and efficient method for solving separable nonlinear least squares (SNLLS) problems. However, it is hard to handle the constrained SNLLS problems since the explicit form of the Jacobian matrix is required in each iteration. In this paper, we propose a secant variable projection (SVP) method, which employs a rank-one update to estimate the Jacobian matrices. The main advantages of our method are efficiency and ease of applicability to constrained SNLLS problems. The local convergence of our SVP method is also analyzed. Finally, some data fitting and image processing problems are solved to compare the performance of our proposed method with the classical variable projection method. Numerical results illustrate the efficiency and stability of our proposed SVP method in solving the SNLLS problems arising from the blind deconvolution problems.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Berisha, S., Nagy, J.G.: Iterative methods for image restoration. In: Academic Press Library in Signal Processing, vol. 4, pp 193–247. Elsevier (2014)
Chung, J., Haber, E., Nagy, J.: Numerical methods for coupled super-resolution. Inverse Prob. 22(4), 1261 (2006)
Chung, J., Nagy, J.G.: An efficient iterative approach for large-scale separable nonlinear inverse problems. SIAM J. Sci. Comput. 31(6), 4654–4674 (2010)
Chung, J., Nagy, J.G., O’Leary, D.P.: A weighted GCV method for Lanczos hybrid regularization. Electronic Transactions on Numerical Analysis, 28(Electronic Transactions on Numerical Analysis) (2008)
Coleman, T.F., Xu, W.: Automatic differentiation in MATLAB using ADMAT with applications, vol. 27. SIAM (2016)
Cornelio, A., Piccolomini, E.L., Nagy, J.G.: Constrained numerical optimization methods for blind deconvolution. Numerical Algorithms 65(1), 23–42 (2014)
Dennis, J.E. Jr., Schnabel, R.B.: Numerical methods for unconstrained optimization and nonlinear equations, vol. 16. SIAM (1996)
Golub, G., Pereyra, V.: Separable nonlinear least squares: the variable projection method and its applications. Inverse Prob. 19(2), R1 (2003)
Golub, G.H., Pereyra, V.: The differentiation of pseudo-inverses and nonlinear least squares problems whose variables separate. SIAM J. Numer. Anal. 10(2), 413–432 (1973)
Hansen, P.C.: Discrete inverse problems: insight and algorithms, vol. 7. SIAM (2010)
Hansen, P.C., Nagy, J.G., O’leary, D.P.: Deblurring images: matrices, spectra, and filtering, vol. 3. SIAM (2006)
Hansen, P.C., Pereyra, V., Scherer, G.: Least squares data fitting with applications. John Hopkins University Press, Baltimore (2013)
Kaufman, L.: A variable projection method for solving separable nonlinear least squares problems. BIT Numer. Math. 15(1), 49–57 (1975)
Kaufman, L., Pereyra, V.: A method for separable nonlinear least squares problems with separable nonlinear equality constraints. SIAM J. Numer. Anal. 15 (1), 12–20 (1978)
Kim, C.-T., Lee, J.-J.: Training two-layered feedforward networks with variable projection method. IEEE Trans. Neural Netw. 19(2), 371–375 (2008)
Lawton, W.H., Sylvestre, E.A.: Elimination of linear parameters in nonlinear regression. Technometrics 13(3), 461–467 (1971)
Madsen, K., Nielsen, H.B., Tingleff, O.: Methods for nonlinear least squares problems. Informatics and Mathematical Modelling, Techincal University of Denmark (2004)
Mullen, K.M., Van Stokkum, I.H.M.: The variable projection algorithm in time-resolved spectroscopy, microscopy and mass spectrometry applications. Numerical Algorithms 51(3), 319–340 (2009)
Nagy, J.G., Palmer, K., Perrone, L.: Iterative methods for image deblurring: a Matlab object-oriented approach. Numerical Algorithms 36(1), 73–93 (2004)
Nagy, J.G., Ruthotto, L.: Lap: a linearize and project method for solving inverse problems with coupled variables. Imaging (MRI) 1, 6 (2018)
O’leary, D.P., Rust, B.W.: Variable projection for nonlinear least squares problems. Comput. Optim. Appl. 54(3), 579–593 (2013)
Ruhe, A., Wedin, P.A.: Algorithms for separable nonlinear least squares problems. Siam Rev. 22(3), 318–337 (1980)
Sima, D.M., Van Huffel, S.: Separable nonlinear least squares fitting with linear bound constraints and its application in magnetic resonance spectroscopy data quantification. Elsevier Science Publishers B. V. (2007)
Wold, H., Lyttkens, E.: Nonlinear iterative partial least squares (NIPALS) estimation procedures. Bull. Int. Stat. Inst. 43, 29–51 (1969)
Wright, S., Nocedal, J.: Numerical optimization. Springer Science 35(67-68), 7 (1999)
Xu, W., Coleman, T.F., Liu, G.: A secant method for nonlinear least-squares minimization. Comput. Optim. Appl. 51(1), 159–173 (2012)
Zheng, N., Hayami, K., Yin, J.-F.: Modulus-type inner outer iteration methods for nonnegative constrained least squares problems. SIAM J. Matrix Anal. Appl. 37(3), 1250–1278 (2016)
Acknowledgments
We would like to thank the three anonymous referees for their valuable comments and suggestion to make the current version more readable.
Funding
The work of Xiongfeng Song and Wei Xu is supported by National Natural Science Fund of China (U1811462, 71771175). The work of Ken Hayami is supported by JJPS KAKEHI Grant number 15K04768.
Author information
Authors and Affiliations
Corresponding author
Additional information
Publisher’s note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
About this article
Cite this article
Song, X., Xu, W., Hayami, K. et al. Secant variable projection method for solving nonnegative separable least squares problems. Numer Algor 85, 737–761 (2020). https://doi.org/10.1007/s11075-019-00835-2
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11075-019-00835-2