Abstract
Principal component analysis (PCA) is a widely used descriptive multivariate technique in the analysis of quantitative data. When applying PCA to mixed quantitative and qualitative data, we utilize an optimal scaling technique for quantifying qualitative data. PCA with optimal scaling is called nonlinear PCA. The alternating least squares (ALS) algorithm is used for computing nonlinear PCA. The ALS algorithm is stable in convergence and simple in implementation; however, the algorithm tends to converge slowly for large data matrices owing to its linear convergence. Then the v\(\varepsilon \)-ALS algorithm, which incorporates the vector \(\varepsilon \) accelerator into the ALS algorithm, is used to accelerate the convergence of the ALS algorithm for nonlinear PCA. In this paper, we improve the v\(\varepsilon \)-ALS algorithm via a restarting procedure and further reduce its number of iterations and computation time. The restarting procedure is performed under simple restarting conditions, and it speeds up the convergence of the v\(\varepsilon \)-ALS algorithm. The v\(\varepsilon \)-ALS algorithm with a restarting procedure is referred to as the v\(\varepsilon \)R-ALS algorithm. Numerical experiments examine the performance of the v\(\varepsilon \)R-ALS algorithm by comparing its number of iterations and computation time with those of the ALS and v\(\varepsilon \)-ALS algorithms.
Similar content being viewed by others
References
Gifi A (1990) Nonlinear multivariate analysis. England, Wiley, Chichester
Henderson NC, Varadhan R (2019) Damped Anderson acceleration with restarts and monotonicity control for accelerating EM and EM-like algorithms. J Comput Graph Stat 28:834–846
Krijnen WP (2006) Convergence of the sequence of parameters generated by alternating least squares algorithms. Comput Stat Data Anal 51:481–489
Kuroda M, Mori Y, Iizuka M, Sakakihara M (2011) Acceleration of the alternating least squares algorithm for principal components analysis. Comput Stat Data Anal 55:143–153
Kruskal JB (1964) Nonmetric multidimensional scaling: a numerical method. Psychometrika 29:115–129
Loisel S, Takane Y (2011) Generalized GIPSCAL re-visited: a fast convergent algorithm with acceleration by the minimal polynomial extrapolation. Adv Data Anal Classif 5:57–75
Michailidis G, de Leeuw J (1998) The Gifi system of descriptive multivariate analysis. Stat Sci 13:307–336
R Core Team (2018) R: a language and environment for statistical computing. R Foundation for statistical computing, Vienna, Austria. URL http://www.R-project.org
Smith DA, Ford WF, Sidi A (1987) Extrapolation methods for vector sequences. SIAM Rev 29:199–233
Takane Y, Zhang Z (2009) Algorithms for DEDICOM: Acceleration, deceleration, or neither? J Chemomet 23:364–370
Takane Y, Jung K, Hwang H (2010) An acceleration method for Ten Berge et al.’s algorithm for orthogonal INDSCAL. Comput Stat 25(3):409–428
Young FW, Takane Y, de Leeuw J (1978) Principal components of mixed measurement level multivariate data: an alternating least squares method with optimal scaling features. Psychometrika 43:279–281
Wynn P (1962) Acceleration techniques for iterated vector and matrix problems. Math Comput 16:301–322
Acknowledgements
The authors would like to thank the editor and two referees for their valuable comments and helpful suggestions that led to an improvement of this paper.
Author information
Authors and Affiliations
Corresponding author
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Appendices
A: The pseudo-code of the v\(\varepsilon \)R-ALS algorithm
# Initialization
Set the initial value \({\varvec{\theta }}_{0}\) of \({\varvec{\theta }}\) and the initial values of the other parameters of the ALS step. Specify \(\delta \), \(\delta _{Re} (>\delta )\), and D. Determine the maximum number of iterations (max.itr).
# v\(\varepsilon \)R-ALS iterations
\({\varvec{\theta }}_{1} \leftarrow A({\varvec{\theta }}_{0})\)
\(\dot{{\varvec{\theta }}}_{old} \leftarrow {\varvec{\theta }}_{1}\)
\(itr \leftarrow 0\)
repeat
\(itr \leftarrow itr+1\)
# The ALS step
\({\varvec{\theta }_{2}} \leftarrow A({\varvec{\theta }}_{1})\)
# The Acceleration step
\(\dot{{\varvec{\theta }}}_{new} \leftarrow {\varvec{\theta }}_{1} + \left[ \left[ {\varvec{\theta }}_{2}-{\varvec{\theta }}_{1} \right] ^{-1}- \left[ {\varvec{\theta }}_{1}-{\varvec{\theta }}_{0}\right] ^{-1} \right] ^{-1}\)
# The restarting procedure
if \(\Vert \dot{{\varvec{\theta }}}_{new}-\dot{{\varvec{\theta }}}_{old}\Vert ^{2} < \delta _{Re}\) then
if \(\Vert \dot{{\varvec{\theta }}}_{new}-\dot{{\varvec{\theta }}}_{old}\Vert ^{2} < \delta \) or \(itr > max.itr\) then
Terminate iterations
end if
Compute \(\sigma ^{(t+1)}\) and \(\dot{\sigma }^{(t-1)}\)
if \(\dot{\sigma }^{(t-1)} < \sigma ^{(t+1)}\) then
\({\varvec{\theta }}_{2} \leftarrow A(\dot{{\varvec{\theta }}}_{new})\)
\({\varvec{\theta }}_{1} \leftarrow \dot{{\varvec{\theta }}}_{new}\)
\(\delta _{Re} \leftarrow \delta _{Re} /D\)
end if
end if
\(\dot{{\varvec{\theta }}}_{old} \leftarrow \dot{{\varvec{\theta }}}_{new}\)
\({\varvec{\theta }}_{0} \leftarrow {\varvec{\theta }}_{1}\)
\({\varvec{\theta }}_{1} \leftarrow {\varvec{\theta }}_{2}\)
end repeat
B: The procedure of the v\(\varepsilon \)ER-ALS algorithm
For specified \(\delta \) and given initial values of the ALS step, the v\(\varepsilon \)ER-ALS algorithm performs the following steps:
ALS step: Obtain
v\(\varvec{\varepsilon }\)-acceleration step: Calculate \(\dot{{\varvec{\theta }}}^{(t-1)}\) from \(({\varvec{\theta }}^{(t+1)},{\varvec{\theta }}^{(t)},{\varvec{\theta }}^{(t-1)})\) using
restarting step: Set
and update
Check the convergence by
where \(\delta \) is a desired accuracy.
Rights and permissions
About this article
Cite this article
Kuroda, M., Mori, Y. & Iizuka, M. Speeding up the convergence of the alternating least squares algorithm using vector \(\varepsilon \) acceleration and restarting for nonlinear principal component analysis. Comput Stat 38, 243–262 (2023). https://doi.org/10.1007/s00180-022-01225-4
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s00180-022-01225-4