Skip to main content

Speeding up the convergence of the alternating least squares algorithm using vector \(\varepsilon \) acceleration and restarting for nonlinear principal component analysis

  • Original paper
  • Published:
Computational Statistics Aims and scope Submit manuscript

Abstract

Principal component analysis (PCA) is a widely used descriptive multivariate technique in the analysis of quantitative data. When applying PCA to mixed quantitative and qualitative data, we utilize an optimal scaling technique for quantifying qualitative data. PCA with optimal scaling is called nonlinear PCA. The alternating least squares (ALS) algorithm is used for computing nonlinear PCA. The ALS algorithm is stable in convergence and simple in implementation; however, the algorithm tends to converge slowly for large data matrices owing to its linear convergence. Then the v\(\varepsilon \)-ALS algorithm, which incorporates the vector \(\varepsilon \) accelerator into the ALS algorithm, is used to accelerate the convergence of the ALS algorithm for nonlinear PCA. In this paper, we improve the v\(\varepsilon \)-ALS algorithm via a restarting procedure and further reduce its number of iterations and computation time. The restarting procedure is performed under simple restarting conditions, and it speeds up the convergence of the v\(\varepsilon \)-ALS algorithm. The v\(\varepsilon \)-ALS algorithm with a restarting procedure is referred to as the v\(\varepsilon \)R-ALS algorithm. Numerical experiments examine the performance of the v\(\varepsilon \)R-ALS algorithm by comparing its number of iterations and computation time with those of the ALS and v\(\varepsilon \)-ALS algorithms.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Fig. 1
Fig. 2
Fig. 3

Similar content being viewed by others

References

  • Gifi A (1990) Nonlinear multivariate analysis. England, Wiley, Chichester

    MATH  Google Scholar 

  • Henderson NC, Varadhan R (2019) Damped Anderson acceleration with restarts and monotonicity control for accelerating EM and EM-like algorithms. J Comput Graph Stat 28:834–846

    Article  MathSciNet  MATH  Google Scholar 

  • Krijnen WP (2006) Convergence of the sequence of parameters generated by alternating least squares algorithms. Comput Stat Data Anal 51:481–489

    Article  MathSciNet  MATH  Google Scholar 

  • Kuroda M, Mori Y, Iizuka M, Sakakihara M (2011) Acceleration of the alternating least squares algorithm for principal components analysis. Comput Stat Data Anal 55:143–153

    Article  MathSciNet  MATH  Google Scholar 

  • Kruskal JB (1964) Nonmetric multidimensional scaling: a numerical method. Psychometrika 29:115–129

    Article  MathSciNet  MATH  Google Scholar 

  • Loisel S, Takane Y (2011) Generalized GIPSCAL re-visited: a fast convergent algorithm with acceleration by the minimal polynomial extrapolation. Adv Data Anal Classif 5:57–75

    Article  MathSciNet  MATH  Google Scholar 

  • Michailidis G, de Leeuw J (1998) The Gifi system of descriptive multivariate analysis. Stat Sci 13:307–336

    Article  MathSciNet  MATH  Google Scholar 

  • R Core Team (2018) R: a language and environment for statistical computing. R Foundation for statistical computing, Vienna, Austria. URL http://www.R-project.org

  • Smith DA, Ford WF, Sidi A (1987) Extrapolation methods for vector sequences. SIAM Rev 29:199–233

    Article  MathSciNet  MATH  Google Scholar 

  • Takane Y, Zhang Z (2009) Algorithms for DEDICOM: Acceleration, deceleration, or neither? J Chemomet 23:364–370

    Article  Google Scholar 

  • Takane Y, Jung K, Hwang H (2010) An acceleration method for Ten Berge et al.’s algorithm for orthogonal INDSCAL. Comput Stat 25(3):409–428

    Article  MathSciNet  MATH  Google Scholar 

  • Young FW, Takane Y, de Leeuw J (1978) Principal components of mixed measurement level multivariate data: an alternating least squares method with optimal scaling features. Psychometrika 43:279–281

    Article  MATH  Google Scholar 

  • Wynn P (1962) Acceleration techniques for iterated vector and matrix problems. Math Comput 16:301–322

    Article  MathSciNet  MATH  Google Scholar 

Download references

Acknowledgements

The authors would like to thank the editor and two referees for their valuable comments and helpful suggestions that led to an improvement of this paper.

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Masahiro Kuroda.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Appendices

A: The pseudo-code of the v\(\varepsilon \)R-ALS algorithm

# Initialization

Set the initial value \({\varvec{\theta }}_{0}\) of \({\varvec{\theta }}\) and the initial values of the other parameters of the ALS step. Specify \(\delta \), \(\delta _{Re} (>\delta )\), and D. Determine the maximum number of iterations (max.itr).

# v\(\varepsilon \)R-ALS iterations

\({\varvec{\theta }}_{1} \leftarrow A({\varvec{\theta }}_{0})\)

\(\dot{{\varvec{\theta }}}_{old} \leftarrow {\varvec{\theta }}_{1}\)

\(itr \leftarrow 0\)

repeat

\(itr \leftarrow itr+1\)

# The ALS step

\({\varvec{\theta }_{2}} \leftarrow A({\varvec{\theta }}_{1})\)

# The Acceleration step

\(\dot{{\varvec{\theta }}}_{new} \leftarrow {\varvec{\theta }}_{1} + \left[ \left[ {\varvec{\theta }}_{2}-{\varvec{\theta }}_{1} \right] ^{-1}- \left[ {\varvec{\theta }}_{1}-{\varvec{\theta }}_{0}\right] ^{-1} \right] ^{-1}\)

# The restarting procedure

if \(\Vert \dot{{\varvec{\theta }}}_{new}-\dot{{\varvec{\theta }}}_{old}\Vert ^{2} < \delta _{Re}\) then

if \(\Vert \dot{{\varvec{\theta }}}_{new}-\dot{{\varvec{\theta }}}_{old}\Vert ^{2} < \delta \) or \(itr > max.itr\) then

Terminate iterations

end if

Compute \(\sigma ^{(t+1)}\) and \(\dot{\sigma }^{(t-1)}\)

if \(\dot{\sigma }^{(t-1)} < \sigma ^{(t+1)}\) then

\({\varvec{\theta }}_{2} \leftarrow A(\dot{{\varvec{\theta }}}_{new})\)

\({\varvec{\theta }}_{1} \leftarrow \dot{{\varvec{\theta }}}_{new}\)

\(\delta _{Re} \leftarrow \delta _{Re} /D\)

end if

end if

\(\dot{{\varvec{\theta }}}_{old} \leftarrow \dot{{\varvec{\theta }}}_{new}\)

\({\varvec{\theta }}_{0} \leftarrow {\varvec{\theta }}_{1}\)

\({\varvec{\theta }}_{1} \leftarrow {\varvec{\theta }}_{2}\)

end repeat

B: The procedure of the v\(\varepsilon \)ER-ALS algorithm

For specified \(\delta \) and given initial values of the ALS step, the v\(\varepsilon \)ER-ALS algorithm performs the following steps:

ALS step: Obtain

$$\begin{aligned} {\varvec{\theta }}^{(t+1)} = A({\varvec{\theta }}^{(t)}). \end{aligned}$$

v\(\varvec{\varepsilon }\)-acceleration step: Calculate \(\dot{{\varvec{\theta }}}^{(t-1)}\) from \(({\varvec{\theta }}^{(t+1)},{\varvec{\theta }}^{(t)},{\varvec{\theta }}^{(t-1)})\) using

$$\begin{aligned} \dot{{\varvec{\theta }}}^{(t-1)} = {\varvec{\theta }}^{(t)} + \left[ \left[ \Delta {\varvec{\theta }}^{(t)} \right] ^{-1} - \left[ \Delta {\varvec{\theta }}^{(t-1)} \right] ^{-1} \right] ^{-1}. \end{aligned}$$

restarting step: Set

$$\begin{aligned} {\varvec{\theta }}^{(t)} = \dot{{\varvec{\theta }}}^{(t-1)}, \end{aligned}$$

and update

$$\begin{aligned} {\varvec{\theta }}^{(t+1)} = A(\dot{{\varvec{\theta }}}^{(t-1)}). \end{aligned}$$

Check the convergence by

$$\begin{aligned} \bigl \Vert \dot{{\varvec{\theta }}}^{(t-1)}-\dot{{\varvec{\theta }}}^{(t-2)} \bigr \Vert ^{2} < \delta , \end{aligned}$$

where \(\delta \) is a desired accuracy.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Kuroda, M., Mori, Y. & Iizuka, M. Speeding up the convergence of the alternating least squares algorithm using vector \(\varepsilon \) acceleration and restarting for nonlinear principal component analysis. Comput Stat 38, 243–262 (2023). https://doi.org/10.1007/s00180-022-01225-4

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00180-022-01225-4

Keywords