A robust stochastic quasi-Newton algorithm for non-convex machine learning

Abstract

Stochastic quasi-Newton methods have garnered considerable attention within large-scale machine learning optimization. Nevertheless, the presence of a stochastic gradient equaling zero poses a significant obstacle to updating the quasi-Newton matrix, thereby impacting the stability of the quasi-Newton algorithm. To address this issue, a checkpoint mechanism is introduced, i.e., checking the value of $\textbf{s}_k$ before updating the quasi-Newton matrix, which effectively prevents zero increments in the optimization variable and enhances algorithmic stability during iterations. Meanwhile, a novel gradient incremental formulation is introduced to satisfy curvature conditions, facilitating convergence for non-convex objectives. Additionally, finite-memory techniques are employed to reduce storage requirements in large-scale machine learning tasks. The last iteration of the proposed algorithm is proven to converge in a non-convex setting, which is better than average and minimum iteration convergence. Finally, experiments are conducted on benchmark datasets to compare the proposed RSLBFGS algorithm with other popular first and second-order methods, demonstrating the effectiveness and robustness of RSLBFGS.