Abstract
As an extension of partial least squares (PLS), kernel partial least squares (KPLS) is an very important methods to find nonlinear patterns from data. However, the application of KPLS to large-scale problems remains a big challenge, due to storage and computation issues in the number of examples. To address this limitation, we utilize randomness to design scalable new variants of the kernel matrix to solve KPLS. Specifically, we consider the spectral properties of low-rank kernel matrices constructed as sums of random feature dot-products and present a new method called randomized kernel partial least squares (RKPLS) to approximate KPLS. RKPLS can alleviate the computation requirements of approximate KPLS with linear space and computation in the sample size. Theoretical analysis and experimental results show that the solution of our algorithm converges to exact kernel matrix in expectation.
Similar content being viewed by others
Explore related subjects
Discover the latest articles, news and stories from top researchers in related subjects.References
Wold, S., Ruhe, A., Wold, H., Dunn, W.J.: The collinearity problem in linear regression. The partial least squares (PLS) approach to generalized inverse. SIAM J Sci Stat. Comput. 5, 735–743 (1984)
Helland, S.: On the structure of partial least squares regression. Commun Stat-Simul Comput. 17(2), 581–607 (1988)
Pan, W.T.: A newer equal part linear regression model : a case study of the influence of educational input on gross national income. EURASIA J Math SciTechn Educ. 13(8), 5765–5773 (2017)
Fang, X., Xu, Y., Li, X., Lai, Z., Wong, W.K., Fang, B.: Regularized label relaxation linear regression. IEEE Trans Neural Netw Learn Syst. 29(4), 1006–1018 (2018)
Dong, Y., Tao, D., Li, X., Ma, J., Pu, J.: Texture classification and retrieval using shearlets and linear regression. IEEE Trans Cybern. 45(3), 358–369 (2015)
Frank, I., Friedman, J.: A statistical view of some chemometrics regression tools. Technometrics. 35, 109–148 (1993)
Krämer, N. and Braun, M.: Kernelizing PLS, degrees of freedom, and efficient model selection. In Proceedings of the 24th International Conference on Machine Learning, pp.441–448 (2007)
Rosipal, R., Krämer, N.: Overview and recent advances in partial least squares. Lect. Notes Comput. Sci. 3940, 34–51 (2006)
Rosipal, R., Trejo, L.J.: Kernel partial least squares regression in reproducing kernel Hilbert space. J. Mach. Learn. Res. 2, 97–123 (2001)
Schölkopf, B., Smola, A.J.: Learning with Kernels, 1st edn. MIT Press, Cambridge (2001)
Rosipal, R.: Kernel partial least squares for nonlinear regression and discrimination. Neural Network World. 13, 291–300 (2003)
Talukdar, U., Hazarika, S.M., Gan, J.Q.: A kernel partial least square based feature selection method. Pattern Recogn. 83, 91–106 (2018)
Blanchard, G., Krämer, N.: Kernel partial least squares is universally consistent. In proceedings of the 13th international conference on artificial intelligence and statistics. JMLR Workshop Conf Proc. 9, 57–64 (2010)
Rahimi, A. and Recht, B.: Weighted sums of random kitchen sinks: Replacing minimization with randomization in learning. In Advances in Neural Information Processing Systems, pp.1313–1320 (2008)
Le, Q., Sarlós, T., and Smola, A.: Fastfood-approximating kernel expansions in loglinear time. In Proceedings of the 30th international conference on machine learning, Atlanta, USA, JMLR Workshop and Conference Proceedings, vol.28 (2013)
Mackey, L., Jordan, M.I., Chen, R.Y., Farrell, B., Tropp, J.A.: Matrix concentration inequalities via the method of exchangeable pairs. Ann. Probab. (2014)
Lopez-Paz, D., Sra, S., Smola, A., Ghahramani, Z., and Schölkopf, B. : Randomized nonlinear component analysis. In Proceedings of the 31st International Conference on Machine Learning, pp.1359–1367 (2014)
De Bie, T., Cristianini, N., and Rosipal, R. : Eigenproblems in pattern recognition. In Handbook of Geometric Computing, pp.129–170 (2005)
Bennett, K. and Embrechts, M. : An optimization perspective on kernel partial least squares. In Advances in Learning Theory: Methods, Models and Applications, NATO Science Series III: Computer and Systems Sciences, vol.190, IOS Press Amsterdam, pp. 227–250 (2003)
Boyd. S and Vandenberghe. L. Convex Optimization. Cambridge University Press ( 2004)
Momma, M. and Bennett, K.P.: Sparse kernel partial least squares regression. In Proceedings of the Sixteenth Annual Conference on Learning Theory, pp.216–230 (2003)
Aronszajn, N.: Theory of reproducing kernels. Trans. Am. Math. Soc. 68, 337–404 (1950)
Loosli, G., Canu, S., and Bottou, L. : Training invariant support vector machines using selective sampling. In Large Scale Kernel Machines, pp.301–320. MIT Press (2007)
Rahimi, A. and Recht, B. : Random features for large-scale kernel machines. In Advances in Neural Information Processing Systems (NIPS), vol. 20, pp.1177–1184 (2007)
Williams, C. and Seeger, M. : Using the Nyström method to speed up kernel machines. In Adavances in Neural Infromation Processing Systems, pp.682–688, Cambridge, MA (2001)
Si, S., Hsieh, C.-J., Dhillon, I.: Memory efficient kernel approximation. J. Mach. Learn. Res. 18(20), 1–32 (2017)
Singer, M., Krivobokova, T., Munk, A.: Kernel partial least squares for stationary data. J. Mach. Learn. Res. 18(123), 1–41 (2017)
Yang, T., Li, Y.-F., Mahdavi, M., Jin, R., and Zhou, Z.-H.: Nyström method vs random Fourier features: A theoretical and empirical comparison. In Advances in Neural Information Processing Systems, pp.485–493 (2012)
Durand, A., Hannula, M., Kontinen, J., Meier, A., and Virtema, J.: Approximation and dependence via multiteam semantics. Annals of Mathematics and Artificial Intelligence, pp.1–24 (2018)
Lin,S. and Zhou, D. :Optimal learning rates for kernel partial least squares. Journal of Fourier Analysis and Applications, 24(3):908–933 (2017)
UCI machine learning repository. [Online] Available: http://archive.ics.uci.edu/ml
LeCun, Y. and Cortes, C. : The MNIST database of handwritten digits. [Online] Available: http://yann.lecun.com/exdb/mnist/
Acknowledgements
The authors would like to thank the anonymous reviewers for their considerations and suggestions. We would also give thanks to the National Natural Science Foundation of China under grans no.61772020 for supporting our research work.
Author information
Authors and Affiliations
Corresponding author
Additional information
Publisher’s note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
About this article
Cite this article
Liu, X., Zhou, S. Approximate kernel partial least squares. Ann Math Artif Intell 88, 973–986 (2020). https://doi.org/10.1007/s10472-020-09694-3
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10472-020-09694-3