Abstract
Regularized Least-Squares Classification (RLSC) can be regarded as a kind of 2 layers neural network using regularized square loss function and kernel trick. Poggio and Smale recently reformulated it in the framework of the mathematical foundations of learning and called it a key algorithm of learning theory. The generalization performance of RLSC depends heavily on the setting of its kernel and hyper parameters. Therefore we presented a novel two-step approach for optimal parameters selection: firstly the optimal kernel parameters are selected by maximizing kernel target alignment, and then the optimal hyper-parameter is determined via minimizing RLSC’s leave-one-out bound. Compared with traditional grid search, our method needs no independent validation set. We worked on IDA’s benchmark datasets using Gaussian kernel, the results demonstrate that our method is feasible and time efficient.
Keywords
- Grid Search
- Layer Neural Network
- Hyper Parameter
- Kernel Ridge Regression
- Proximal Support Vector Machine
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.
This is a preview of subscription content, log in via an institution.
Buying options
Tax calculation will be finalised at checkout
Purchases are for personal use only
Learn about institutional subscriptionsPreview
Unable to display preview. Download preview PDF.
References
Poggio, T., Smale, S.: The Mathematics of Learning: Dealing With Data. Notice of American Mathematical Society 5, 537–544 (2003)
Cucker, F., Smale, S.: On the Mathematical Foundations of Learning. Bulletin of American Mathematical Society 39, 1–49 (2001)
Vapnik, V.: The Nature of Statistical Learning Theory, 2nd edn. Springer, Heidelberg (2000)
Rifkin, R.M.: Everything Old Is New Again: A Fresh Look at Historical Approaches to Machine Learning. PhD thesis, Massachusetts Institute of Technology (2002)
Rifkin, R.M., Yeo, G., Poggio, T.: Regularized Least-Squares Classification. In: Advances in Learning Theory: Methods, Model and Applications. NATO Science Series III: Computer and Systems Sciences, vol. l, p. 190. IOS Press, Amsterdam (2003)
Suykens, J.A.K., Gestel, V., De Bra banter, J., et al.: Least Squares Support Vector Machine. World Scientific, Singapore (2002)
Fung, G., Mangasarian, O.L.: Proximal Support Vector Machine Classifiers. In: KDD 2001: Seventh ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, San Francisco, CA (2001)
Chapelle, O., Vapnik, V., Mukjerjee, S.: Choosing Multiple Parameters for Support Vector Machines. Machine Learning 46, 131–159 (2002)
Cristianini, N., Kandola, J., Elisseeff, A., Shawe-Taylor, J.: On Kernel Target Alignment. Journal of Machine Learning Research (submitted)
Jaakkola, T., Haussler, D.: Probabilistic Kernel Regression Models. In: Advances in Neural Information Processing Systems, vol. 11 (1998)
Rätsch, G., Onoda, T., Müller, K.-R.: Soft Margin for AdaBoost. Machine Learning 1, 1–35 (2000)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2005 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Yang, HH., Wang, XY., Wang, Y., Gao, HH. (2005). Model Selection for Regularized Least-Squares Classification. In: Wang, L., Chen, K., Ong, Y.S. (eds) Advances in Natural Computation. ICNC 2005. Lecture Notes in Computer Science, vol 3610. Springer, Berlin, Heidelberg. https://doi.org/10.1007/11539087_72
Download citation
DOI: https://doi.org/10.1007/11539087_72
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-540-28323-2
Online ISBN: 978-3-540-31853-8
eBook Packages: Computer ScienceComputer Science (R0)