Abstract
It has been shown that Kernel Based Regression (KBR) with a least squares loss has some undesirable properties from robustness point of view. KBR with more robust loss functions, e.g. Huber or logistic losses, often give rise to more complicated computations. In this work the practical consequences of this sensitivity are explained, including the breakdown of Support Vector Machines (SVM) and weighted Least Squares Support Vector Machines (LS-SVM) for regression. In classical statistics, robustness is improved by reweighting the original estimate. We study the influence of reweighting the LS-SVM estimate using four different weight functions. Our results give practical guidelines in order to choose the weights, providing robustness and fast convergence. It turns out that Logistic and Myriad weights are suitable reweighting schemes when outliers are present in the data. In fact, the Myriad shows better performance over the others in the presence of extreme outliers (e.g. Cauchy distributed errors). These findings are then illustrated on toy example as well as on a real life data sets.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
Edgeworth, F.Y.: On Observations Relating to Several Quantities. Hermathena 6, 279–285 (1887)
Tukey, J.W.: A survey of sampling from contaminated distributions. In: Olkin, I. (ed.) Contributions to Probability and Statistics, pp. 448–485. Stanford University Press, Stanford (1960)
Huber, P.J.: Robust Estimation of a Location Parameter. Ann. Math. Stat. 35, 73–101 (1964)
Hampel, F.R.: A General Definition of Qualitative Robustness. Ann. Math. Stat. 42, 1887–1896 (1971)
Huber, P.J.: Robust Statistics. Wiley, Chichester (1981)
Rousseeuw, P.J., Leroy, A.M.: Robust Regression and Outlier Detection. Wiley, Chichester (2003)
Maronna, R., Martin, D., Yohai, V.: Robust Statistics. Wiley, Chichester (2006)
Hampel, F.R., Ronchetti, E.M., Rousseeuw, P.J., Stahel, W.A.: Robust Statistics: The Approach Beased on Influence Functions. Wiley, Chichester (1986)
Christmann, A., Steinwart, I.: Consistency and Robustness of Kernel Based Regression in Convex Risk Minimization. Bernoulli 13(3), 799–819 (2007)
Debruyne, M., Christmann, A., Hubert, M., Suykens, J.A.K.: Robustness and Stability of Reweighted Kernel Based Regression. Technical Report 06-09, Department of Mathematics, K.U.Leuven, Leuven, Belgium (2008)
Debruyne, M., Hubert, M., Suykens, J.A.K.: Model Selection in Kernel Based Regression using the Influence Function. J. Mach. Learn. Res. 9, 2377–2400 (2008)
Suykens, J.A.K., Van Gestel, T., De Brabanter, J., De Moor, B., Vandewalle, J.: Least Squares Support Vector Machines. World Scientific, Singapore (2002)
Vapnik, V.N.: Statistical Learning Theory. Wiley, Chichester (1999)
Suykens, J.A.K., De Brabanter, J., Lukas, L., Vandewalle, J.: Weighted Least Squares Support Vector Machines: Robustness and Sparse Approximation. Neurocomputing 48(1-4), 85–105 (2002)
Arce, G.R.: Nonlinear Signal Processing: A Statistical Approach. Wiley, Chichester (2005)
Gonzalez, J.G., Arce, G.R.: Weighted Myriad Filters: A Robust Filtering Framework derived from Alpha-Stable Distributions. In: Proceedings of the 1996 IEEE Conference one Acoustics (1996)
Hubert, M., Rousseeuw, P.J., Vanden Branden, K.: ROBPCA: a New Approach to Robust Principal Components Analysis. Technometrics 47, 64–79 (2005)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2009 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
De Brabanter, K. et al. (2009). Robustness of Kernel Based Regression: A Comparison of Iterative Weighting Schemes. In: Alippi, C., Polycarpou, M., Panayiotou, C., Ellinas, G. (eds) Artificial Neural Networks – ICANN 2009. ICANN 2009. Lecture Notes in Computer Science, vol 5768. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-04274-4_11
Download citation
DOI: https://doi.org/10.1007/978-3-642-04274-4_11
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-642-04273-7
Online ISBN: 978-3-642-04274-4
eBook Packages: Computer ScienceComputer Science (R0)