Abstract
Least squares kernel-based methods have been widely used in regression problems due to the simple implementation and good generalization performance. Among them, least squares support vector regression (LS-SVR) and extreme learning machine (ELM) are popular techniques. However, the noise sensitivity is a major bottleneck. To address this issue, a generalized loss function, called \(\ell _s\)-loss, is proposed in this paper. With the support of novel loss function, two kernel-based regressors are constructed by replacing the \(\ell _2\)-loss in LS-SVR and ELM with the proposed \(\ell _s\)-loss for better noise robustness. Important properties of \(\ell _s\)-loss, including robustness, asymmetry and asymptotic approximation behaviors, are verified theoretically. Moreover, iteratively reweighted least squares are utilized to optimize and interpret the proposed methods from a weighted viewpoint. The convergence of the proposal is proved, and detailed analyses of robustness are given. Experiments on both artificial and benchmark datasets confirm the validity of the proposed methods.
Similar content being viewed by others
References
Krizhevsky A, Sutskever I, Hinton GE (2012) ImageNet classification with deep convolutional neural networks. In: Proceedings of Advances in Neural Information Processing Systems, pp 1097–1105
Lecun Y, Boser B, Denker JS, Henderson D, Howard RE, Hubbard W, Jackel LD (1989) Backpropagation applied to handwritten zip code recognition. Neural Comput 1(4):541–551. https://doi.org/10.1162/neco.1989.1.4.541
Audibert JY, Catoni O (2011) Robust linear least squares regression. Ann Stat 39(5):2766–2794. https://doi.org/10.1214/11-AOS918
Cheung YM, Zeng H (2009) Local kernel regression score for selecting features of high-dimensional data. IEEE Trans Knowl Data Eng 21(12):1798–1802. https://doi.org/10.1109/TKDE.2009.23
Suykens JAK, Gestel TV, Brabanter JD, Moor BD, Vandewalle J (2002) Least squares support vector machines. Int J Circuit Theory Appl 27(6):605–615
Huang GB, Zhu QY, Siew CK (2006) Extreme learning machine: theory and applications. Neurocomputing 70(1):489–501. https://doi.org/10.1016/j.neucom.2005.12.126
Vapnik VN (2008) Statistical learning theory. Wiley, New York
Bartlett P, Mendelson S (2006) Empirical minimization. Probab Theory Relat Field 135(3):311–334. https://doi.org/10.1007/s00440-005-0462-3
Fama F, MacBeth D, Jackel LD (1973) Risk, return, and equilibrium: empirical tests. J Polit Econ 81(3):607–636. https://doi.org/10.1086/260061
Catoni O (2010) Challenging the empirical mean and empirical variance: a deviation study. Ann Inst Henri Poincare-Probab Stat 48(4):1148–1185. https://doi.org/10.1214/11-AIHP454
Kallummil S, Kalyani S (2019) Noise statistics oblivious GARD for robust regression with sparse outliers. IEEE Trans Signal Process 67(2):383–398. https://doi.org/10.1109/TSP.2018.2883025
Christmann A, Steinwart I (2007) Consistency and robustness of kernel based regression. Bernoulli 13(3):799–819. https://doi.org/10.3150/07-BEJ5102
Huang D, Cabral R, Torre FDL (2016) Robust regression. IEEE Trans Pattern Anal Mach Intell 38(2):363–375. https://doi.org/10.1109/TPAMI.2015.2448091
Zhang L, Zhou ZH (2018) ł\(_1\)-regression with heavy-tailed distributions. In: Proceedings of Advances in Neural Information Processing Systems
Yao Q, Tong H (2007) Asymmetric least squares regression estimation: a nonparametric approach. J Nonparametr Stat 6(4):273–292. https://doi.org/10.1080/10485259608832675
Chen C, Li Y, Yan C, Guo J, Liu G (2017) Least absolute deviation-based robust support vector regression. Knowledge Based Syst 131(1):183–194. https://doi.org/10.1016/j.knosys.2017.06.009
Chen C, Yan C, Li Y (2015) A robust weighted least squares support vector regression based on least trimmed squares. Neurocomputing 168(30):941–946. https://doi.org/10.1016/j.neucom.2015.05.031
Mangasarian OL, Musicant DR (2002) Robust linear and support vector regression. IEEE Trans Pattern Anal Mach Intell 22(9):950–955. https://doi.org/10.1109/34.877518
Huber PJ (2014) Robust statistics. Springer, New York
Huber PJ (1964) Robust estimation of a location parameter. Ann Math Stat 35(1):73–101. https://doi.org/10.1214/aoms/1177703732
Christmann A, Steinwart I (2007) How svms can estimate quantiles and the median. In: Proceedings of Advances in Neural Information Processing Systems, pp 305–312
Karal O (2017) Maximum likelihood optimal and robust support vector regression with lncosh loss function. Neural Netw 94(10):1–12. https://doi.org/10.1016/j.neunet.2017.06.008
Ren Z, Yang Y (2018) Correntropy-based robust extreme learning machine for classification. Neurocomputing 313(11):74–84. https://doi.org/10.1016/j.neucom.2018.05.100
Kai Z, Luo M (2015) Outlier-robust extreme learning machine for regression problems. Neurocomputing 151(3):1519–1527. https://doi.org/10.1016/j.neucom.2014.09.022
Yang L, Dong H (2018) Support vector machine with truncated pinball loss and its application in pattern recognition. Chemom Intell Lab Syst 177(6):89–99. https://doi.org/10.1016/j.chemolab.2018.04.003
Yang L, Dong H (2019) Robust support vector machine with generalized quantile loss for classification and regression. Appl Soft Comput 81(8):105483. https://doi.org/10.1016/j.asoc.2019.105483
Holland MJ, Ikeda K (2017) Robust regression using biased objectives. Mach Learn 106(4):1–37. https://doi.org/10.1007/s10994-017-5653-5
Lugosi G, Mendelson S (2020) Risk minimization by median-of-means tournaments. J Eur Math Soc 22(3):925–965. https://doi.org/10.4171/JEMS/937
Catoni O (2009) High confidence estimates of the mean of heavy-tailed real random variables. arXiv:0909.5366
Suykens JAK, Brabanter JD, Lukas L, Vandewalle J (2002) Weighted least squares support vector machines: robustness and sparse approximation. Neurocomputing 48(10):85–105. https://doi.org/10.1016/s0925-2312(01)00644-0
Wang K, Zhong P (2014) Robust non-convex least squares loss function for regression with outliers. Knowl Based Syst 71:290–302. https://doi.org/10.1016/j.knosys.2014.08.003
Zhao Y, Sun J (2010) Robust truncated support vector regression. Expert Syst Appl 37(7):5126–5133. https://doi.org/10.1016/j.eswa.2009.12.082
Chen K, Lv Q, Lu Y, Dou Y (2016) Robust regularized extreme learning machine for regression using iteratively reweighted least squares. Neurocomputing 230(12):345–358. https://doi.org/10.1016/j.neucom.2016.12.029
Dinh DP, Thi HAL, Akoa F (2008) Combining DCA (DC Algorithms) and interior point techniques for large-scale nonconvex quadratic programming. Optim Methods Softw 23(4):609–629. https://doi.org/10.1080/10556780802263990
Yang L, Qian Y (2016) A sparse logistic regression framework by difference of convex functions programming. Appl Intell 45(2):241–254. https://doi.org/10.1007/s10489-016-0758-2
Yuille AL (2002) CCCP algorithms to minimize the Bethe and Kikuchi free energies: convergent alternatives to belief propagation. Neural Comput 14(7):1691–1722. https://doi.org/10.1162/08997660260028674
Zhang Y, Sun Y, He R, Tan T (2013) Robust subspace clustering via half-quadratic minimization. In: Proceedings of IEEE International Conference on Computer Vision, pp 3096–3103
He R, Zheng W, Tan T, Sun Z (2014) Half-quadratic-based iterative minimization for robust sparse representation. IEEE Trans Pattern Anal Mach Intell 36(2):261–275. https://doi.org/10.1109/TPAMI.2013.102
Feng Y, Yang Y, Huang X, Mehrkanoon S, Suykens JAK (2016) Robust support vector machines for classification with nonconvex and smooth losses. Neural Comput 28(6):1217–1247. https://doi.org/10.1162/NECO_a_00837
Li C, Zhou S (2017) Sparse algorithm for robust LSSVM in primal space. Neurocomputing 275:2880–2891. https://doi.org/10.1016/j.neucom.2017.10.011
Xu G, Hu B, Principe JC (2016) Robust C-loss kernel classifiers. IEEE Trans Neural Netw Learn Syst 29(3):510–522. https://doi.org/10.1109/TNNLS.2016.2637351
Lai MJ, Xu Y, Yin W (2013) Improved iteratively reweighted least squares for unconstrained smoothed łq minimization. SIAM J Numer Anal 51(2):927–957. https://doi.org/10.1137/110840364
Green PJ (1984) Iteratively reweighted least squares for maximum likelihood estimation, and some robust and resistant alternatives. J R Stat Soc Ser B-Stat Methodol 46(2):149–192. https://doi.org/10.1111/j.2517-6161.1984.tb01288.x
Debruyne M, Christmann A, Hubert M, Suykens JAK (2010) Robustness of reweighted least squares kernel based regression. J Multivar Anal 101:447–463. https://doi.org/10.1016/j.jmva.2009.09.007
Yi S, He Z, Cheung YM, Chen WS (2018) Unified sparse subspace learning via self-contained regression. IEEE Trans Circuits Syst Video Technol 28(10):2537–2550. https://doi.org/10.1109/TCSVT.2017.2721541
Deng W, Zheng Q, Chen L (2009) Regularized extreme learning machine. In: Proceedings of IEEE Symposium on Computational Intelligence and Data Mining, pp 389–395 https://doi.org/10.1109/CIDM.2009.4938676
Christmann A, Steinwart I (2008) Support vector machines. Springer, New York
Hampel FR, Ronchetti EM, Rousseeuw PJ, Stahel WA (1986) Robust statistics: the approach based on influence functions. Wiley, New York
Huang X, Shi L, Suykens JAK (2014) Support vector machine classifier with pinball loss. IEEE Trans Pattern Anal Mach Intell 36(5):984–997. https://doi.org/10.1109/TPAMI.2013.1786
Dua D, Graff C (2019) UCI machine learning repository. (http://archive.ics.uci.edu/ml)
Deng N, Tian Y, Zhang C (2012) Support vector machines: optimization based theory, algorithms and extensions. CRC Press, Boca Raton
Crambes C, Gannoun A, Henchiri Y (2013) Support vector machine quantile regression approach for functional data: simulation and application studies. J Multivar Anal 121(11):50–68. https://doi.org/10.1016/j.jmva.2013.06.004
Acknowledgements
This work is supported by National Nature Science Foundation of China (Nos. 11471010, 11271367).
Author information
Authors and Affiliations
Corresponding author
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
About this article
Cite this article
Dong, H., Yang, L. Kernel-based regression via a novel robust loss function and iteratively reweighted least squares. Knowl Inf Syst 63, 1149–1172 (2021). https://doi.org/10.1007/s10115-021-01554-8
Received:
Revised:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10115-021-01554-8