Skip to main content
SpringerLink
Log in

A highly efficient ADMM-based algorithm for outlier-robust regression with Huber loss

  • Published:
Applied Intelligence Aims and scope Submit manuscript

Abstract

Huber robust regression (HRR) has attracted much attention in machine learning due to its greater robustness to outliers compared to least-squares regression. However, existing algorithms for HRR are computationally much less efficient than those for least-squares regression. Based on a maximally split alternating direction method of multipliers (MS-ADMM) for model fitting, a highly computationally efficient algorithm referred to as the modified MS-ADMM is derived in this article for HRR. After analyzing the convergence of the modified MS-ADMM, a parameter selection scheme is presented for the algorithm. With the parameter values calculated via this scheme, the modified MS-ADMM converges very rapidly, much faster than several typical HRR algorithms. Through applications in the training of stochastic neural networks and comparisons with existing algorithms, the modified MS-ADMM is shown to be computationally much more efficient than the convex quadratic programming method, the Newton method, the iterative reweighted least-squares method, and Nesterov’s accelerated gradient method. Implementation of the proposed algorithm on a GPU-based parallel computing platform demonstrates its higher GPU acceleration ratio compared to the competing methods and, thus, its greater superiority in computational efficiency over the competing methods.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13

Similar content being viewed by others

Data Availability

All the data used in this article are sourced from open and publicly accessible platforms. No proprietary, confidential, or private data has been used.

References

  1. Huber PJ (1981) Robust Statistics. John Wiley & Sons, New York

    Book  Google Scholar 

  2. Li W, Swetits JJ (1998) The linear 1 estimator and the Huber M-estimator. SIAM J Optim 8(2):457–475

    Article  MathSciNet  Google Scholar 

  3. Chen B, Pinar MC (1998) On Newton’s method for Huber’s robust M-estimation problems in linear regression. BIT Numer Math 38(4):674–684

    Article  MathSciNet  Google Scholar 

  4. Mangasarian OL, Musicant DR (2000) Robust linear and support vector regression. IEEE Trans Pattern Anal Mach Intell 22(9):950–955

    Article  Google Scholar 

  5. Zhu J, Hoi SCH, Lyu MR-T (2008) Robust regularized kernel regression. IEEE Trans Syst Man Cybern Part B: Cybern 38(6):1639–1644

    Article  Google Scholar 

  6. Zhang K, Luo M (2015) Outlier-robust extreme learning machine for regression problems. Neurocomputing 151(3):1519–1527

    Article  Google Scholar 

  7. Huang D, Cabral R, De la Torre F (2016) Robust regression. IEEE Trans Pattern Anal Mach Intell 38(2):363–375

    Article  Google Scholar 

  8. Barreto GA, Barros ALBP (2016) A robust extreme learning machine for pattern classification with outliers. Neurocomputing 176:3–13

    Article  Google Scholar 

  9. Chen K, Lv Q, Lu Y, Dou Y (2017) Robust regularized extreme learning machine for regression using iteratively reweighted least squares. Neurocomputing 230:345–358

    Article  Google Scholar 

  10. Chen B, Wang X, Lu N, Wang S, Cao J (2018) Mixture correntropy for robust learning. Pattern Recogn 79:318–327

    Article  Google Scholar 

  11. Jin J-W, Chen CLP (2018) Regularized robust broad learning system for uncertain data modeling. Neurocomputing 322:58–69

    Article  Google Scholar 

  12. Zoubir AM, Koivunen V, Ollila E et al. (2018) Robust Statistics for Signal Processing, Cambridge University Press

  13. Barron JT (2019) A general and adaptive robust loss function, in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 4331–4339

  14. Yang L, Dong H (2019) Robust support vector machine with generalized quantile loss for classification and regression. Appl Soft Comput 81:105483

    Article  Google Scholar 

  15. Xie S, Yang C, Yuan X, Wang X, Xie Y (2019) A novel robust data reconciliation method for industrial processes. Control Eng Pract 83:203–212

    Article  Google Scholar 

  16. Balasundaram S, Prasad SC (2020) Robust twin support vector regression based on Huber loss function. Neural Comput Appl 32:11285–11309

    Article  Google Scholar 

  17. Chu F, Liang T, Chen CLP, Wang X, Ma X (2020) Weighted broad learning system and its application in nonlinear industrial process modeling. IEEE Trans Neural Netw Learn Syst 31(8):3017–3031

    Article  MathSciNet  Google Scholar 

  18. da Silva BLS, Inaba FK, Salles EOT, Ciarelli PM (2020) Outlier robust extreme machine learning for multi-target regression, Expert Systems With Applications, vol. 140, Article 112877, 1–13

  19. Khan DM, Ali M, Ahmad Z, Manzoor S, Hussain S (2021) A new efficient redescending M-estimator for robust fitting of linear regression models in the presence of outliers, Mathematical Problems in Engineering, vol. 2021, Article 3090537, 1–11

  20. Dong H, Yang L (2021) Kernel-based regression via a novel robust loss function and iteratively reweighted least squares. Knowl Inform Syst 63(5):1149–1172

    Article  Google Scholar 

  21. Sabzekar M, Hasheminejad SMH (2021) Robust regression using support vector regressions, Chaos, Solitons & Fractals, vol. 144, Article 110738

  22. Zheng Y, Wang S, Chen B (2023) Quantized minimum error entropy with fiducial points for robust regression. Neural Netw 168:405–418

    Article  Google Scholar 

  23. Liu L, Liu T, Chen CLP, Wang Y (2023) Modal-regression-based broad learning system for robust regression and classification. IEEE Trans Neural Netw Learn Syst (Early Access). (https://doi.org/10.1109/TNNLS.2023.3256999)

  24. Zheng Y, Wang S, Chen B (2023) Robust one-class classification with support vector data description and mixed exponential loss function. Eng Appl Artif Intell 122:106153

    Article  Google Scholar 

  25. Boyd S, Parikh N, Chu E, Peleato B, Eckstein J (2011) Distributed optimization and statistical learning via the alternating direction method of multipliers, Foundations and Trends® in Machine Learning, 3:(1) 1–122

  26. Luo M, Zhang L, Liu J, Guo J, Zheng Q (2017) Distributed extreme learning machine with alternating direction method of multiplier. Neurocomputing 261:164–170

    Article  Google Scholar 

  27. Wang H, Gao Y, Shi Y, Wang R (2017) Group-based alternating direction method of multipliers for distributed linear classification. IEEE Trans Cybern 47(11):3568–3582

    Article  Google Scholar 

  28. Wang H, Feng R, Han Z-F, Leung C-S (2018) ADMM-based algorithm for training fault tolerant RBF networks and selecting centers. IEEE Trans Neural Netw Learn Syst 29(8):3870–3878

    Article  MathSciNet  Google Scholar 

  29. Inaba FK, Salles EOT, Perron S, Caporossi G (2018) DGR-ELMDistributed generalized regularized ELM for classification. Neurocomputing 275:1522–1530

    Article  Google Scholar 

  30. Kim B, Yu D, Won JH (2018) Comparative study of computational algorithms for the Lasso with high-dimensional, highly correlated data. Appl Intell 48(8):1933–1952

    Article  Google Scholar 

  31. Lai X, Cao J, Huang X, Wang T, Lin Z (2020) A maximally split and relaxed ADMM for regularized extreme learning machines. IEEE Trans Neural Netw Learn Syst 31(6):1899–1913

    Article  MathSciNet  Google Scholar 

  32. Wang Y, Guan Y, Wang B, Li X (2022) Learning with partial multi-labeled data by leveraging low-rank constraint and decomposition. Applied Intelligence, Online, pp 1–13

    Google Scholar 

  33. Hu J, Guo T, Zhao T (2022) A faster stochastic alternating direction method for large scale convex composite problems. Applied Intelligence, Online, pp 1–13

    Google Scholar 

  34. Lai X, Cao J, Lin Z (2023) An accelerated maximally split ADMM for a class of generalized ridge regression. IEEE Trans Neural Netw Learn Syst 34(2):958–972

    Article  MathSciNet  Google Scholar 

  35. Bernier JL, Ortega J, Rodriguez MM, Rojas I, Prieto A (1999) An accurate measure for multilayer perceptron tolerance to weight deviations. Neural Process Lett 10(2):121–130

    Article  Google Scholar 

  36. Sum JP-F, Leung C-S, Ho KI-J (2009) On objective function, regularizer, and prediction error of a learning algorithm for dealing with multiplicative weight noise. IEEE Trans Neural Netw 20(1):124–138

    Article  Google Scholar 

  37. Han Z-F, Feng R-B, Wan WY, Leung C-S (2015) Online training and its convergence for faulty networks with multiplicative weight noise. Neurocomputing 155:53–61

    Article  Google Scholar 

  38. Feng R-B, Han Z-F, Wan WY, Leung C-S (2017) Properties and learning algorithms for faulty RBF networks with coexistence of weight and node failures. Neurocomputing 224:166–176

    Article  Google Scholar 

  39. Boyd S, Vandenberghe L (2009) Convex Optimization. Cambridge University Press, Cambridge

    Google Scholar 

  40. Y. Nesterov, Introductory Lectures on Convex Optimization: A Basic Course, ser. Applied Optimization, vol. 87, Norwell, MA: Kluwer, 2004.

  41. Cevher V, Becker S, Schmidt M (May2014) Convex optimization for big data: scalable, randomized, and parallel algorithms for big data analytics. IEEE Signal Process Mag 31(5):32–43

    Article  Google Scholar 

  42. Huang G-B, Zhou H, Ding X, Zhang R (2012) Extreme learning machine for regression and multiclass classification. IEEE Transactions on Systems, Man, and Cybernetics - Part B: Cybernetics 42(2):513–529

    Article  Google Scholar 

  43. Chen CLP, Liu Z, Feng S (2018) Broad Learning System: An effective and efficient incremental learning system without the need for deep architecture. IEEE Trans Neural Netw Learn Syst 29(1):10–24

    Article  MathSciNet  Google Scholar 

Download references

Acknowledgements

This work was supported by the Natural Science Foundation of Zhejiang Province (LZ22F030002, LZ24F030010), the National Natural Science Foundation of China (U1909209), the National Key Research Program of China (2021YFE0100100), and the Research Funding of Education of Zhejiang Province (Y202249784).

Author information

Authors and Affiliations

  1. Artificial Intelligence Institute, Hangzhou Dianzi University, Hangzhou, 310018, China

    Tianlei Wang, Xiaoping Lai & Jiuwen Cao

  2. Machine Learning and I-Health International Cooperation Base of Zhejiang Province, Hangzhou Dianzi University, Hangzhou, 310018, China

    Tianlei Wang, Xiaoping Lai & Jiuwen Cao

Authors
  1. Tianlei Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Xiaoping Lai
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Jiuwen Cao
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

Tianlei Wang: Conceptualization, Investigation, Writing—review and editing. Xiaoping Lai: Conceptualization, Methodology, Software, Investigation, Writing—original draft preparation. Jiuwen Cao: Conceptualization, Investigation, Writing—review and editing, Supervision.

Corresponding author

Correspondence to Jiuwen Cao.

Ethics declarations

Competing Interests

All authors declare that they have no conflicts of interest.

Ethical Compliance

This study does not involve any human participant or animal.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Wang, T., Lai, X. & Cao, J. A highly efficient ADMM-based algorithm for outlier-robust regression with Huber loss. Appl Intell 54, 5147–5166 (2024). https://doi.org/10.1007/s10489-024-05370-9

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10489-024-05370-9

Keywords

Search

Navigation