Skip to main content
SpringerLink
Log in

Robust Support Vector Regression with Generalized Loss Function and Applications

  • Published:
Neural Processing Letters Aims and scope Submit manuscript

Abstract

The classical support vector machine (SVM) is sensitive to outliers. This paper proposes a robust support vector regression based on a generalized non-convex loss function with flexible slope and margin. The robust model is more flexible for regression estimation. Meanwhile, it has strong ability of suppressing the impact of outliers. The generalized loss function is neither convex nor differentiable. We approximate it by combining two differentiable Huber functions, and the resultant optimization problem is a difference of convex functions (d.c.) program. We develop a Newton algorithm to solve the robust model. The numerical experiments on benchmark datasets, financial time series datasets and document retrieval dataset confirm the robustness and effectiveness of the proposed method. It also reduces the downside risk in financial time series prediction, and significantly outperforms ranking SVM for performing real information retrieval tasks.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6

Similar content being viewed by others

Notes

  1. http://www.ics.uci.edu/~mlearn/MLRepository.html

  2. Available from http://lib.stat.cmu.edu/datasets/

  3. http://www.liaad.up.pt/~ltorgo/Regression/DataSets.html

References

  1. An LTH, Tao P (2005) The DC (difference of convex functions) programming and DCA revisited with DC models of real world nonconvex optimization problems. Ann Oper Res 133:23–46

    Article  MATH  MathSciNet  Google Scholar 

  2. Bo L, Wang L, Jiao L (2007) Recursive finite Newton algorithm for support vector regression in the primal. Neural Comput 19(4):1082–1096

    Article  MATH  MathSciNet  Google Scholar 

  3. Cao Y, Xu J, Liu T, Li H, Huang Y, Hon H (2006). Adapting ranking SVM to document retrieval. In: Proceedings of the 29th annual international ACM SIGIR conference on research and development in information retrieval, pp. 186–193

  4. Cao Z, Qin T, Liu T, Tsai M, Li H (2007) Learning to rank: from pairwise approach to listwise approach. In: Proceedings of the 24th international conference on machine learning (ICML), pp. 129–136

  5. Chang C, Lin C (2001) LIBSVM: a library for support vector machines. http://www.csie.ntu.edu.tw/~cjlin

  6. Chapelle O (2007) Training a support vector machine in the primal. Neural Comput 19(5):1155–1178

    Article  MATH  MathSciNet  Google Scholar 

  7. Collobert R, Sinz F, Weston J, Bottou L (2006) Trading convexity for scalability. In: Proceeding of the 23rd international conference on machine learning (ICML)

  8. Geng B, Yang Y, Xu C, Hua X (2010) Content-aware ranking for visual search. In: IEEE conference on computer vision and pattern recognition (CVPR), pp. 3400–3407

  9. Huang G, Song S, Wu C, You K (2012) Robust support vector regression for uncertain input and output data. IEEE Trans Neural Netw Learn Syst 23:1690–1700

    Article  Google Scholar 

  10. Huang H, Liu Y (2002) Fuzzy support vector machines for pattern recognition and data mining. Int J Fuzzy Syst 4:3–12

    Google Scholar 

  11. Joachims T, Finley T, Yu C (2009) Cutting-plane training of structural SVMs. Mach Learn 77(1):27–59

    Article  MATH  Google Scholar 

  12. Luo Y, Tao D, Geng B, Xu C, Maybank SJ (2013) Manifold regularized multitask learning for semi-supervised multilabel image classification. IEEE Trans Image Process 22(2):523–536

    Article  MathSciNet  Google Scholar 

  13. Luo Y, Tao D, Xu C, Xu C, Liu H, Wen Y (2013) Multiview vector-valued manifold regularization for multilabel image classification. IEEE Trans Neural Netw Learn Syst 24(5):709–722

    Article  Google Scholar 

  14. Ong CS, An LTH (2013) Learning sparse classifiers with difference of convex functions algorithms. Optim Methods Softw 28(4):830–854

    Article  MATH  MathSciNet  Google Scholar 

  15. Osuna E, Freund R, Girosi F (1997) An improved training algorithm for support vector machines. In: Principe J, Gile L, Morgan N, Wilson E (eds) Neural networks for signal processing VII-proceedings of the 1997 IEEE Workshop, pp. 276–285

  16. Peng X (2009) TSVR: an efficient twin support vector machine for regression. Neural Netw 23(3):365–372

    Article  Google Scholar 

  17. Platt JC (1999) Fast training of support vector machines using sequential minimal optimization. In: Schölkopf B, Burges CJC, Smola AJ (eds) Advances in Kernel methods-support vector learning. MIT Press, Cambridge, pp 185–208

    Google Scholar 

  18. Sriperumbudur BK, Lanckriet GRG (2009) On the convergence of the concave–convex procedure. Advances in neural information processing systems, pp. 1759–1767

  19. Suykens J, DeBrabanter J, Lukas L (2002) Weighted least squares support vector machines: robustness and sparse approximation. Neurocomputing 48:85–105

    Article  MATH  Google Scholar 

  20. Tao P, An LTH (1998) D.c. optimization algorithms for solving the trust region subproblem. SIAM J Optim 8:476–505

    Article  MATH  MathSciNet  Google Scholar 

  21. Trafalis B, Gilbert C (2006) Robust classification and regression using support vector machines. Eur J Operat Res 173:893–909

    Article  MATH  MathSciNet  Google Scholar 

  22. Tsochantaridis I, Joachims T, Hofmann T, Altun Y (2005) Large margin methods for structured and interdependent output variables. J Mach Learn Res 6:1453–1484

    MATH  MathSciNet  Google Scholar 

  23. Vapnik VN (1995) The nature of statistical learning theory. Springer, New York

    Book  MATH  Google Scholar 

  24. Wang K, Zhong P, Zhao Y (2010) Training robust support vector regression via d.c. program. J Inf Comput Sci 7(12):2385–2394

    Google Scholar 

  25. Wang L, Jia H, Li J (2008) Training robust support vector machine with smooth ramp loss in the primal space. Neurocomputing 71(13–15):3020–3025

    Article  Google Scholar 

  26. Wu Y, Liu Y (2007) Robust truncated hinge loss support vector machines. J Am Stat Assoc 102(479):974–983

    Article  MATH  Google Scholar 

  27. Xu L, Crammer K, Schuurmans D (2006) Robust support vector machine training via convex outlier ablation. In: Proceedings of the 21st national conference on artifical Intelligence (AAAI)

  28. Xu Y, Wang L (2012) A weighted twin support vector regression. Knowl-Based Syst 33:92–101

    Article  Google Scholar 

  29. Yang C, Chou J, Lian F (2013) Robust classifier learning with fuzzy class labels for large-margin support vector machines. Neurocomputing 99:1–14

    Article  Google Scholar 

  30. Yang H, Chan L, King I (2002) Support vector machine regression for volatile stock market prediction. In: Proceedings of the third international conference on intelligent data engineering and automated learning, pp. 391–396

  31. Yu J, Wang M, Tao D (2012) Semisupervised multiview distance metric learning for cartoon synthesis. IEEE Trans Image Process 21(11):4636–4648

    Article  MathSciNet  Google Scholar 

  32. Yu J, Tao D, Wang M (2012) Adaptive hypergraph learning and its application in image classification. IEEE Trans Image Process 21(7):3262–3272

    Article  MathSciNet  Google Scholar 

  33. Yu J, Tao D (2013) Modern machine learning techniques and their applications in cartoon animation research. Wiley, Hoboken

    Book  Google Scholar 

  34. Yu J, Liu D, Tao D, Seah H (2011) Complex object correspondence construction in two-dimensional animation. IEEE Trans Image Process 20(11):3257–3269

    Article  MathSciNet  Google Scholar 

  35. Yuille AL, Rangarajan A (2003) The concave–convex procedure. Neural Comput 15(4):915–936

    Article  MATH  Google Scholar 

  36. Zhao Y, Sun J (2008) Robust support vector regression in the primal. Neural Netw 21(10):1548–1555

    Article  MATH  MathSciNet  Google Scholar 

  37. Zhong P (2012) Training robust support vector regression with smooth non-convex loss function. Optim Methods Softw 27(6):1039–1058

    Article  MATH  MathSciNet  Google Scholar 

  38. Zhong P, Fukushima M (2007) Second order cone programming formulations for robust multi-class classification. Neural Comput 19:258–282

    Article  MATH  MathSciNet  Google Scholar 

  39. Zhong P, Wang L (2008) Support vector regression with input data uncertainty. Int J Innov Comput Inf Control 4:2325–2332

    Google Scholar 

  40. Liu W, Tao D (2013) Multiview Hessian regularization for image annotation. IEEE Trans Image Process 22(7):2676–2687

    Article  MathSciNet  Google Scholar 

  41. Herbrich R, Graepel T, Obermayer K (1999) Large margin rank boundaries for ordinal regression. Advances in neural information processing systems. MIT Press, Cambridge, pp 115–132

    Google Scholar 

  42. Järvelin K, Kekäläinen J (2000) IR evaluation methods for retrieving highly relevant documents. In: Proceedings of the 23rd annual international ACM SIGIR conference on research and development in information retrieval, pp. 41–48

  43. Qin T, Liu T, Xu J, Li H (2010) Letor: a benchmark collection for research on learning to rank for information retrieval. Inf Retr 13(4):346–374

    Article  Google Scholar 

  44. Hersh W, Buckley C, Leone T, Hickam D (1994) OHSUMED: an interactive retrieval evaluation and new large test collection for research. In: Proceedings of the 17th annual ACM SIGIR conference, pp. 192–201.

Download references

Acknowledgments

The work is supported by National Natural Science Foundation of China Grant No.11171346 and Chinese Universities Scientific Fund No.2013YJ010. The authors also gratefully acknowledge the helpful comments and suggestions of the reviewers, which have improved the presentation.

Author information

Authors and Affiliations

  1. College of Science, China Agricultural University, Beijing, 100083, China

    Kuaini Wang, Wenxin Zhu & Ping Zhong

  2. Basic Science Department, TianJin Agricultural University, Tianjin, 300384, China

    Wenxin Zhu

Authors
  1. Kuaini Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Wenxin Zhu
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Ping Zhong
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Ping Zhong.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Wang, K., Zhu, W. & Zhong, P. Robust Support Vector Regression with Generalized Loss Function and Applications. Neural Process Lett 41, 89–106 (2015). https://doi.org/10.1007/s11063-013-9336-3

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11063-013-9336-3

Keywords

Search

Navigation