Skip to main content
SpringerLink
Log in

Regularization feature selection projection twin support vector machine via exterior penalty

  • Original Article
  • Published:
Neural Computing and Applications Aims and scope Submit manuscript

Abstract

In the past years, non-parallel plane classifiers that seek projection direction instead of hyperplane for each class have attracted much attention, such as the multi-weight vector projection support vector machine (MVSVM) and the projection twin support vector machine (PTSVM). Instead of solving two generalized eigenvalue problems in MVSVM, PTSVM solves two related SVM-type problems to obtain the two projection directions by solving two smaller quadratic programming problems, similar to twin support vector machine. In order to suppress input space features, we propose a novel non-parallel classifier to automatically select significant features, called regularization feature selection projection twin support vector machine (RFSPTSVM). In contrast to the PTSVM, we first incorporate a regularization term to ensure the optimization problems are convex, and then replace all the terms with L1-norm ones. By minimizing an exterior penalty function of the linear programming problem and using a fast generalized Newton algorithm, our RFSPTSVM obtains very sparse solutions. For nonlinear case, the method utilizes minimal number of kernel functions. The experimental results on toy datasets, Myeloma dataset, several UCI benchmark datasets, and NDCC generated datasets show the feasibility and effectiveness of the proposed method.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Fig. 1

Similar content being viewed by others

References

  1. Burges C (1998) A tutorial on support vector machines for pattern recognition. Data Min Knowl Discov 2:121–167

    Article  Google Scholar 

  2. Cristianini N, Shawe-Taylor J (2000) An introduction to support vector machines: and other kernel-based learning methods. Cambridge University Press, Cambridge

    Book  MATH  Google Scholar 

  3. Cortes C, Vapnik VN (1995) Support vector networks. Mach Learn 20:273–297

    MATH  Google Scholar 

  4. Demiriz A, Bennett KP, Breneman CM, Embrechts MJ (2001) Support vector machine regression in chemometrics. In: Computing science and statistics, proceedings of the 33rd symposium on the interface. American Statistical Association for the Interface Foundation of North America, Washington, DC

  5. Osuna E, Freund R, Girosi F (1997) Training support vector machines: an application to face detection. In: Proceedings of the 1997 IEEE Computer Society conference on computer vision pattern recognition, pp 130–136

  6. Jia G, Martinez AM (2009) Support vector machines in face recognition with occlusions. In: Proceedings of the 2009 IEEE conference on comput vision and pattern recognition, Miami, Florida, pp 136–141

  7. Hotta Kazuhiro (2008) Robust face recognition under partial occlusion based on support vector machine with local Gaussian summation kernel. Image Vis Comput 26(11):1490–1498

    Article  Google Scholar 

  8. Wang Zhenyu, Yang Wankou, Ben Xianye (2015) Low-resolution degradation face recognition over long distance based on CCA. Neural Comput Appl 26(7):1645–1652

    Article  Google Scholar 

  9. Yang Wankou, Wang Zhenyu, Sun Changyin (2015) A collaborative representation based projections method for feature extraction. Pattern Recogn 48(1):20–27

    Article  Google Scholar 

  10. Joachims T (1998) Text categorization with support vector machines: learning with many relevant features. Machine Learning ECML-98:137–142

    Google Scholar 

  11. Cao L (2003) Support vector machines experts for time series forecasting. Neurocomputing 51:321–339

    Article  Google Scholar 

  12. Chen X, Yang J, Liang J, Ye Q (2012) Recursive robust least squares support vector regression based on maximum correntropy criterion. Neurocomputing 97:63–73

    Article  Google Scholar 

  13. Zhao Y, Zhao J, Zhao M (2013) Twin least squares support vector regression. Neurocomputing 118:225–236

    Article  Google Scholar 

  14. Ben Xianye, Zhang Peng et al (2015) Gait recognition and micro-expression recognition based on maximum margin projection with tensor representation. Neural Comput Appl. doi:10.1007/s00521-015-2031-8

    Google Scholar 

  15. Ben Xianye, Meng Weixiao et al (2013) Kernel coupled distance metric learning for gait recognition and face recognition. Neurocomputing 120:577–589

    Article  Google Scholar 

  16. Du B, Zhang L (2015) Target detection based on a dynamic subspace. Pattern Recogn 47(1):344–358

    Article  Google Scholar 

  17. Du B, Zhang L (2014) A discriminative metric learning based anomaly detection method. IEEE Trans Geosci Remote Sens 52(11):6844–6857

    Article  Google Scholar 

  18. Fung G, Mangasarian OL (2001) Proximal support vector machine classifiers. In: Provost F, Srikant R (eds) Proceedings of the knowledge discovery and data mining, pp 77–86

  19. Mangasarian O, Wild E (2006) Multisurface proximal support vector classification via generalize eigenvalues. IEEE Trans Pattern Anal Mach Intell 28(1):69–74

    Article  Google Scholar 

  20. Jayadeva R, Khemchandani S (2007) Chandra, twin support vector machines for pattern classification. IEEE Trans Pattern Anal Mach Intell 29:905–910

    Article  MATH  Google Scholar 

  21. Arun Kumar M, Gopal M (2009) Least squares twin support vector machines for pattern classification. Expert Syst Appl 36:7535–7543

    Article  Google Scholar 

  22. Ye Q, Zhao C, Ye N, Chen Y (2010) Multi-weight vector projection support vector machines. Pattern Recogn Lett 31(13):2006–2011

    Article  Google Scholar 

  23. Chen X, Yang J, Ye Q, Liang J (2011) Recursive projection twin support vector machine via within-class variance minimization. Pattern Recogn. doi:10.1016/j.patcog.2011.03.001

    MATH  Google Scholar 

  24. Shao Yuan-Hai, Wang Zhen, Chen Wei-Jie, Deng Nai-Yang (2013) A regularization for the projection twin support vector machine. Knowl Based Syst 37:203–210

    Article  Google Scholar 

  25. Zhu J, Rosset S, Hastie T, Tibshirani R (2004) 1-norm support vector machines. In: Thrun S, Saul LK, Scholkopf BH (eds) Advances in neural information processing systems16–NIPS2003. MIT Press, Cambridge

    Google Scholar 

  26. Golub T, Slonim D, Tamayo P, Huard C, Gaasenbeek M, Mesirov J, Coller H, Loh M, Downing J, Caligiuri M (1999) Molecular classification of cancer: class discovery and class prediction by gene expression monitoring. Science 286:531–536

    Article  Google Scholar 

  27. Guyon I, Weston J, Barhill S, Vapnik V (2002) Gene selection for cancer classification using support vector machines. Mach Learn 46:389–422

    Article  MATH  Google Scholar 

  28. Zhou WD, Zhang L, Jiao LC (2002) Linear programming support vector machines. Pattern Recogn 35(12):2927–2936

    Article  MATH  Google Scholar 

  29. Zou H (2007) An improved 1-norm SVM for simultaneous classification and variable selection. In: Proceedings of the eleventh international conference on artificial intelligence and statistics

  30. Fung G, Mangasarian OL (2004) A feature selection Newton method for support vector machine classification. Comput Optim Appl 28(2):185–202

    Article  MathSciNet  MATH  Google Scholar 

  31. Mangasarian OL (2006) Exact 1-norm support vector machines via unconstrained convex differentiable minimization. J Mach Learn Res 7:1517–1530

    MathSciNet  MATH  Google Scholar 

  32. Gao Shangbing, Ye Q, Ye N (2011) 1-norm least squares twin support vector machines. Neurocomputing 74:3590–3597

    Article  Google Scholar 

  33. Bai L, Wang Z, Shao YH et al (2014) A novel feature selection method for twin support vector machine. Knowl Based Syst 59:1–8

    Article  Google Scholar 

  34. Ye Q, Zhao C, Ye N, Zheng H, Chen X (2012) A feature selection method for nonparallel plane support vector machine classification. Optim Methods Softw 27(3):431–443

    Article  MathSciNet  MATH  Google Scholar 

  35. Guo J et al (2014) Feature selection for least squares projection twin support vector machine. NeuroComputing 144:174–183

    Article  Google Scholar 

  36. Tao Y, Yang J (2010) Quotient vs. difference: comparison between the two discriminant criteria. Neurocomputing 73:1808–1817

    Article  Google Scholar 

  37. Mangasarian OL (1994) Nonlinear programming. SIAM, Philadelphia

    Book  MATH  Google Scholar 

  38. Mangasarian OL, Meyer RR (1979) Nonlinear perturbation of linear programs. SIAM J Control Optim 17(6):745–752

    Article  MathSciNet  MATH  Google Scholar 

  39. Blake C, Merz C (1998) UCI repository of machine learning databases, Department of Information and Computer Sciences, University of California, Irvine. http://www.ics.uci.edu/~mlearn/MLRepository.html

  40. Page D, Zhan F, Cussens J, Waddell M, Hardin J, Barlogie B, Shaughnessy J Jr (2002) Comparative data mining for microarrays: a case study based on Multiple Myeloma. Technical Report 1453, Computer Sciences Department, University of Wisconsin

  41. Thompson ME (2006) NDCC: normally distributed clustered datasets on cubues. www.cs.wisc.edu/dmi/svm/ndcc

  42. LeCun Y, Cortes C (2010) MNIST handwritten digit database. http://yann.lecun.com/exdb/mnist/

Download references

Acknowledgments

The authors would like to thank the anonymous reviewers for their constructive comments and suggestions. This work was partially supported by the Natural Science Foundation of Jiangsu Province of China (Grant No. BK20140794), the China Postdoctoral Science Foundation (Grant No. 2014M551599), and the Fundamental Research Funds for the Central Universities (Grant No. 30916011326).

Author information

Authors and Affiliations

  1. School of Instrument Science and Engineering, Southeast University, Nanjing, China

    Ping Yi & Aiguo Song

  2. School of Computer Science and Engineering, Nanjing University of Science and Technology, Nanjing, China

    Jianhui Guo

  3. School of Engineering and Advanced Technology, Massey University, Auckland, New Zealand

    Ruili Wang

Authors
  1. Ping Yi
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Aiguo Song
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Jianhui Guo
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Ruili Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Jianhui Guo.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Yi, P., Song, A., Guo, J. et al. Regularization feature selection projection twin support vector machine via exterior penalty. Neural Comput & Applic 28 (Suppl 1), 683–697 (2017). https://doi.org/10.1007/s00521-016-2375-8

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00521-016-2375-8

Keywords

Search

Navigation