Abstract
We consider the supervised pattern classification in the high-dimensional setting, in which the number of features is much larger than the number of observations. We present a novel approach to the sparse linear discriminant analysis (LDA) using the zero-norm. The resulting optimization problem is non-convex, discontinuous and very hard to solve. We overcome the discontinuity by using an appropriate continuous approximation to zero-norm such that the resulting problem can be formulated as a DC (Difference of Convex functions) program to which DC programming and DC Algorithms (DCA) can be investigated. The computational results show the efficiency and the superiority of our approach versus the l 1 regularization model on both feature selection and classification.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
Bickel, P., Levina, E.: Some theory for Fisher’s linear discriminant function, naive Bayes, and some alternatives when there are many more variables than observations. Bernoulli 6, 989–1010 (2004)
Bradley, P.S., Magasarian, O.L., Street, W.N.: Feature Selection via mathematical Programming. INFORMS Journal on Computing, 209–217 (1998)
Clemmensen, L., Hastie, T., Witten, D., Ersboll, B.: Sparse discriminant analysis. Technometrics 53(4), 406–413 (2011)
Duan, K.B., Rajapakse, J.C., Wang, H., Azuaje, F.: Multiple SVM-RFE for Genne Selection in Cancer Classification With Expression Data. IEEE Transactions on Nanobioscience 4, 228–234 (2005)
Dudoit, S., Fridlyand, J., Speed, T.: Comparison of discrimination methods for the classification of tumors using gene expression data. J. Amer. Statist. Assoc. 96, 1151–1160 (2001)
Fisher, R.A.: The use of multiple measurements in taxonomic problems. Annals of Eugenics 7, 179–188 (1936)
Friedman, J.: Regularized discriminant analysis. Journal of the American Statistical Association 84, 165–175 (1989)
Guo, Y., Hastie, T., Tibshirani, R.: Regularized linear discriminant analysis and its application in microarrays. Biostatistics 8, 86–100 (2007)
Krzanowski, W., Jonathan, P., McCarthy, W., Thomas, M.: Discriminant analysis with singular covariance matrices: methods and applications to spectroscopic data. Journal of the Royal Statistical Society, Series C 44, 101–115 (1995)
Le Thi, H.A., Pham Dinh, T.: The DC (difference of convex func-tions) programming and DCA revisited with DC models of real world nonconvex optimization problems. Annals of Operations Research 133, 23–46 (2005)
Le Thi, H.A., Pham Dinh, T.: Solving a class of linearly constrained indefinite quadratic problems by DC algorithms. Journal of Global Optimization 11(3), 253–285 (1997)
Le Thi, H.A., Le Hoai, M., Pham Dinh, T.: Optimization based DC programming and DCA for Hierarchical Clustering. European Journal of Operational Research 183, 1067–1085 (2007)
Le Thi, H.A., Le Hoai, M., Nguyen, N.V., Pham Dinh, T.: A DC Programming approach for Feature Selection in Support Vector Machines learning. Journal of Advances in Data Analysis and Classification 2(3), 259–278 (2008)
Le Thi, H.A.: DC Programming and DCA, http://lita.sciences.univ-metz.fr/~lethi/DCA.html
Le Thi, H.A., Pham Dinh, T.: DC optimization algorithm for solving the trust region subproblem. SIAM Journal of Optimization 8(1), 476–505 (1998)
Le Thi, H.A., Huynh, V., Pham Dinh, T.: Exact penalty and error bounds in DC programming. Journal of Global Optimization 52(3), 509–535 (2011)
Liu, Y., Shen, X., Doss, H.: Multicategory ψ-Learning and Support Vector Machine: Computational Tools. Journal of Computational and Graphical Statistics 14, 219–236 (2005)
Liu, Y., Shen, X.: Multicategory ψ-Learning. Journal of the American Statistical Association 101, 500–509 (2006)
Pham Dinh, T., Le Thi, H.A.: Convex analysis approach to D.C. programming: Theory, algorithms and applications. Acta Mathematica Vietnamica 22(1), 289–355 (1997)
Pham Dinh, T., Le Thi, H.A.: DC optimization algorithms for solving the trust region subproblem. SIAM J. Opt. 8, 476–505 (1998)
Thiao, M., Pham Dinh, T., Le Thi, H.A.: DC programming approach for a class of nonconvex programs involving l0 norm. In: Le Thi, H.A., Bouvry, P., Pham Dinh, T. (eds.) MCO 2014. CCIS, vol. 14, pp. 358–367. Springer, Heidelberg (2008)
Tibshirani, R., Hastie, T., Narasimhan, B., Chu, G.: Diagnosis of multiple cancer types by shrunken centroids of gene expression. Proc. Natl. Acad. Sci. 99, 6567–6572 (2002)
Witten, Tibshirani: Penalized classification using Fisher’s linear discriminant. Journal Royal Statistical Society, 753–772 (2011)
Xu, P., Brock, G., Parrish, R.: Modified linear discriminant analysis approaches for classification of high-dimensional microarray data. Computational Statistics and Data Analysis 53, 1674–1687 (2009)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2014 Springer International Publishing Switzerland
About this paper
Cite this paper
Nhat, P.D., Nguyen, M.C., Le Thi, H.A. (2014). A DC Programming Approach for Sparse Linear Discriminant Analysis. In: van Do, T., Thi, H., Nguyen, N. (eds) Advanced Computational Methods for Knowledge Engineering. Advances in Intelligent Systems and Computing, vol 282. Springer, Cham. https://doi.org/10.1007/978-3-319-06569-4_4
Download citation
DOI: https://doi.org/10.1007/978-3-319-06569-4_4
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-06568-7
Online ISBN: 978-3-319-06569-4
eBook Packages: EngineeringEngineering (R0)