Skip to main content
SpringerLink
Log in

Covariance Matrix Estimation with Multi-Regularization Parameters based on MDL Principle

  • Published:
Neural Processing Letters Aims and scope Submit manuscript

Abstract

Regularization is a solution for the problem of unstable estimation of covariance matrix with a small sample set in Gaussian classifier. In many applications such as image restoration, sparse representation, we have to deal with multi-regularization parameters problem. In this paper, the case of covariance matrix estimation with multi-regularization parameters is investigated, and an estimate method called as KLIM_L is derived theoretically based on Minimum Description Length (MDL) principle for the small sample size problem with high dimension setting. KLIM_L estimator can be regarded as a generalization of KLIM estimator in which local difference in each dimension is considered. Under the framework of MDL principle, a selection method of multi-regularization parameters is also developed based on the minimization of the Kullback-Leibler information measure, which is simply and directly estimated by point estimation under the approximation of two-order Taylor expansion. The computational cost to estimate multi-regularization parameters with KLIM_L method is less than those with RDA (Regularized Discriminant Analysis) and LOOC (leave-one-out covariance matrix estimate) in which cross validation technique is adopted. Experiments show that higher classification accuracy can be achieved by using the proposed KLIM_L estimator.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

Notes

  1. In this paper, KLIM is used to stand for an estimation method.

References

  1. Bishop CM (2007) Pattern recognition and machine learning. Springer, New York

    Google Scholar 

  2. Everitt BS, Hand D (1981) Finite mixture distributions. Chapman and Hall, London

    Book  MATH  Google Scholar 

  3. McLachlan GJ, Basford KE (1988) Mixture models: inference and applications to clustering. Dekker, New York

    MATH  Google Scholar 

  4. Jolliffe IT (1996) Principal component analysis. Springer, New York

    Google Scholar 

  5. Friedman JH (1989) Regularized discriminant analysis. J Am Stat Assoc 84(405):165–175

    Article  Google Scholar 

  6. Hoffbeck JP, Landgrebe DA (1996) Covariance matrix estimation and classification with limited training data. IEEE Trans Pattern Anal Mach Intell 18(7):763–767

    Article  Google Scholar 

  7. Srivastava S, Gupta MR, Frigyik BA (2007) Bayesian quadratic discriminant analysis. J Mach Learn Res 8:1277–1305

    MathSciNet  MATH  Google Scholar 

  8. Friedman J, Hastie T, Tibshirani R (2008) Sparse inverse covariance estimation with the graphical lasso. Biostatistics 9(3):432–441

    Article  MATH  Google Scholar 

  9. Daniels MJ, Kass RE (2001) Shrinkage estimators for covariance matrices. Biometrics 57(4):1173–1184

    Article  MathSciNet  MATH  Google Scholar 

  10. Ledoit O, Wolf M (2004) A well-conditioned estimator for large-dimensional covariance matrices. J Multivar Anal 88(2):365–411

    Google Scholar 

  11. Schafer J, Strimmer K (2005) A shrinkage approach to large-scale covariance matrix estimation and implications for functional genomics. Stat Appl Genet Mol Biol 4(1):1–28

    MathSciNet  Google Scholar 

  12. Kuo BC, Landgrebe DA (2002) A covariance estimator for small sample size classification problems and its application to feature extraction. IEEE Trans Geosci Remote Sens 40(4):814–819

    Article  Google Scholar 

  13. Bickel PJ, Levina E (2008) Regularized estimation of large covariance matrices. Ann Stat 36(1):199–227

    Article  MathSciNet  MATH  Google Scholar 

  14. Cao G, Bouman C, Webb K (2009) Noniterative map reconstruction using sparse matrix representations. IEEE Trans Image Process 18(9):2085–2099

    Article  MathSciNet  Google Scholar 

  15. Cao G, Bachega LR, Bouman CA (2011) The sparse matrix transform for covariance estimation and analysis of high dimensional signals. IEEE Trans Image Process 20(3):625–640

    Article  MathSciNet  Google Scholar 

  16. Rivals I, Personnaz L (1999) On cross validation for model selection. Neural Comput 11:863–870

    Article  Google Scholar 

  17. Guo P, Jia Y, Lyu MR (2008) A study of regularized Gaussian classifier in high-dimension small sample set case based on MDL principle with application to spectrum recognition. Pattern Recognit 41:2842–2854

    Article  MATH  Google Scholar 

  18. Redner RA, Walker HF (1984) Mixture densities, maximum likelihood and the EM algorithm. SIAM Rev 26:195–239

    Article  MathSciNet  MATH  Google Scholar 

  19. Aeberhard S, Coomans D, Vel DO (1994) Comparative analysis of statistical pattern recognition methods in high dimensional settings. Pattern Recognit 27(8):1065–1077

    Article  Google Scholar 

  20. Rissanen J (1978) Modeling by shortest data description. Automatica 14:465–471

    Article  MATH  Google Scholar 

  21. Barron A, Rissanen J, Yu B (1998) The minimum description length principle in coding and modelling. IEEE Trans Inf Theory 44(6):2743–2760

    Article  MathSciNet  MATH  Google Scholar 

  22. Kullback S (1959) Information theory and statistics. Wiley, New York

    MATH  Google Scholar 

  23. Webb AR (1994) Statistical pattern recognition. Oxford University Press, London

    Google Scholar 

  24. Nene SA, Nayar SK, Murase H (1996) Columbia Object Image Library (COIL-20). Technical report CUCS-005-96

  25. Frank A, Asuncion A (2010) UCI Machine Learning Repository. http://archive.ics.uci.edu/ml. Irvine, CA: University of California, School of Information and Computer Science

Download references

Acknowledgments

The research work described in this paper was fully supported by the grants from the National Natural Science Foundation of China (Project No. 90820010, 60911130513). Prof. Guo is the author to whom the correspondence should be addressed, his e-mail address is pguo@ieee.org.

Author information

Authors and Affiliations

  1. The Laboratory of Image Processing and Pattern Recognition, Beijing Normal University, Beijing, 100875, China

    Xiuling Zhou & Ping Guo

  2. Research Department, Beijing City University, Beijing, China

    Xiuling Zhou

  3. The Faculty of Science & Technology, University of Macau, Macau, SAR China

    C. L. Philip Chen

Authors
  1. Xiuling Zhou
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Ping Guo
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. C. L. Philip Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Ping Guo.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Zhou, X., Guo, P. & Chen, C.L.P. Covariance Matrix Estimation with Multi-Regularization Parameters based on MDL Principle. Neural Process Lett 38, 227–238 (2013). https://doi.org/10.1007/s11063-012-9272-7

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11063-012-9272-7

Keywords

Search

Navigation