Skip to main content
SpringerLink
Log in

Dynamically building diversified classifier pruning ensembles via canonical correlation analysis

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

Empirical studies on ensemble learning that combines multiple classifiers have shown that, it is an effective technique to improve accuracy and stability of a single classifier. In this paper, we propose a novel method of dynamically building diversified sparse ensembles. We first apply a technique known as the canonical correlation to model the relationship between the input data variables and output base classifiers. The canonical (projected) output classifiers and input training data variables are encoded globally through a multi-linear projection of CCA, to decrease the impacts of noisy input data and incorrect classifiers to a minimum degree in such a global view. Secondly, based on the projection, a sparse regression method is used to prune representative classifiers by combining classifier diversity measurement. Based on the above methods, we evaluate the proposed approach by several datasets, such as UCI and handwritten digit recognition. Experimental results of the study show that, the proposed approach achieves better accuracy as compared to other ensemble methods such as QFWEC, Simple Vote Rule, Random Forest, Drep and Adaboost.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4

Similar content being viewed by others

Notes

  1. http://www.cad.zju.edu.cn/home/dengcai/Data/MLData.html.

  2. http://www.cs.nyu.edu/~roweis/data.html.

  3. http://archive.ics.uci.edu/ml/about.html.

References

  1. Bao BK, Zhu G, Shen J, Yan S (2013) Robust image analysis with sparse representation on quantized visual features. IEEE Trans Image Process 22(3):860–871

    Article  MathSciNet  MATH  Google Scholar 

  2. Breiman L (1996) Bagging predictors. Mach Learn 24(2):123–140

    MATH  Google Scholar 

  3. Britto AS, Sabourin R, Oliveira LE (2014) Dynamic selection of classifiers-a comprehensive review. Pattern Recogn 47(11):3665–3680

    Article  Google Scholar 

  4. Chen H, Tiňo P, Yao X (2009) Predictive ensemble pruning by expectation propagation. IEEE Trans Knowl Data Eng 21(7):999–1013

    Article  Google Scholar 

  5. Efron B, Hastie T, Johnstone I, Tibshirani R (2004) Least angle regression. Ann Stat 32(2):407–451

    Article  MathSciNet  MATH  Google Scholar 

  6. Franc V, Hlavác V (2002) Multi-class support vector machine. In: 2002. Proceedings. 16th international conference on pattern recognition, vol 2, pp 236–239

  7. Freund Y (1996) Experiments with a new boosting algorithm. In: Thirteenth international conference on machine learning, pp 148–156

  8. Fumera G, Roli F (2005) A theoretical and experimental analysis of linear combiners for multiple classifier systems. IEEE Trans Pattern Anal Mach Intell 27 (6):942–956

    Article  Google Scholar 

  9. Gao X, Sun Q, Xu H (2017) Multiple-rank supervised canonical correlation analysis for feature extraction, fusion and recognition. Pergamon Press Inc.,

  10. Ghorai S, Mukherjee A, Sengupta S, Dutta PK (2011) Cancer classification from gene expression data by nppc ensemble. IEEE/ACM Trans Comput Biol Bioinform 8(3):659–671

    Article  Google Scholar 

  11. Giacinto G, Roli F (2001) Design of effective neural network ensembles for image classification purposes. Image Vis Comput 19(9):699–707

    Article  Google Scholar 

  12. Hardoon DR, Szedmak SR, Shawe-Taylor JR (2004) Canonical correlation analysis: an overview with application to learning methods. Neural Comput 16 (12):2639–2664

    Article  MATH  Google Scholar 

  13. Ho TK (1998) The random subspace method for constructing decision forests. IEEE Trans Pattern Anal Mach Intell 20(8):832–844

    Article  Google Scholar 

  14. Ko AH, Sabourin R, Britto JrAS (2008) From dynamic classifier selection to dynamic ensemble selection. Pattern Recogn 41(5):1718–1731

    Article  MATH  Google Scholar 

  15. Krogh A, Vedelsby J et al. (1995) Neural network ensembles, cross validation, and active learning. Adv Neural Inf Proces Syst 7:231–238

    Google Scholar 

  16. Kuncheva LI (2004) Combining Pattern Classifiers: Methods and Algorithms. Wiley-Interscience

  17. Kuncheva LI (2013) A bound on kappa-error diagrams for analysis of classifier ensembles. IEEE Trans Knowl Data Eng 25(3):494–501

    Article  Google Scholar 

  18. Kuncheva LI, Rodriguez JJ (2007) Classifier ensembles with a random linear oracle. IEEE Trans Knowl Data Eng 19(4):500–508

    Article  Google Scholar 

  19. Kuncheva LI, Rodríguez JJ (2014) A weighted voting framework for classifiers ensembles. Knowl Inf Syst 38(2):259–275

    Article  Google Scholar 

  20. Kuncheva LI, Whitaker CJ (2003) Measures of diversity in classifier ensembles and their relationship with the ensemble accuracy. Machine learning 51(2):181–207

    Article  MATH  Google Scholar 

  21. Li N, Yu Y, Zhou ZH (2012) Diversity regularized ensemble pruning. In: European conference on machine learning and knowledge discovery in databases, pp 330–345

  22. Liu L, Shao L, Rockett P (2013) Boosted key-frame selection and correlated pyramidal motion-feature representation for human action recognition. Pattern Recogn 46(7):1810–1818

    Article  Google Scholar 

  23. Liu J, Shang S, Zheng K, Wen J-R (2016) Multi-view ensemble learning for dementia diagnosis from neuroimaging: an artificial neural network approach. Neurocomputing 195:112–116

    Article  Google Scholar 

  24. Liu W, Zha ZJ, Wang Y, Lu K, Tao D (2016) P-laplacian regularized sparse coding for human activity recognition. IEEE Trans Ind Electron 63(8):5120–5129

    Google Scholar 

  25. Mao S, Jiao L, Xiong L, Gou S, Chen B, Yeung S-K (2015) Weighted classifier ensemble based on quadratic form. Pattern Recogn 48(5):1688–1706

    Article  MATH  Google Scholar 

  26. Martínez-Muñoz G., Hernández-Lobato D., Suárez A. (2009) An analysis of ensemble pruning techniques based on ordered aggregation. IEEE Trans Pattern Anal Mach Intell 31(2):245–259

    Article  Google Scholar 

  27. Needell D, Tropp JA (2009) Cosamp iterative signal recovery from incomplete and inaccurate samples. Appl Comput Harmon Anal 26(3):301–321

    Article  MathSciNet  MATH  Google Scholar 

  28. Quan Y, Xu Y, Sun Y, Huang Y (2016) Supervised dictionary learning with multiple classifier integration. Pattern Recogn 55:247–260

    Article  Google Scholar 

  29. Saitta L (2006) Hypothesis diversity in ensemble classification. In: International symposium on methodologies for intelligent systems, pp 662–670

  30. Sim J, Wright CC (2005) The kappa statistic in reliability studies: use, interpretation, and sample size requirements. Phys Ther 85(3):257

    Google Scholar 

  31. Skalak DB et al (1996) The sources of increased accuracy for two proposed boosting algorithms. In: Proceedings American association for artificial intelligence, AAAI-96, Integrating Multiple Learned Models Workshop, pp 120–125

  32. Tang S, Zhang Y-D, Xu Z-X, Li H-J, Zheng Y-T, Li J-T (2015) An efficient concept detection system via sparse ensemble learning. Neurocomputing 169:124–133

    Article  Google Scholar 

  33. Tao D, Tang X, Li X, Wu X (2006) Asymmetric bagging and random subspace for support vector machines-based relevance feedback in image retrieval. IEEE Trans Pattern Anal Mach Intell 28(7):1088–1099

    Article  Google Scholar 

  34. Ueda N (2000) Optimal linear combination of neural networks for improving classification performance. IEEE Trans Pattern Anal Mach Intell 22(2):207–215

    Article  Google Scholar 

  35. Via J, Santamaria I, Perez J (2005) Canonical correlation analysis (cca) algorithms for multiple data sets: application to blind simo equalization. In: Signal processing conference, 2005 European, pp 1–4

  36. Wahlberg B, Boyd S, Annergren M, Wang Y (2012) An admm algorithm for a class of total variation regularized estimation problems. IFAC Proceedings Volumes 45(16):83–88

    Article  Google Scholar 

  37. Wang X-Z, Xing H-J, Li Y, Hua Q, Dong C-R, Pedrycz W (2015) A study on relationship between generalization abilities and fuzziness of base classifiers in ensemble learning. IEEE Trans Fuzzy Syst 23(5):1638–1654

    Article  Google Scholar 

  38. Woloszynski T, Kurzynski M (2011) A probabilistic model of classifier competence for dynamic ensemble selection. Pattern Recogn 44(10):2656–2668

    Article  MATH  Google Scholar 

  39. Wright J, Ma Y, Mairal J, Sapiro G, Huang TS, Yan S (2010) Sparse representation for computer vision and pattern recognition. Proc IEEE 98(6):1031–1044

    Article  Google Scholar 

  40. Yang X, Liu W, Tao D, Cheng J (2017) Canonical correlation analysis networks for two-view image recognition. Information Sciences An International Journal 385(C):338–352

    Article  Google Scholar 

  41. Yin XC, Yang C, Hao HW (2014) Learning to diversify via weighted kernels for classifier ensemble. Eprint Arxiv

  42. Yin X-C, Huang K, Hao H-W (2015) De 2: dynamic ensemble of ensembles for learning nonstationary data. Neurocomputing 165:14–22

    Article  Google Scholar 

  43. Zhang L, Zhou W (2010) On the sparseness of 1-norm support vector machines. Neural Netw 23(3):373–385

    Article  MATH  Google Scholar 

  44. Zhang L, Zhou W-D (2011) Sparse ensembles using weighted combination methods based on linear programming. Pattern Recogn 44(1):97–106

    Article  MathSciNet  MATH  Google Scholar 

  45. Zhang Y, Burer S, Street WN (2006) Ensemble pruning via semi-definite programming. J Mach Learn Res 7:1315–1338

    MathSciNet  MATH  Google Scholar 

  46. Zhang E, Zhang X, Jiao L, Li L, Hou B (2016) Spectral–spatial hyperspectral image ensemble classification via joint sparse representation. Pattern Recogn 59:42–54

    Article  Google Scholar 

  47. Zhao Z, Jiao L, Liu F, Zhao J, Chen P (2016) Semisupervised discriminant feature learning for sar image category via sparse ensemble. IEEE Trans Geosci Remote Sens 54(6):3532–3547

    Article  Google Scholar 

  48. Zhou ZH, Wu JX, Jiang Y, Chen SF (2001) Genetic algorithm based selective neural network ensemble. In: International joint conference on artificial intelligence, pp 797–802

Download references

Acknowledgements

This work was funded in part by the National Natural Science Foundation of China(No.61572240, 61601202,61502208), the Open Project Program of the National Laboratory of Pattern Recognition(NLPR)(No.201600005), Natural Science Foundation of Jiangsu Province (Grant No. BK20140571).

Author information

Authors and Affiliations

  1. School of Computer Science and Telecommunication Engineering, Jiangsu University, Zhenjiang, China

    Zhong-Qiu Jiang, Xiang-Jun Shen, Jian-Ping Gou & Liangjun Wang

  2. School of Information Science and Technology, University of Science and Technology of China, Hefei, China

    Zheng-Jun Zha

Authors
  1. Zhong-Qiu Jiang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Xiang-Jun Shen
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Jian-Ping Gou
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Liangjun Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Zheng-Jun Zha
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Xiang-Jun Shen.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Jiang, ZQ., Shen, XJ., Gou, JP. et al. Dynamically building diversified classifier pruning ensembles via canonical correlation analysis. Multimed Tools Appl 78, 271–288 (2019). https://doi.org/10.1007/s11042-018-5718-x

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-018-5718-x

Keywords

Search

Navigation