Skip to main content

Advertisement

Springer Nature Link
Log in

End-to-End Lifelong Learning: a Framework to Achieve Plasticities of both the Feature and Classifier Constructions

  • Published:
Cognitive Computation Aims and scope Submit manuscript

Abstract

Plasticity in our brain offers us promising ability to learn and know the world. Although great successes have been achieved in many fields, few bio-inspired machine learning methods have mimicked this ability. Consequently, when meeting large-scale or time-varying data, these bio-inspired methods are infeasible, due to the reasons that they lack plasticity and need all training data loaded into memory. Furthermore, even the popular deep convolutional neural network (CNN) models have relatively fixed structures and cannot process time varying data well. Through incremental methodologies, this paper aims at exploring an end-to-end lifelong learning framework to achieve plasticities of both the feature and classifier constructions. The proposed model mainly comprises of three parts: Gabor filters followed by max pooling layer offering shift and scale tolerance to input samples, incremental unsupervised feature extraction, and incremental SVM trying to achieve plasticities of both the feature learning and classifier construction. Different from CNN, plasticity in our model has no back propogation (BP) process and does not need huge parameters. Our incremental models, including IncPCANet and IncKmeansNet, have achieved better results than PCANet and KmeansNet on minist and Caltech101 datasets respectively. Meanwhile, IncPCANet and IncKmeansNet show promising plasticity of feature extraction and classifier construction when the distribution of data changes. Lots of experiments have validated the performance of our model and verified a physiological hypothesis that plasticity exists in high level layer better than that in low level layer.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6

Similar content being viewed by others

Explore related subjects

Discover the latest articles, news and stories from top researchers in related subjects.

References

  1. Jim M, David LG. Multiclass object recognition with sparse, localized features. In: IEEE Computer society conference on computer vision and pattern recognition; 2006. p. 11–18.

  2. Rolls ET, Milward T. A model of invariant object recognition in the visual system: learning rules, activation functions, lateral inhibition, and information-based performance measures. Neural Comput. 2000;12(11):2547–72.

    Article  CAS  PubMed  Google Scholar 

  3. Thorpe S, Fize D, Marlot C. Speed of processing in the human visual system. Nature. 1996;381(6582):520–2.

    Article  CAS  PubMed  Google Scholar 

  4. Schmidhuber J. Deep learning in neural networks: an overview. Neural Netw. 2015;61:85–117.

    Article  PubMed  Google Scholar 

  5. LeCun Y, Bengio Y. Convolutional networks for images, speech, and time series. In: The handbook of brain theory and neural networks; 1995. 3361(10).

  6. Krizhevsky A, Sutskever I, Hinton GE. Imagenet classification with deep convolutional neural networks. In: Advances in neural information processing systems; 2012. p. 1097–1105.

  7. Simonyan K, Zisserman A. Very deep convolutional networks for large-scale image recognition. arXiv:1409.1556. 2014.

  8. Szegedy C, Liu W, Jia Y, Sermanet P, Reed S, Anguelov D, Rabinovich A. Going deeper with convolutions. In: Proceedings of the IEEE conference on computer vision and pattern recognition; 2015. p. 1–9.

  9. LeCun Y, Bengio Y, Hinton G. Deep learning. Nature. 2015;521(7553):436–44.

    Article  CAS  PubMed  Google Scholar 

  10. Chan TH, Jia K, Gao S, Lu J, Zeng Z, Ma Y. PCANet: a simple deep learning baseline for image classification. IEEE Trans Image Process. 2015;24(12):5017–32.

    Article  PubMed  Google Scholar 

  11. Fukushima K. Neocognitron: a self-organizing neural network model for a mechanism of pattern recognition unaffected by shift in position. Biol Cybern. 1980;36(4):193–202.

    Article  CAS  PubMed  Google Scholar 

  12. LeCun Y, Boser B, Denker JS, Henderson D, Howard RE, Hubbard W, Jackel LD. Backpropagation applied to handwritten zip code recognition. Neural Comput. 1989;1(4):541–51.

    Article  Google Scholar 

  13. He K, Zhang X, Ren S, Sun J. Deep residual learning for image recognition. arXiv:1512.03385. 2015.

  14. Bruna J, Mallat S. Invariant scattering convolution networks. IEEE Trans Pattern Anal Mach Intell. 2013;35(8):1872C–86.

    Article  Google Scholar 

  15. Turk M, Pentland A. Eigenfaces for recognition. J Cogn Neurosci. 1991;3(1):71–86.

    Article  CAS  PubMed  Google Scholar 

  16. Hyvarinen A. Survey on independent component analysis. Neural Computing Surveys. 1999;2(4):94–128.

    Google Scholar 

  17. Bartlett MS. Independent component representations for face recognition. In: Face image analysis by unsupervised learning. US: Springer; 2001. p. 39–67.

  18. Belhumeur PN, Hespanha JP, Kriegman DJ. Eigenfaces vs. fisherfaces: recognition using class specific linear projection. IEEE Trans Pattern Anal Mach Intell. 1997;19(7):711–20.

    Article  Google Scholar 

  19. Lee H, Battle A, Raina R, Ng AY. Efficient sparse coding algorithms. In: Advances in neural information processing systems; 2007. p. 801–808.

  20. Schölkopf B, Smola A, Müller KR. Nonlinear component analysis as a kernel eigenvalue problem. Neural Comput. 1998;10(5):1299–319.

  21. Mika S, Ratsch G, Weston J, Scholkopf B, Mullers KR. Fisher discriminant analysis with kernels. In: Neural networks for signal processing IX, proceedings of the 1999 IEEE signal processing society workshop; 1999. p. 41–48.

  22. Bach FR, Jordan MI. Kernel independent component analysis. J Mach Learn Res. 2002;3:1–48.

    Google Scholar 

  23. MacQueen J. Some methods for classification and analysis of multivariate observations. In: Proceedings of the fifth Berkeley symposium on mathematical statistics and probability; 1967. 1(14): p. 281–297.

  24. Hegde A, Principe JC, Erdogmus D, Ozertem U, Rao YN, Peddaneni H. Perturbation-based eigenvector updates for on-line principal components analysis and canonical correlation analysis. Journal of VLSI Signal Processing Systems for Signal, Image and Video Technology. 2006;45(1–2):85–95.

    Article  Google Scholar 

  25. Weng J, Zhang Y, Hwang WS. Candid covariance-free incremental principal component analysis. IEEE Trans Pattern Anal Mach Intell. 2003;25:1034–40.

    Article  Google Scholar 

  26. Krasulina T. Method of stochastic approximation in the determination of the largest eigenvalue of the mathematical expectation of random matrices. In: Automatation and remote control; 1970. p. 50–56.

  27. Diehl CP, Cauwenberghs G. SVM incremental learning, adaptation and optimization. Proceedings of the International Joint Conference on Neural Networks. 2003;4:2685–90.

    Google Scholar 

  28. Thrun S. Explanation-based neural network learning: a lifelong learning approach. Springer Science & Business Media. 2012;(357).

  29. Thrun S, O’Sullivan J. Discovering structure in multiple learning tasks: the TC algorithm. ICML. 1996;96:489–97.

    Google Scholar 

  30. Thrun S, Mitchell TM. Lifelong robot learning. Robot Auton Syst. 1995;15(1–2):25–46.

    Article  Google Scholar 

  31. Caruana R. Multitask learning. Mach Learn. 1997;28:41C75.

    Article  Google Scholar 

  32. Donmez P, Carbonell JG. Proactive learning: cost-sensitive active learning with multiple imperfect oracles. In: Proceedings of the 17th ACM conference on information and knowledge management. ACM; p. 619–628.

  33. Tong S, Koller D. Active learning for structure in bayesian networks. In: IJCAI; 2001.

  34. Brunskill E, Leffler B, Li L, Littman ML, Roy N. Corl: a continuous-state offset-dynamics reinforcement learner. In: Proceedings of the 24th conference on uncertainty in artificial intelligence (UAI); 2012. p. 53–61.

  35. Mitchell TM, Cohen WW, Hruschka Jr ER, Talukdar PP, Betteridge J, Carlson A, Lao N. Never ending learning. In: AAAI; 2015. p. 2302–2310.

  36. Carlson A, Betteridge J, Kisiel B, Settles B, Hruschka Jr ER, Mitchell TM. Toward an architecture for never-ending language learning. AAAI. 2010;5:3.

    Google Scholar 

  37. Riesenhuber M, Poggio T. Hierarchical models of object recognition in cortex. Nat Neurosci. 1999;2(11):1019–25.

    Article  CAS  PubMed  Google Scholar 

  38. Larochelle H, Erhan D, Courville A, Bergstra J, Bengio Y. An empirical evaluation of deep architectures on problems with many factors of variation. In: Proceedings of the 24th international conference on machine learning. ACM; 2007. p. 473–480.

  39. Fei-Fei L, Fergus R, Perona P. Learning generative visual models from few training examples: an incremental bayesian approach tested on 101 object categories. Comput Vis Image Underst. 2007;106(1):59–70.

    Article  Google Scholar 

  40. Chang CC, Lin CJ. LIBSVM: a library for support vector machines. ACM Trans Intell Syst Technol. 2011;2(3):27.

    Article  Google Scholar 

Download references

Funding

This work was supported in part by the National Natural Science Foundation of China under Grant 61773375, Grant 61375036, and Grant 61511130079, in part by the Microsoft Collaborative Research Project.

Author information

Author notes

    Authors and Affiliations

    1. Institute of Automation, University of Chinese Academy of Sciences (UCAS), Beijing, China

      Wangli Hao, Junsong Fan, Zhaoxiang Zhang & Guibo Zhu

    2. CAS Center for Excellence in Brain Science and Intelligence Technology (CEBSIT), Beijing, China

      Zhaoxiang Zhang

    3. National Laboratory of Pattern Recognition, Institute of Automation, Chinese Academy of Sciences (NLPR, CASIA), Beijing, China

      Zhaoxiang Zhang

    Authors
    1. Wangli Hao
      View author publications

      You can also search for this author in PubMed Google Scholar

    2. Junsong Fan
      View author publications

      You can also search for this author in PubMed Google Scholar

    3. Zhaoxiang Zhang
      View author publications

      You can also search for this author in PubMed Google Scholar

    4. Guibo Zhu
      View author publications

      You can also search for this author in PubMed Google Scholar

    Corresponding author

    Correspondence to Zhaoxiang Zhang.

    Ethics declarations

    Conflict of Interests

    The authors declare that they have no conflict of interest.

    Ethical Approval

    This article does not contain any studies with human participants or animals performed by any of the authors.

    Additional information

    Wangli Hao and Junsong Fan contribute equally to this study and share first authorship.

    Rights and permissions

    Reprints and permissions

    About this article

    Check for updates. Verify currency and authenticity via CrossMark

    Cite this article

    Hao, W., Fan, J., Zhang, Z. et al. End-to-End Lifelong Learning: a Framework to Achieve Plasticities of both the Feature and Classifier Constructions. Cogn Comput 10, 321–333 (2018). https://doi.org/10.1007/s12559-017-9514-0

    Download citation

    • Received:

    • Accepted:

    • Published:

    • Issue Date:

    • DOI: https://doi.org/10.1007/s12559-017-9514-0

    Keywords