Abstract
Class Incremental Learning, learning concepts over time, is a promising research topic. Due to unknowing the number of output classes, researchers have to develop different methods to model new classes while preserving pre-trained performance. However, they will meet the catastrophic forgetting problem. That is, the performance will be deteriorated when updating the pre-trained model using new class data without including old data. Hence, in this paper, we propose a novel learning framework, namely Broad Class Incremental Learning System (BCILS) to tackle the above issue. The BCILS updates the model when there are training data from unknown classes by using the deduced iterative formula. This is different from most of the existing fine-tuning based class incremental learning algorithms. The advantages of the proposed approach including (1) easy to model; (2) flexible structure; (3) pre-trained performance preserved well. Finally, we conduct extensive experiments to demonstrate the superiority of the proposed BCILS.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Rebuffi, S.-A., Kolesnikov, A., Sperl, G., Lampert, C.H.: iCaRL: incremental classifier and representation learning. In: CVPR, pp. 5533–5542 (2017)
Shmelkov, K., Schmid, C.: Incremental learning of object detectors without catastrophic forgetting. In: ICCV, pp. 3420–3429. Karteek Alahari (2017)
Lopez-Paz, D., Ranzato, M.A.: Gradient episodic memory for continual learning. In: NIPS 2017, pp. 6470–6479 (2017)
McCloskey, M., Cohen, N.J.: Catastrophic interference in connectionist networks: the sequential learning problem. Psychol. Learn. Motiv. 24, 109–165 (1989)
French, R.M.: Catastrophic interference in connectionist networks: can it be predicted, can it be prevented? In: NIPS 1993, pp. 1176–1177 (1993)
Huang, G.-B., Zhu, Q.-Y., Siew, C.K.: Extreme learning machine: theory and applications. Neurocomputing 70(1–3), 489–501 (2006)
LeCun, Y., Bengio, Y., Hinton, G.E.: Deep learning. Nature 521(7553), 436–444 (2015)
Salakhutdinov, R., Hinton, G.E.: Deep Boltzmann machines. In: AISTATS, pp. 448–455 (2009)
LeCun, Y., Bottou, L., Bengio, Y., Haffner, P.: Gradient-based learning applied to document recognition. Proc. IEEE 86(11), 2278–2324 (1998)
Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. CoRR abs/1409.1556 (2014)
He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: CVPR, pp. 770–778 (2016)
Chen, C.L.P., Liu, Z.: Broad learning system: an effective and efficient incremental learning system without the need for deep architecture. IEEE Trans. Neural Netw. Learn. Syst. 29(1), 10–24 (2018)
Pao, Y.-H., Takefuji, Y.: Functional-link net computing: theory, system architecture, and functionalities. IEEE Comput. 25(5), 76–79 (1992)
Mensink, T., Verbeek, J., Perronnin, F., Csurka, G.: Metric learning for large scale image classification: generalizing to new classes at near-zero cost. In: Fitzgibbon, A., Lazebnik, S., Perona, P., Sato, Y., Schmid, C. (eds.) ECCV 2012. LNCS, pp. 488–501. Springer, Heidelberg (2012). https://doi.org/10.1007/978-3-642-33709-3_35
Kuzborskij, I., Orabona, F., Caputo, B.: From N to N+1: multiclass transfer incremental learning. In: CVPR 2013, pp. 3358–3365 (2013)
Ristin, M., Guillaumin, M., Gall, J., Van Gool, L.J.: Incremental learning of NCM forests for large-scale image classification. In: CVPR 2014, pp. 3654–3661 (2014)
Li, Z., Hoiem, D.: Learning without forgetting. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9908, pp. 614–629. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-46493-0_37
Triki, A.R., Aljundi, R., Blaschko, M.B., Tuytelaars, T.: Encoder based lifelong learning. In: ICCV, pp. 1329–1337 (2017)
Kirkpatrick, J., Pascanu, R., Rabinowitz, N., Veness, J., Desjardins, G., Rusu, A.A., Milan, K., Quan, J., Ramalho, T., Grabska-Barwinska, A., Hassabis, D.: Overcoming catastrophic forgetting in neural networks. Proc. Natl. Acad. Sci. 114, 3521–3526 (2017). 201611835
Chen, C.L.P., Wan, J.Z.: A rapid learning and dynamic stepwise updating algorithm for flat neural networks and the application to time-series prediction. IEEE Trans. Syst. Man Cybern. Part B 29(1), 62–72 (1999)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2018 Springer International Publishing AG, part of Springer Nature
About this paper
Cite this paper
Liu, W., Yang, H., Sun, Y., Sun, C. (2018). A Broad Neural Network Structure for Class Incremental Learning. In: Huang, T., Lv, J., Sun, C., Tuzikov, A. (eds) Advances in Neural Networks – ISNN 2018. ISNN 2018. Lecture Notes in Computer Science(), vol 10878. Springer, Cham. https://doi.org/10.1007/978-3-319-92537-0_27
Download citation
DOI: https://doi.org/10.1007/978-3-319-92537-0_27
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-92536-3
Online ISBN: 978-3-319-92537-0
eBook Packages: Computer ScienceComputer Science (R0)