Abstract
Knowledge transfer is the task of transferring the knowledge learned by a model A to a new model B. This task is essential in Deep Learning, since there are complex models with excellent results, but computationally costly to be executed. The Similarity Knowledge Transfer (SKT) method proposes an approach to transfer the knowledge layer-by-layer between a donor model and a receiver model. This transfer is carried out through the representations learned by the layers from the teacher model. Despite presenting good results, the SKT method proposes just a way to transfer knowledge between two models. Therefore, this work presents the Similarity Knowledge Transfer Ensemble (SKTE) method, a generic form of SKT that allows the transfer from several teachers to a single student model. We carried out experiments with the CIFAR10 benchmark, where the results obtained showed promising results in this activity.
Supported by organization CAPES (Brazilian research agency).
D. Pereira and F. Santos—Equal contribution.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Buciluǎ, C., Caruana, R., Niculescu-Mizil, A.: Model compression. In: Proceedings of the 12th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 535–541. ACM (2006)
Carneiro, D., Novais, P., Pêgo, J.M., Sousa, N., Neves, J.: Using mouse dynamics to assess stress during online exams. In: Onieva, E., Santos, I., Osaba, E., Quintián, H., Corchado, E. (eds.) HAIS 2015. LNCS (LNAI), vol. 9121, pp. 345–356. Springer, Cham (2015). https://doi.org/10.1007/978-3-319-19644-2_29
Chen, T., Goodfellow, I., Shlens, J.: Net2net: accelerating learning via knowledge transfer. arXiv preprint arXiv:1511.05641 (2015)
Cho, J.H., Hariharan, B.: On the Efficacy of Knowledge Distillation (2019)
Furlanello, T., Lipton, Z.C., Tschannen, M., Itti, L., Anandkumar, A.: Born Again Neural Networks (2018)
Hinton, G., Vinyals, O., Dean, J.: Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531 (2015)
Krizhevsky, A.: Learning Multiple Layers of Features From Tiny Images. University of Toronto (05 2009)
Lima, L., Novais, P., Costa, R., Bulas-Cruz, J., Neves, J.: Group decision making and quality-of-information in e-health systems. Log. J. IGPL 19, 315–332 (2011). https://doi.org/10.1093/jigpal/jzq029
Malinin, A., Gales, M.: Predictive uncertainty estimation via prior networks. In: Advances in Neural Information Processing Systems, pp. 7047–7058 (2018)
Malinin, A., Mlodozeniec, B., Gales, M.: Ensemble Distribution Distillation (2019)
Mirzadeh, S.I., Farajtabar, M., Li, A., Levine, N., Matsukawa, A., Ghasemzadeh, H.: Improved Knowledge Distillation Via Teacher Assistant (2019)
Passalis, N., Tefas, A.: Unsupervised knowledge transfer using similarity embeddings. IEEE Trans. Neural Netw. Learn. Syst. 30(3), 946–950 (2019). https://doi.org/10.1109/TNNLS.2018.2851924
Pimenta, A., Carneiro, D., Novais, P., Neves, J.: Monitoring mental fatigue through the analysis of keyboard and mouse interaction patterns. In: Pan, J.-S., Polycarpou, M.M., Woźniak, M., de Carvalho, A.C.P.L.F., Quintián, H., Corchado, E. (eds.) HAIS 2013. LNCS (LNAI), vol. 8073, pp. 222–231. Springer, Heidelberg (2013). https://doi.org/10.1007/978-3-642-40846-5_23
Romero, A., Ballas, N., Kahou, S.E., Chassang, A., Gatta, C., Bengio, Y.: Fitnets: hints for thin deep nets. arXiv preprint arXiv:1412.6550 (2014)
Tang, Z., Wang, D., Zhang, Z.: Recurrent neural network training with dark knowledge transfer. In: 2016 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 5900–5904. IEEE (2016)
Tian, Y., Krishnan, D., Isola, P.: Contrastive Representation Distillation (2019)
Yang, C., Xie, L., Qiao, S., Yuille, A.: Knowledge Distillation in Generations: More Tolerant Teachers Educate Better Students (2018)
Zagoruyko, S., Komodakis, N.: Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks Via Attention Transfer (2016)
Acknowledgments
This work has been supported by FCT - Fundação para a Ciência e a Tecnologia within the R&D Units project scope UIDB/00319/2020. The authors from affiliations 1 and 2 thank CAPES and CNPq for the financial support.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2020 Springer Nature Switzerland AG
About this paper
Cite this paper
Pereira, D., Santos, F., Matos, L.N., Novais, P., Zanchettin, C., Ludermir, T.B. (2020). On Analysing Similarity Knowledge Transfer by Ensembles. In: Analide, C., Novais, P., Camacho, D., Yin, H. (eds) Intelligent Data Engineering and Automated Learning – IDEAL 2020. IDEAL 2020. Lecture Notes in Computer Science(), vol 12490. Springer, Cham. https://doi.org/10.1007/978-3-030-62365-4_20
Download citation
DOI: https://doi.org/10.1007/978-3-030-62365-4_20
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-62364-7
Online ISBN: 978-3-030-62365-4
eBook Packages: Computer ScienceComputer Science (R0)