Abstract
The paper considers general machine learning models, where knowledge transfer is positioned as the main method to improve their convergence properties. Previous research was focused on mechanisms of knowledge transfer in the context of SVM framework; the paper shows that this mechanism is applicable to neural network framework as well. The paper describes several general approaches for knowledge transfer in both SVM and ANN frameworks and illustrates algorithmic implementations and performance of one of these approaches for several synthetic examples.
Similar content being viewed by others
References
Brachman, R., Levesque, H.: Knowledge Representation and Reasoning. Morgan Kaufmann, San Francisco (2004)
Fouad, S., Tino, P., Raychaudhury, S., Schneider, P.: Incorporating privileged information through metric learning. IEEE Transactions on Neural Networks and Learning Systems 24, 1086–1098 (2013)
Demuth, H., Beale, M.: Neural Network Toolbox for Use with MATLAB. Mathworks, Inc., Natick, MA (2000)
Hagan, M., Demuth, H., Beale, M.: Neural Network Design. PWS Publishing Co, Boston, MA (1996)
Heaton, J.: Introduction to Neural Networks for Java. Heaton Research Inc (2008)
Ilin, R., Streltsov, S., Izmailov, R.: Learning with privileged information for improved target classification. International Journal of Monitoring and Surveillance Technologies Research 2(3), 5–66 (2014)
Kolmogorov, A.: Mathematics as a Profession. Moscow, Nauka (1988). (in Russian)
LeCun, Y., Bottou, L., Orr, G., Müller, K.: Efficient BackProp. In: Orr, G., Müller, K. (eds.) Neural Networks: Tricks of the Trade, pp 5–50. Springer, Heidelberg (1998)
Lichman, M.: UCI Machine Learning Repository [http://archive.ics.uci.edu/ml]. Irvine, CA: University of California School of Information and Computer Science (2013)
Pechyony, D., Izmailov, R., Vashist, A., Vapnik, V.: SMO-style algorithms for learning using privileged information. In: 2010 International Conference on Data Mining, pp 235–241 (2010)
Ribeiro, B., Silva, C., Chen, N., Vieirac, A., Carvalho das Nevesd, J.: Enhanced default risk models with SVM + . Expert Syst. Appl. 39, 10140–10152 (2012)
Sharmanska, V., Lampert, C.: Learning to rank using privileged information. In: 2013 IEEE International Conference on Computer Vision (ICCV), pp 825–832 (2013)
Vapnik, V.: The Nature of Statistical Learning Theory. Springer, New York (1995)
Vapnik, V.: Statistical Learning Theory. Wiley, New York (1998)
Vapnik, V.: Estimation of Dependencies Based on Empirical Data, 2nd edn. Springer, New York (2006)
Vapnik, V., Vashist, A.: A new learning paradigm: learning using privileged information. Neural Netw. 22, 546–557 (2009)
Vapnik, V., Izmailov, R.: Learning with intelligent teacher: similarity control and knowledge transfer. In: Statistical Learning and Data Sciences. LNAI 9047, pp 3–32. Springer, London (2015)
Vapnik, V., Izmailov, R.: Learning using privileged information: similarity control and knowledge transfer. J. Mach. Learn. Res. 16, 2023–2049 (2015)
Yang, H., Patras, I.: Privileged information-based conditional regression forest for facial feature detection. In: 2013 IEEE International Conference on Automatic Face and Gesture Recognition, pp 1–6 (2013)
Author information
Authors and Affiliations
Corresponding author
Additional information
This material is based upon work partially supported by AFRL and DARPA under contract FA8750-14-C-0008 and the work partially supported by AFRL under contract FA9550-15-1-0502. Any opinions, findings and / or conclusions in this material are those of the authors and do not necessarily reflect the views of AFRL and DARPA.
Rights and permissions
About this article
Cite this article
Vapnik, V., Izmailov, R. Knowledge transfer in SVM and neural networks. Ann Math Artif Intell 81, 3–19 (2017). https://doi.org/10.1007/s10472-017-9538-x
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10472-017-9538-x
Keywords
- Intelligent teacher
- Privileged information
- Similarity control
- Knowledge transfer
- Knowledge representation
- Frames
- Support vector machine
- Neural network
- Classification
- Learning theory
- Regression