Abstract
One reason why deep neural networks require lots of data is that most current training methods are only driven by the task goal information. We propose a novel instructor which can guide networks to learn useful abstraction. Since the instructor provides additional learning power, the efficiency of data is significantly improved. To get appropriate instructor, we design a generative instructor mechanism which supports learning an instructor generator from multiple tasks. The generator can generate the corresponding instructor for different tasks by using fast weights. Experiment results demonstrate the efficiency and robustness of the generated instructor. Meanwhile, our generator also shows the property relating to continuous learning.
References
[1]
R. Caruana. Multitask learning. In Learning to learn, pages 95--133. Springer, 1998.
[2]
T. Chen, I. Goodfellow, and J. Shlens. Net2net: Accelerating learning via knowledge transfer. arXiv preprint arXiv:1511.05641, 2015.
[3]
R. Collobert and J. Weston. A unified architecture for natural language processing: Deep neural networks with multitask learning. In Proceedings of the 25th international conference on Machine learning, pages 160--167. ACM, 2008.
[4]
L. Deng, G. Hinton, and B. Kingsbury. New types of deep neural network learning for speech recognition and related applications: An overview. In Acoustics, Speech and Signal Processing (ICASSP), 2013 IEEE International Conference on, pages 8599--8603. IEEE, 2013.
[5]
C. Finn, P. Abbeel, and S. Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In International Conference on Machine Learning, pages 1126--1135, 2017.
[6]
Y. Ganin and V. Lempitsky. Unsupervised domain adaptation by backpropagation. In International Conference on Machine Learning, pages 1180--1189, 2015.
[7]
R. Girshick. Fast r-cnn. arXiv preprint arXiv:1504.08083, 2015.
[8]
P. Greengard. The neurobiology of slow synaptic transmission. Science, 294(5544):1024--1030, 2001.
[9]
B. Han, J. Sim, and H. Adam. Branchout: Regularization for online ensemble tracking with convolutional neural networks. In Proceedings of IEEE International Conference on Computer Vision, pages 2217--2224, 2017.
[10]
K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770--778, 2016.
[11]
G. Hinton, O. Vinyals, and J. Dean. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531, 2015.
[12]
G. E. Hinton and D. C. Plaut. Using fast weights to deblur old memories. In Proceedings of the ninth annual conference of the Cognitive Science Society, pages 177--186, 1987.
[13]
M. Jaderberg, V. Mnih, W. M. Czarnecki, T. Schaul, J. Z. Leibo, D. Silver, and K. Kavukcuoglu. Reinforcement learning with unsupervised auxiliary tasks. arXiv preprint arXiv:1611.05397, 2016.
[14]
J. Kirkpatrick, R. Pascanu, N. Rabinowitz, J. Veness, G. Desjardins, A. A. Rusu, K. Milan, J. Quan, T. Ramalho, A. Grabska-Barwinska, et al. Overcoming catastrophic forgetting in neural networks. Proceedings of the National Academy of Sciences, 114(13):3521--3526, 2017.
[15]
B. M. Lake, R. R. Salakhutdinov, and J. Tenenbaum. One-shot learning by inverting a compositional causal process. In Advances in neural information processing systems, pages 2526--2534, 2013.
[16]
Y. LeCun, Y. Bengio, and G. Hinton. Deep learning. nature, 521(7553):436, 2015.
[17]
Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11):2278--2324, 1998.
[18]
H. Lee, C. Ekanadham, and A. Y. Ng. Sparse deep belief net model for visual area v2. In Advances in neural information processing systems, pages 873--880, 2008.
[19]
V. Mnih, K. Kavukcuoglu, D. Silver, A. A. Rusu, J. Veness, M. G. Bellemare, A. Graves, M. Riedmiller, A. K. Fidjeland, G. Ostrovski, et al. Human-level control through deep reinforcement learning. Nature, 518(7540):529, 2015.
[20]
P. Morgado and N. Vasconcelos. Semantically consistent regularization for zero-shot recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, volume 9, page 10, 2017.
[21]
T. Munkhdalai and H. Yu. Meta networks. In International Conference on Machine Learning, pages 2554--2563, 2017.
[22]
F. Navarro, S. Conjeti, F. Tombari, and N. Navab. Webly supervised learning for skin lesion classification. arXiv preprint arXiv:1804.00177, 2018.
[23]
S. Ren, K. He, R. Girshick, and J. Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. In Advances in neural information processing systems, pages 91--99, 2015.
[24]
A. Romero, N. Ballas, S. E. Kahou, A. Chassang, C. Gatta, and Y. Bengio. Fitnets: Hints for thin deep nets. arXiv preprint arXiv:1412.6550, 2014.
[25]
S. Ruder. An overview of multi-task learning in deep neural networks. arXiv preprint arXiv:1706.05098, 2017.
[26]
M. Schuster and K. K. Paliwal. Bidirectional recurrent neural networks. IEEE Transactions on Signal Processing, 45(11):2673--2681, 1997.
[27]
D. Silver, A. Huang, C. J. Maddison, A. Guez, L. Sifre, G. Van Den Driessche, J. Schrittwieser, I. Antonoglou, V. Panneershelvam, M. Lanctot, et al. Mastering the game of go with deep neural networks and tree search. nature, 529(7587):484--489, 2016.
[28]
R. K. Srivastava, J. Masci, S. Kazerounian, F. Gomez, and J. Schmidhuber. Compete to compute. In Advances in neural information processing systems, pages 2310--2318, 2013.
[29]
S. Sukhbaatar, J. Weston, R. Fergus, et al. End-to-end memory networks. In Advances in neural information processing systems, pages 2440--2448, 2015.
[30]
J. Yim, D. Joo, J. Bae, and J. Kim. A gift from knowledge distillation: Fast optimization, network minimization and transfer learning. In the IEEE Conference on Computer Vision and Pattern Recognition, 2017.
[31]
J. Yu and J. Jiang. Learning sentence embeddings with auxiliary tasks for cross-domain sentiment classification. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 236--246, 2016.
[32]
Y. Zhang, D. Shi, J. Gao, and D. Cheng. Low-rank-sparse subspace representation for robust regression. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 7445--7454, 2017.
Index Terms
- Learning to Instruct Learning
Recommendations
Personalized E-learning system with self-regulated learning assisted mechanisms for promoting learning performance
With the rapid development of Internet technologies, the conventional computer-assisted learning (CAL) is gradually moving toward to web-based learning. Additionally, instructors typically base their teaching methods to simultaneously interact with all ...
Promoting self-regulated learning in web-based learning environments
Self-regulated learning with the Internet or hypermedia requires not only cognitive learning strategies, but also specific and general meta-cognitive strategies. The purposes of the Study2000 project, carried out at the TU Dresden, were to develop and ...
Personalising learning
Flexible teaching and learning and the 'flipped classroom' are current buzzwords in higher education in Australia and elsewhere in the world. They are reflections of the progressive change in higher education over the last few decades towards more ...
Comments
Information & Contributors
Information
Published In
December 2018
365 pages
ISBN:9781450365703
DOI:10.1145/3305275
Copyright © 2018 ACM.
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]
In-Cooperation
- International Engineering and Technology Institute, Hong Kong: International Engineering and Technology Institute, Hong Kong
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 29 December 2018
Check for updates
Author Tags
Qualifiers
- Research-article
- Research
- Refereed limited
Funding Sources
- National High-tech Research and Development Plan
- Tsinghua University Initiative Scientific Research Program
- National Natural Science Foundation of China
Conference
ISBDAI '18
ISBDAI '18: International Symposium on Big Data and Artificial Intelligence
December 29 - 30, 2018
Hong Kong, Hong Kong
Acceptance Rates
ISBDAI '18 Paper Acceptance Rate 70 of 340 submissions, 21%;
Overall Acceptance Rate 70 of 340 submissions, 21%
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 51Total Downloads
- Downloads (Last 12 months)2
- Downloads (Last 6 weeks)0
Reflects downloads up to 05 Mar 2025
Other Metrics
Citations
View Options
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in