Abstract
Semi-supervised deep learning, which aims to effectively use the available labeled and unlabeled data together to improve the accuracy of model, is a hot topic recently. In this paper, we propose a novel attention-based label consistency (ALC) model for semi-supervised deep learning. The relationships between different samples are well exploited by the proposed scheme of channel and sample attention, while the class estimations are required to be smooth for nearby unlabeled data. We have implemented the proposed ALC model in the framework of \(\varPi \) model and MeanTeacher, and the experimental results on three benchmark datasets, (e.g., Fashion-MNIST, CIFAR-10 and SVHN) clearly show the advantages of our proposed method.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Dai, A.M., Le, Q.V.: Semi-supervised sequence learning. In: Advances in Neural Information Processing Systems, vol. 28, pp. 3079–3087. Curran Associates, Inc. (2015)
Dai, Z., Yang, Z., Yang, F., Cohen, W.W., Salakhutdinov, R.R.: Good semi-supervised learning that requires a bad GAN. In: Advances in Neural Information Processing Systems, vol. 30, pp. 6510–6520. Curran Associates, Inc. (2017)
Goodfellow, I.J., Mirza, M., Courville, A., Bengio, Y.: Multi-prediction deep Boltzmann machines. In: Proceedings of the 26th International Conference on Neural Information Processing Systems, NIPS 2013, vol. 1, pp. 548–556. Curran Associates Inc., USA (2013)
He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2016
Hu, J., Shen, L., Sun, G.: Squeeze-and-excitation networks. In: The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2018
Krizhevsky, A., Sutskever, I., Hinton, G.E.: Imagenet classification with deep convolutional neural networks. In: Proceedings of the 25th International Conference on Neural Information Processing Systems, NIPS 2012, vol. 1, pp. 1097–1105. Curran Associates Inc., USA (2012)
Laine, S., Aila, T.: Temporal ensembling for semisupervised learning. In: Proceedings of ICLR (2017)
Lee, D.H.: Pseudo-label: the simple and efficient semi-supervised learning method for deep neural networks. In: ICML 2013 Workshop: Challenges in Representation Learning (WREPL), July 2013
Luo, Y., Zhu, J., Li, M., Ren, Y., Zhang, B.: Smooth neighbors on teacher graphs for semi-supervised learning. In: The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2018
Miyato, T., Maeda, S., Ishii, S., Koyama, M.: Virtual adversarial training: a regularization method for supervised and semi-supervised learning. IEEE Trans. Pattern Anal. Mach. Intell. 41, 1979–1993 (2018)
Netzer, Y., Wang, T., Coates, A., Bissacco, A., Wu, B., Ng, A.Y.: Reading digits in natural images with unsupervised feature learning. In: NIPS Workshop on Deep Learning and Unsupervised Feature Learning (2011)
Rasmus, A., Valpola, H., Honkala, M., Berglund, M., Raiko, T.: Semi-supervised learning with ladder network. In: Advances in Neural Information Processing Systems, July 2015
Rifai, S., Dauphin, Y.N., Vincent, P., Bengio, Y., Muller, X.: The manifold tangent classifier. In: Proceedings of the 24th International Conference on Neural Information Processing Systems, NIPS 2011, pp. 2294–2302. Curran Associates Inc., USA (2011)
Salimans, T., et al.: Improved techniques for training GANs. In: Advances in Neural Information Processing Systems, vol. 29, pp. 2234–2242 (2016)
Szegedy, C., et al.: Going deeper with convolutions. In: The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2015
Tarvainen, A., Valpola, H.: Mean teachers are better role models: weight-averaged consistency targets improve semisupervised deep learning results. In: Advances in Neural Information Processing Systems, pp. 1195–1204 (2017)
Wang, F., et al.: Residual attention network for image classification. In: The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), July 2017
Woo, S., Park, J., Lee, J.Y., So Kweon, I.: CBAM: convolutional block attention module. In: The European Conference on Computer Vision (ECCV), September 2018
Wu, H., Prasad, S.: Semi-supervised deep learning using pseudo labels for hyperspectral image classification. IEEE Trans. Image Process. 27(3), 1259–1270 (2018)
Xiao, H., Rasul, K., Vollgraf, R.: Fashion-MNIST: a novel image dataset for benchmarking machine learning algorithms (2017)
Xu, D., Wang, W., Tang, H., Liu, H., Sebe, N., Ricci, E.: Structured attention guided convolutional neural fields for monocular depth estimation. In: The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2018
Acknowledgement
This work is partially supported by National Natural Science Foundation of China (Grant no. 61772568), the Guangzhou Science and Technology Program (Grant no. 201804010288), and the Fundamental Research Funds for the Central Universities (Grant no. 18lgzd15).
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2019 Springer Nature Switzerland AG
About this paper
Cite this paper
Chen, J., Yang, M. (2019). Attention-Based Label Consistency for Semi-supervised Deep Learning. In: Lin, Z., et al. Pattern Recognition and Computer Vision. PRCV 2019. Lecture Notes in Computer Science(), vol 11857. Springer, Cham. https://doi.org/10.1007/978-3-030-31654-9_39
Download citation
DOI: https://doi.org/10.1007/978-3-030-31654-9_39
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-31653-2
Online ISBN: 978-3-030-31654-9
eBook Packages: Computer ScienceComputer Science (R0)