Abstract
Multi-label image annotation based on convolutional neural networks (CNN) has seen significant improvements in recent years. One problem, however, is that it is difficult to prepare complete labels for the training images and usually training data has missing or incomplete labels. Restricted Boltzmann Machines (RBM) can explore the co-occurrence distribution of the labels and estimate the missing labels efficiently. Hence we intend to propose a novel learning model for multi-label image annotation with missing labels based on CNNs, which aims to regenerate the missing labels for an image by learning the generative model of labels using an RBM. Firstly, label sets are reconstructed by the pre-trained RBM model which is trained on data with some missing labels. Then the reconstructed label sets are used as a teacher signal to train the CNN. The effectiveness of the proposed approach is confirmed by comparing the performance with baseline CNNs using various performance evaluation metrics on two different data sets. Experimental results prove that our RBM-CNN formulation exceeds the performance of the baseline CNN.
This work was partly supported by JSPS KAKENHI Grant Number 16K00239.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Boutell, M.R., Luo, J., Shen, X., Brown, C.M.: Learning multi-label scene classification. Pattern Recogn. 37(9), 1757–1771 (2004)
Chen, X., Mu, Y., Yan, S., Chua, T.S.: Efficient large-scale image annotation by probabilistic collaborative multi-label propagation. In: Proceedings of the 18th ACM International Conference on Multimedia, pp. 35–44 (2010)
Yeh, C.K., Wu, W.C., Ko, W.-J., Wang, Y.C.F.: Learning deep latent space for multi-label classification. In: AAAI, pp. 2838–2844 (2017)
Chua, T.S., Tang, J., Hong, R., Li, H., Luo, Z., Zheng, Y.T.: NUS-WIDE: a real-world web image database from National University of Singapore. In: Proceedings of ACM Conference on Image and Video Retrieval, CIVR 2009, 8–10 July 2009
Duygulu, P., Barnard, K., de Freitas, J.F.G., Forsyth, D.A.: Object recognition as machine translation: learning a lexicon for a fixed image vocabulary. In: Heyden, A., Sparr, G., Nielsen, M., Johansen, P. (eds.) ECCV 2002. LNCS, vol. 2353, pp. 97–112. Springer, Heidelberg (2002). https://doi.org/10.1007/3-540-47979-1_7
Fischer, A., Igel, C.: An introduction to restricted Boltzmann machines. In: Alvarez, L., Mejail, M., Gomez, L., Jacobo, J. (eds.) CIARP 2012. LNCS, vol. 7441, pp. 14–36. Springer, Heidelberg (2012). https://doi.org/10.1007/978-3-642-33275-3_2
Gong, Y., Jia, Y., Leung, T., Toshev, A., Ioffe, S.: Deep convolutional ranking for multilabel image annotation. arXiv preprint arXiv:1312.4894 (2013)
Hinton, G.: Training products of experts by minimizing contrastive divergence. Neural Comput. 14(8), 1771–1800 (2002)
Hugo, L., Michael, M., Razvan, P., Yoshua, B.: Learning algorithms for the classification restricted Boltzmann machine. J. Mach. Learn. Res. 13(1), 643–669 (2012)
Krizhevsky, A., Sutskever, I., Hinton, G.E.: ImageNet classification with deep convolutional neural networks. In: Pereira, F., Burges, C.J.C., Bottou, L., Weinberger, K.Q. (eds.) Advances in Neural Information Processing Systems 25, pp. 1097–1105 (2012)
Li, X., Zhao, F., Guo, Y.: Conditional restricted Boltzmann machines for multi-label learning with incomplete labels. In: AISTATS, pp. 635–643 (2015)
Mojoo, J., Kurosawa, K., Kurita, T.: Deep CNN with graph Laplacian regularization for multi-label image annotation. In: ICIAR, pp. 19–26 (2017)
Nam, J., Kim, J., Loza MencÃa, E., Gurevych, I., Fürnkranz, J.: Large-scale multi-label text classification — revisiting neural networks. In: Calders, T., Esposito, F., Hüllermeier, E., Meo, R. (eds.) ECML PKDD 2014. LNCS (LNAI), vol. 8725, pp. 437–452. Springer, Heidelberg (2014). https://doi.org/10.1007/978-3-662-44851-9_28
Qi, G.J., Hua, X.S., Rui, Y., Tang, J., Mei, T., Zhang, H.J.: Correlative multi-label video annotation. In: Proceedings of the 15th International Conference on Multimedia, pp. 17–26 (2007)
Salakhutdinov, R., Hinton, G.: Deep Boltzmann machines. In: van Dyk, D., Welling, M. (eds.) Proceedings of the Twelth International Conference on Artificial Intelligence and Statistics. Proceedings of Machine Learning Research, vol. 5, pp. 448–455. PMLR, Hilton Clearwater Beach Resort, Clearwater Beach, 16–18 April 2009. http://proceedings.mlr.press/v5/salakhutdinov09a.html
Salakhutdinov, R., Mnih, A., Hinton, G.: Restricted Boltzmann machines for collaborative filtering. In: Proceedings of the 24th International Conference on Machine Learning, ICML 2007, pp. 791–798. ACM (2007)
Tsoumakas, G., Katakis, I., Vlahavas, I.: Mining multi-label data. In: Maimon, O., Rokach, L. (eds.) Datamining and Knowledge Discovery Handbook, pp. 667–685. Springer, Boston (2009). https://doi.org/10.1007/978-0-387-09823-4_34
Wang, C., Yan, S., Zhang, L., Zhang, H.J.: Multi-label sparse coding for automatic image annotation. In: The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1643–1650 (2009)
Wang, J., Yang, Y., Mao, J., Huang, Z., Huang, C., Xu, W.: CNN-RNN: a unified framework for multi-label image classification. In: The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2285–2294 (2016)
Wu, B., Liu, Z., Wang, S., Hu, B.G., Ji, Q.: Multi-label learning with missing labels. In: The 22nd International Conference on Pattern Recognition (ICPR), pp. 2279–2289 (2014)
Wu, B., Lyu, S., Ghanem, B.: ML-MG: multi-label learning with missing labels using a mixed graph. In: The IEEE International Conference on Computer Vision (ICCV), pp. 4157–4165 (2015)
Zhang, M.L., Li, Y.K., Liu, X.Y., Geng, X.: Binary relevance for multi-label learning: an overview. Front. Comput. Sci. 12, 1–12 (2018)
Zhao, Y., Miyao, J., Kurita, T.: Multi-label image annotation via CNN with graph Laplacian regularization based on Word2Vec. In: IW-FCV (2018)
Acknowledgments
This work was partly supported by JSPS KAKENHI Grant Number 16K00239.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2019 Springer Nature Switzerland AG
About this paper
Cite this paper
Mojoo, J., Zhao, Y., Kavitha, M., Miyao, J., Kurita, T. (2019). Learning with Incomplete Labels for Multi-label Image Annotation Using CNN and Restricted Boltzmann Machines. In: Gedeon, T., Wong, K., Lee, M. (eds) Neural Information Processing. ICONIP 2019. Lecture Notes in Computer Science(), vol 11954. Springer, Cham. https://doi.org/10.1007/978-3-030-36711-4_25
Download citation
DOI: https://doi.org/10.1007/978-3-030-36711-4_25
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-36710-7
Online ISBN: 978-3-030-36711-4
eBook Packages: Computer ScienceComputer Science (R0)