Abstract
To learn a well-performed image annotation model, a large number of labeled samples are usually required. Although the unlabeled samples are readily available and abundant, it is a difficult task for humans to annotate large numbers of images manually. In this article, we propose a novel semi-supervised approach based on adaptive weighted fusion for automatic image annotation that can simultaneously utilize the labeled data and unlabeled data to improve the annotation performance. At first, two different classifiers, constructed based on support vector machine and covolutional neural network, respectively, are trained by different features extracted from the labeled data. Therefore, these two classifiers are independently represented as different feature views. Then, the corresponding features of unlabeled images are extracted and input into these two classifiers, and the semantic annotation of images can be obtained respectively. At the same time, the confidence of corresponding image annotation can be measured by an adaptive weighted fusion strategy. After that, the images and its semantic annotations with high confidence are submitted to the classifiers for retraining until a certain stop condition is reached. As a result, we can obtain a strong classifier that can make full use of unlabeled data. Finally, we conduct experiments on four datasets, namely, Corel 5K, IAPR TC12, ESP Game, and NUS-WIDE. In addition, we measure the performance of our approach with standard criteria, including precision, recall, F-measure, N+, and mAP. The experimental results show that our approach has superior performance and outperforms many state-of-the-art approaches.
- David M. Blei and Michael I. Jordan. 2003. Modeling annotated data. In Proceedings of the 26th Annual International ACM SIGIR Conference on Research and Development in Informaion Retrieval (SIGIR). ACM, 127–134.Google Scholar
- David M. Blei, Andrew Y. Ng, and Michael I. Jordan. Jan. 2003. Latent Dirichlet allocation. J. Mach. Learn. Res. 3 (Jan. 2003), 993–1022.Google Scholar
- Avrim Blum and Tom Mitchell. 1998. Combining labeled and unlabeled data with co-training. In Proceedings of the 11th Annual Conference on Computational Learning Theory (COLT’98). 92–100.Google ScholarDigital Library
- Gustavo Carneiro, Antoni B. Chan, Pedro J. Moreno, and Nuno Vasconcelos. 2007. Supervised learning of semantic classes for image annotation and retrieval. IEEE Trans. Pattern Anal. Mach. Intell. 29, 3 (2007), 394–410.Google ScholarDigital Library
- Hakan Cevikalp, Burak Benligiray, and Omer Nezih Gerek. Apr. 2020. Semi-supervised robust deep neural networks for multi-label image classification. Pattern Recogn. 100, Article 107164 (Apr. 2020), 9 pages.Google Scholar
- Chih-Chung Chang and Chih-Jen Lin. 2011. LIBSVM: A library for support vector machines. ACM Trans. Intell. Syst. Technol. 2, 3 (2011), 1–27.Google ScholarDigital Library
- Xiangyu Chen, Yadong Mu, Shuicheng Yan, and Tat-Seng Chua. 2010. Efficient large-scale image annotation by probabilistic collaborative multi-label propagation. In Proceedings of the 18th ACM International Conference on Multimedia (MM’10). ACM, 35–44.Google ScholarDigital Library
- Tat-Seng Chua, Jinhui Tang, Richang Hong, Haojie Li, Zhiping Luo, and Yantao Zheng. 2009. NUS-WIDE: A real-world web image database from National University of Singapore. In Proceedings of the ACM International Conference on Image and Video Retrieval (CIVR’09). ACM, 1–9.Google ScholarDigital Library
- Ritendra Datta, Dhiraj Joshi, Jia Li, and James Z. Wang. 2008. Image retrieval: Ideas, influences, and trends of the new age. Comput. Surv. 40, 2 (2008), 1–60.Google ScholarDigital Library
- Janez Demšar. Jun. 2006. Statistical comparisons of classifiers over multiple data sets. J. Mach. Learn. Res. 7 (Jun. 2006), 1–30.Google Scholar
- Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. 2009. ImageNet: A large-scale hierarchical image database. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR’09). IEEE, 248–255.Google ScholarCross Ref
- Pinar Duygulu, Kobus Barnard, Joao F. G. de Freitas, and David A. Forsyth. 2002. Object recognition as machine translation: Learning a lexicon for a fixed image vocabulary. In Proceedings of the European Conference on Computer Vision (ECCV’02). Springer, 97–112.Google Scholar
- Hugo Jair Escalante, Carlos A. Hernández, Jesus A. Gonzalez, Aurelio López-López, Manuel Montes, Eduardo F. Morales, L. Enrique Sucar, Luis Villaseñor, and Michael Grubinger. 2010. The segmented and annotated IAPR TC-12 benchmark. Comput. Vis. Image Understand. 114, 4 (2010), 419–428.Google ScholarDigital Library
- Shao Lei Feng, Raghavan Manmatha, and Victor Lavrenko. 2004. Multiple Bernoulli relevance models for image and video annotation. In Proceedings of the 2004 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR’04), Vol. 2. IEEE, 1002–1009.Google ScholarCross Ref
- King Shy Goh, Edward Y. Chang, and Beitao Li. 2005. Using one-class and two-class SVMs for multiclass image annotation. IEEE Trans. Knowl. Data Eng. 17, 10 (2005), 1333–1346.Google ScholarDigital Library
- Sally A Goldman and Yan Zhou. 2000. Enhancing supervised learning with unlabeled data. In Proceedings of the 17th International Conference on Machine Learning (ICML’00). ACM, 327–334.Google Scholar
- Yunchao Gong, Yangqing Jia, Thomas Leung, Alexander Toshev, and Sergey Ioffe. 2013. Deep convolutional ranking for multilabel image annotation. arXiv:1312.4894. Retrieved from https://arxiv.org/abs/1312.4894.Google Scholar
- Matthieu Guillaumin, Thomas Mensink, Jakob Verbeek, and Cordelia Schmid. 2009. Tagprop: Discriminative metric learning in nearest neighbor models for image auto-annotation. In Proceedings of the 12th IEEE International Conference on Computer Vision (ICCV’09). IEEE, 309–316.Google ScholarCross Ref
- Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2015. Spatial pyramid pooling in deep convolutional networks for visual recognition. IEEE Trans. Pattern Anal. Mach. Intell. 37, 9 (2015), 1904–1916.Google ScholarDigital Library
- Thomas Hofmann. 2001. Unsupervised learning by probabilistic latent semantic analysis. Mach. Learn. 42, 1–2 (2001), 177–196.Google ScholarCross Ref
- Jiwoon Jeon, Victor Lavrenko, and Raghavan Manmatha. 2003. Automatic image annotation and retrieval using cross-media relevance models. In Proceedings of the 26th Annual International ACM SIGIR Conference on Research and Development in Informaion Retrieval (SIGIR’03). ACM, 119–126.Google ScholarDigital Library
- Liping Jing, Chenyang Shen, Liu Yang, Jian Yu, and Michael K. Ng. 2017. Multi-label classification by semi-supervised singular value decomposition. IEEE Trans. Image Process. 26, 10 (2017), 4612–4625.Google ScholarDigital Library
- Xiao-Yuan Jing, Fei Wu, Zhiqiang Li, Ruimin Hu, and David Zhang. 2016. Multi-label dictionary learning for image annotation. IEEE Trans. Image Process. 25, 6 (2016), 2712–2725.Google ScholarDigital Library
- Justin Johnson, Lamberto Ballan, and Li Fei-Fei. 2015. Love thy neighbors: Image annotation by exploiting image metadata. In Proceedings of the IEEE International Conference on Computer Vision (ICCV’15). IEEE, 4624–4632.Google ScholarDigital Library
- Xiao Ke, Mingke Zhou, Yuzhen Niu, and Wenzhong Guo. Nov. 2017. Data equilibrium based automatic image annotation by fusing deep model and semantic propagation. Pattern Recogn. 71 (Nov. 2017), 60–77.Google Scholar
- Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton. 2012. ImageNet classification with deep convolutional neural networks. In Proceedings of the Conference on Advances in Neural Information Processing Systems (NIPS’12). MIT Press, 1106–1114.Google ScholarDigital Library
- Victor Lavrenko, Raghavan Manmatha, and Jiwoon Jeon. 2004. A model for learning the semantics of pictures. In Proceedings of the Conference on Advances in Neural Information Processing Systems (NIPS’04). MIT Press, 553–560.Google Scholar
- Zhixin Li, Lingzhi Li, Kaobi Yan, and Canlong Zhang. 2017. Automatic image annotation using fuzzy association rules and decision tree. Multimedia Syst. 23, 6 (2017), 679–690.Google ScholarDigital Library
- Zhixin Li, Lan Lin, Canlong Zhang, Huifang Ma, and Weizhong Zhao. 2019. Collaborating CNN and SVM for automatic image annotation. In Proceedings of the 9th ACM International Conference on Multimedia Retrieval (ICMR’19). ACM, 63–67.Google ScholarDigital Library
- Zechao Li, Jing Liu, Changsheng Xu, and Hanqing Lu. 2013. MLRank: Multi-correlation learning to rank for image annotation. Pattern Recogn. 46, 10 (2013), 2700–2710.Google ScholarDigital Library
- Zhixin Li, Zhiping Shi, Xi Liu, and Zhongzhi Shi. 2011. Modeling continuous visual features for semantic image annotation and retrieval. Pattern Recogn. Lett. 32, 3 (2011), 516–523.Google ScholarDigital Library
- Zhixin Li, Zhongzhi Shi, Weizhong Zhao, Zhiqing Li, and Zhenjun Tang. 2013. Learning semantic concepts from image database with hybrid generative/discriminative approach. Eng. Appl. Artif. Intell. 26, 9 (2013), 2143–2152.Google ScholarDigital Library
- Ameesh Makadia, Vladimir Pavlovic, and Sanjiv Kumar. 2010. Baselines for image annotation. Int. J. Comput. Vis. 90, 1 (2010), 88–105.Google ScholarDigital Library
- Florent Monay and Daniel Gatica-Perez. 2007. Modeling semantic aspects for cross-media image indexing. IEEE Trans. Pattern Anal. Mach. Intell. 29, 10 (2007), 1802–1817.Google ScholarDigital Library
- Venkatesh N. Murthy, Subhransu Maji, and R. Manmatha. 2015. Automatic image annotation using deep learning representations. In Proceedings of the 5th ACM International Conference on Multimedia Retrieval (ICMR’15). ACM, 603–606.Google Scholar
- Kamal Nigam and Rayid Ghani. 2000. Analyzing the effectiveness and applicability of co-training. In Proceedings of the 9th ACM International Conference on Information and Knowledge Management (CIKM’00). ACM, 86–93.Google ScholarDigital Library
- Maxime Oquab, Leon Bottou, Ivan Laptev, and Josef Sivic. 2014. Learning and transferring mid-level image representations using convolutional neural networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR’14). 1717–1724.Google ScholarDigital Library
- Xiaojun Qi and Yutao Han. 2007. Incorporating multiple SVMs for automatic image annotation. Pattern Recogn. 40, 2 (2007), 728–741.Google ScholarDigital Library
- Jesse Read, Bernhard Pfahringer, Geoff Holmes, and Eibe Frank. 2011. Classifier chains for multi-label classification. Machine Learn. 85, 3 (2011), 254–269.Google ScholarDigital Library
- Karen Simonyan and Andrew Zisserman. 2014. Very deep convolutional networks for large-scale image recognition. arXiv:1409.1556. Retrieved from https://arxiv.org/abs/1409.1556.Google Scholar
- Chang Tang, Xinwang Liu, Pichao Wang, Changqing Zhang, Miaomiao Li, and Lizhe Wang. 2019. Adaptive hypergraph embedded semi-supervised multi-label image annotation. IEEE Trans. Multimedia 21, 11 (2019), 2837–2849.Google ScholarCross Ref
- Amara Tariq and Hassan Foroosh. Jan. 2018. Designing a symmetric classifier for image annotation using multi-layer sparse coding. Image Vis. Comput. 69 (Jan. 2018), 33–43.Google Scholar
- Tiberio Uricchio, Lamberto Ballan, Lorenzo Seidenari, and Alberto Del Bimbo. Nov. 2017. Automatic image annotation via label transfer in the semantic space. Pattern Recogn. 71 (Nov. 2017), 144–157.Google Scholar
- Yashaswi Verma. Sept. 2019. Diverse image annotation with missing labels. Pattern Recogn. 93 (Sept. 2019), 470–484.Google Scholar
- Yashaswi Verma and C. V. Jawahar. 2017. Image annotation by propagating labels from semantic neighbourhoods. Int. J. Comput. Vis. 121, 1 (2017), 126–148.Google ScholarDigital Library
- Luis Von Ahn and Laura Dabbish. 2004. Labeling images with a computer game. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM, 319–326.Google ScholarDigital Library
- Fei Wu, Zhuhao Wang, Zhongfei Zhang, Yi Yang, Jiebo Luo, Wenwu Zhu, and Yueting Zhuang. 2015. Weakly semi-supervised deep learning for multi-label image annotation. IEEE Trans. Big Data 1, 3 (2015), 109–122.Google ScholarCross Ref
- Yuying Xing, Guoxian Yu, Carlotta Domeniconi, Jun Wang, and Zili Zhang. 2018. Multi-label co-training. In Proceedings of the 27th International Joint Conference on Artificial Intelligence (IJCAI’18). 2882–2888.Google ScholarDigital Library
- Wang Zhan and Minling Zhang. 2017. Inductive semi-supervised multi-label learning with co-training. In Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD’17). ACM, 1305–1314.Google ScholarDigital Library
- Dengsheng Zhang, Md Monirul Islam, and Guojun Lu. 2012. A review on automatic image annotation techniques. Pattern Recogn. 45, 1 (2012), 346–362.Google ScholarDigital Library
- Shaoting Zhang, Junzhou Huang, Yuchi Huang, Yang Yu, Hongsheng Li, and Dimitris N. Metaxas. 2010. Automatic image annotation using group sparsity. In Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR’10). IEEE, 3312–3319.Google Scholar
- Mingbo Zhao, Tommy W. S. Chow, Zhao Zhang, and Bing Li. Mar. 2015. Automatic image annotation via compact graph based semi-supervised learning. Knowl.-Based Syst. 76 (Mar. 2015), 148–165.Google Scholar
- Tao Zhou, Zhixin Li, Canlong Zhang, and Huifang Ma. 2020. Classify multi-label images via improved CNN model with adversarial network. Multimedia Tools Appl. 79, 9–10 (2020), 6871–6890.Google ScholarCross Ref
- Xiaojin Zhu. 2007. Semi-supervised Learning Literature Survey. Technical Report. University of Wisconsin—Madison.Google Scholar
Index Terms
- A Semi-supervised Learning Approach Based on Adaptive Weighted Fusion for Automatic Image Annotation
Recommendations
Collaborating CNN and SVM for Automatic Image Annotation
ICMR '19: Proceedings of the 2019 on International Conference on Multimedia RetrievalTo learn a well-performed image annotation model, a large number of labeled samples are usually required. In this paper, we propose a novel semi-supervised approach based on adaptive weighted fusion for automatic image annotation, which can utilize the ...
Inductive Semi-supervised Multi-Label Learning with Co-Training
KDD '17: Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data MiningIn multi-label learning, each training example is associated with multiple class labels and the task is to learn a mapping from the feature space to the power set of label space. It is generally demanding and time-consuming to obtain labels for training ...
A Novel Region-based Image Annotation Using Multi-instance Learning
WKDD '09: Proceedings of the 2009 Second International Workshop on Knowledge Discovery and Data MiningIn this paper, we formulate image annotation as a semi-supervised learning problem under multi-instance learning framework. A novel graph based semi-supervised learning approach to image annotation using multiple instances is presented, which extends ...
Comments