Abstract
Several approaches have been proposed in the area of Automatic Image Annotation (AIA) in order to exploit the relationships between words that are extracted from image categories, and to automatically generate annotation words for a given image. Other methods exploit ontologies, where the annotation keywords were derived from ontology to improve image annotation. In this paper, we propose an ontology-based image annotation driven by classification using HMAX features. The idea is (1) to train visual-feature-classifiers and to build an ontology that can finely represent the semantic information associated with training images, and (2) to combine classifier outputs and ontology for image annotation. To annotate images, we define a membership value of words in images. In particular, we propose to evaluate the membership value based on the confidence value of classifiers and the semantic similarity between words. The membership value depends on the word relationships found in the ontology that serve to select annotation words. The obtained experimental results show that the exploitation of both classifier outputs and ontology by evaluating our proposed membership value enables an improvement of image annotation.
Similar content being viewed by others
References
Bannour H, Hudelot C (2012) Hierarchical image annotation using semantic hierarchies. In: 21st ACM international conference on information and knowledge management, CIKM’12, Maui, HI, USA, October 29 - November 02, 2012. ACM, pp 2431–2434
Breiman L (2001) Random forests. Machine Learn 45(1):5–32
Cortes C, Vapnik V (1995) Support-vector networks. Machine Learn 20(3):273–297
Dutt A, Pellerin D, Quénot G. (2017) Improving image classification using coarse and fine labels. In: Proceedings of the 2017 ACM on international conference on multimedia retrieval, ICMR 2017, Bucharest, Romania, June 6-9, 2017. ACM, pp 438–442
Filali J, Zghal HB, Martinet J (2017) Visually supporting image annotation based on visual features and ontologies. In: 21st international conference information visualisation, IV 2017, London, United Kingdom, July 11–14, 2017. IEEE Computer Society, pp 182–187
Filali J, Zghal HB, Martinet J (2019) Ontology and HMAX features-based image classification using merged classifiers. In: Proceedings of the 14th international joint conference on computer vision, imaging and computer graphics theory and applications, VISIGRAPP 2019. VISAPP, Prague, Czech Republic, February 25-27, 2019, vol 5. SciTePress, pp 124–134
Filali J, Zghal HB, Martinet J (2020) Ontology-based image classification and annotation. International Journal of Pattern Recognition and Artificial Intelligence 34(11)
Gao H, Dou L, Chen W, Sun J (2013) Image classification with bag-of-words model based on improved SIFT algorithm. In: 9th Asian Control Conference, ASCC 2013, Istanbul, Turkey, June 23-26, 2013. IEEE, pp 1–6
Han Y, Li G (2015) Describing images with hierarchical concepts and object class localization. In: Proceedings of the 5th ACM on international conference on multimedia retrieval, Shanghai, China, June 23-26, 2015. ACM, pp 251–258
He X, Peng Y (2017) Weakly supervised learning of part selection model with spatial constraints for fine-grained image classification. In: Proceedings of the thirty-first AAAI conference on artificial intelligence, February 4-9, 2017, San Francisco, California, USA. AAAI Press, pp 4075–4081
Hu X, Zhang J, Li J, Zhang B (2014) Sparsity-regularized hmax for visual recognition. PloS one 9(1)
Lau KH, Tay YH, Lo FL (2015) A HMAX with LLC for visual recognition. arXiv:1502.02772
Lei J, Guo Z, Wang Y (2017) Weakly supervised image classification with coarse and fine labels. In: 14th conference on computer and robot vision, CRV 2017, Edmonton, AB, Canada, May 16-19, 2017. IEEE Computer Society, pp 240–247
Li Y, Liu J, Wang Y, Liu B, Fu J, Gao Y, Wu H, Song H, Ying P, Lu H (2015a) Hybrid learning framework for large-scale web image annotation and localization. In: CLEF (working notes)
Li Y, Wu W, Zhang B, Li F (2015b) Enhanced HMAX model with feedforward feature learning for multiclass categorization. Front Comput Neurosci 9:123
Ma Y, Liu Y, Xie Q, Li L (2019) Cnn-feature based automatic image annotation method. Multimedia Tools Appl 78(3):3767–3780
Niu Y, Lu Z, Wen J, Xiang T, Chang S (2019) Multi-modal multi-scale deep learning for large-scale image annotation. IEEE Trans Image Process 28(4):1720–1731
Olszewska JI (2013) Semantic, automatic image annotation based on multi-layered active contours and decision trees. Int J Adv Comput Sci Appl 4(8):201–208
Priyadarshini A et al (2015) A map reduce based support vector machine for big data classification. Int J Database Theory Appl 8(5):77–98
Reshma IA, Ullah MZ, Aono M (2014) Ontology based classification for multi-label image annotation. In: Advanced Informatics: Concept, Theory and Application (ICAICTA), 2014 international conference of. IEEE, pp 226–231
Riesenhuber M, Poggio T (1999) Hierarchical models of object recognition in cortex. Nature Neurosc 2(11):1019
Rinaldi AM (2014) Using multimedia ontologies for automatic image annotation and classification. In: 2014 IEEE International Congress on Big Data, Anchorage, AK, USA, June 27 - July 2, 2014. IEEE Computer Society, pp 242–249
Ristin M, Gall J, Guillaumin M, Gool LV (2015) From categories to subcategories: Large-scale image classification with partial class label refinement. In: IEEE conference on computer vision and pattern recognition, CVPR 2015, Boston, MA, USA, June 7-12, 2015. IEEE Computer Society, pp 231–239
Serre T, Wolf L, Bileschi S, Riesenhuber M, Poggio T (2007) Robust object recognition with cortex-like mechanisms. IEEE Trans Pattern Anal Mach Intell 3:411–426
Sun F, Xu Y, Zhou J (2016) Active learning svm with regularization path for image classification. Multimed Tools Appl 75(3):1427–1442
Sun C, Zhu S, Shi Z (2015) Image annotation via deep neural network. In: 14th IAPR international conference on machine vision applications, MVA 2015, Miraikan, Tokyo, Japan, 18-22 May, 2015. IEEE, pp 518–521
Theriault C, Thome N, Cord M (2011) HMAX-S: deep scale representation for biologically inspired image categorization. In: 18th IEEE International Conference on Image Processing, ICIP 2011, Brussels, Belgium, September 11-14, 2011. IEEE, pp 1261–1264
Theriault C, Thome N, Cord M (2013) Extended coding and pooling in the HMAX model. IEEE Trans Image Process 22(2):764–777
Tian D (2015) Support vector machine for automatic image annotation. Int J Hybrid Inf Technol 8(11):435–446
Tsai D, Jing Y, Liu Y, Rowley HA, Ioffe S, Rehg JM (2011) Large-scale image annotation using visual synset. In: IEEE International Conference on Computer Vision, ICCV 2011, Barcelona, Spain, November 6-13, 2011. IEEE Computer Society, pp 611–618
Wang C, Huang K (2015) How to use bag-of-words model better for image classification. Image Vis Comput 38:65–74
Wei Z, Luo X, Zhou F (2013) Ontology based automatic image annotation using multi-class SVM. In: Proceedings of the seventh international conference on image and graphics, ICIG 2013, Qingdao, China, July 26-28, 2013. IEEE Computer Society, pp 434–438
Weston J, Bengio S, Usunier N (2010) Large scale image annotation: learning to rank with joint word-image embeddings. Mach Learn 81(1):21–35
Wu Z, Palmer M (1994) Verbs semantics and lexical selection. In: Proceedings of the 32nd annual meeting on association for computational linguistics, pp 133–138
Zarka M, Ammar AB, Alimi AM (1391) Regimvid at imageclef 2015 scalable concept image annotation task: Ontology based hierarchical image annotation
Zhang H, Lu Y, Kang T, Lim M (2016) B-HMAX: a fast binary biologically inspired model for object recognition. Neurocomputing 218:242–250
Zou F, Liu Y, Wang H, Song J, Shao J, Zhou K, Zheng S (2016) Multi-view multi-label learning for image annotation. Multimed Tools Appl 75(20):12627–12644
Author information
Authors and Affiliations
Corresponding author
Additional information
Publisher’s note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
About this article
Cite this article
Filali, J., Zghal, H.B. & Martinet, J. OntoAnnClass: ontology-based image annotation driven by classification using HMAX features. Multimed Tools Appl 80, 6823–6851 (2021). https://doi.org/10.1007/s11042-020-09864-9
Received:
Revised:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11042-020-09864-9