Skip to main content
SpringerLink
Log in

OntoAnnClass: ontology-based image annotation driven by classification using HMAX features

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

Several approaches have been proposed in the area of Automatic Image Annotation (AIA) in order to exploit the relationships between words that are extracted from image categories, and to automatically generate annotation words for a given image. Other methods exploit ontologies, where the annotation keywords were derived from ontology to improve image annotation. In this paper, we propose an ontology-based image annotation driven by classification using HMAX features. The idea is (1) to train visual-feature-classifiers and to build an ontology that can finely represent the semantic information associated with training images, and (2) to combine classifier outputs and ontology for image annotation. To annotate images, we define a membership value of words in images. In particular, we propose to evaluate the membership value based on the confidence value of classifiers and the semantic similarity between words. The membership value depends on the word relationships found in the ontology that serve to select annotation words. The obtained experimental results show that the exploitation of both classifier outputs and ontology by evaluating our proposed membership value enables an improvement of image annotation.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7

Similar content being viewed by others

Notes

  1. https://wordnet.princeton.edu/

  2. http://owlapi.sourceforge.net/

  3. http://www.image-net.org/

  4. https://storage.googleapis.com/openimages/web/download.html

  5. http://owlapi.sourceforge.net/

References

  1. Bannour H, Hudelot C (2012) Hierarchical image annotation using semantic hierarchies. In: 21st ACM international conference on information and knowledge management, CIKM’12, Maui, HI, USA, October 29 - November 02, 2012. ACM, pp 2431–2434

  2. Breiman L (2001) Random forests. Machine Learn 45(1):5–32

    Article  Google Scholar 

  3. Cortes C, Vapnik V (1995) Support-vector networks. Machine Learn 20(3):273–297

    MATH  Google Scholar 

  4. Dutt A, Pellerin D, Quénot G. (2017) Improving image classification using coarse and fine labels. In: Proceedings of the 2017 ACM on international conference on multimedia retrieval, ICMR 2017, Bucharest, Romania, June 6-9, 2017. ACM, pp 438–442

  5. Filali J, Zghal HB, Martinet J (2017) Visually supporting image annotation based on visual features and ontologies. In: 21st international conference information visualisation, IV 2017, London, United Kingdom, July 11–14, 2017. IEEE Computer Society, pp 182–187

  6. Filali J, Zghal HB, Martinet J (2019) Ontology and HMAX features-based image classification using merged classifiers. In: Proceedings of the 14th international joint conference on computer vision, imaging and computer graphics theory and applications, VISIGRAPP 2019. VISAPP, Prague, Czech Republic, February 25-27, 2019, vol 5. SciTePress, pp 124–134

  7. Filali J, Zghal HB, Martinet J (2020) Ontology-based image classification and annotation. International Journal of Pattern Recognition and Artificial Intelligence 34(11)

  8. Gao H, Dou L, Chen W, Sun J (2013) Image classification with bag-of-words model based on improved SIFT algorithm. In: 9th Asian Control Conference, ASCC 2013, Istanbul, Turkey, June 23-26, 2013. IEEE, pp 1–6

  9. Han Y, Li G (2015) Describing images with hierarchical concepts and object class localization. In: Proceedings of the 5th ACM on international conference on multimedia retrieval, Shanghai, China, June 23-26, 2015. ACM, pp 251–258

  10. He X, Peng Y (2017) Weakly supervised learning of part selection model with spatial constraints for fine-grained image classification. In: Proceedings of the thirty-first AAAI conference on artificial intelligence, February 4-9, 2017, San Francisco, California, USA. AAAI Press, pp 4075–4081

  11. Hu X, Zhang J, Li J, Zhang B (2014) Sparsity-regularized hmax for visual recognition. PloS one 9(1)

  12. Lau KH, Tay YH, Lo FL (2015) A HMAX with LLC for visual recognition. arXiv:1502.02772

  13. Lei J, Guo Z, Wang Y (2017) Weakly supervised image classification with coarse and fine labels. In: 14th conference on computer and robot vision, CRV 2017, Edmonton, AB, Canada, May 16-19, 2017. IEEE Computer Society, pp 240–247

  14. Li Y, Liu J, Wang Y, Liu B, Fu J, Gao Y, Wu H, Song H, Ying P, Lu H (2015a) Hybrid learning framework for large-scale web image annotation and localization. In: CLEF (working notes)

  15. Li Y, Wu W, Zhang B, Li F (2015b) Enhanced HMAX model with feedforward feature learning for multiclass categorization. Front Comput Neurosci 9:123

    Google Scholar 

  16. Ma Y, Liu Y, Xie Q, Li L (2019) Cnn-feature based automatic image annotation method. Multimedia Tools Appl 78(3):3767–3780

    Article  Google Scholar 

  17. Niu Y, Lu Z, Wen J, Xiang T, Chang S (2019) Multi-modal multi-scale deep learning for large-scale image annotation. IEEE Trans Image Process 28(4):1720–1731

    Article  MathSciNet  Google Scholar 

  18. Olszewska JI (2013) Semantic, automatic image annotation based on multi-layered active contours and decision trees. Int J Adv Comput Sci Appl 4(8):201–208

    Google Scholar 

  19. Priyadarshini A et al (2015) A map reduce based support vector machine for big data classification. Int J Database Theory Appl 8(5):77–98

    Article  Google Scholar 

  20. Reshma IA, Ullah MZ, Aono M (2014) Ontology based classification for multi-label image annotation. In: Advanced Informatics: Concept, Theory and Application (ICAICTA), 2014 international conference of. IEEE, pp 226–231

  21. Riesenhuber M, Poggio T (1999) Hierarchical models of object recognition in cortex. Nature Neurosc 2(11):1019

    Article  Google Scholar 

  22. Rinaldi AM (2014) Using multimedia ontologies for automatic image annotation and classification. In: 2014 IEEE International Congress on Big Data, Anchorage, AK, USA, June 27 - July 2, 2014. IEEE Computer Society, pp 242–249

  23. Ristin M, Gall J, Guillaumin M, Gool LV (2015) From categories to subcategories: Large-scale image classification with partial class label refinement. In: IEEE conference on computer vision and pattern recognition, CVPR 2015, Boston, MA, USA, June 7-12, 2015. IEEE Computer Society, pp 231–239

  24. Serre T, Wolf L, Bileschi S, Riesenhuber M, Poggio T (2007) Robust object recognition with cortex-like mechanisms. IEEE Trans Pattern Anal Mach Intell 3:411–426

    Article  Google Scholar 

  25. Sun F, Xu Y, Zhou J (2016) Active learning svm with regularization path for image classification. Multimed Tools Appl 75(3):1427–1442

    Article  Google Scholar 

  26. Sun C, Zhu S, Shi Z (2015) Image annotation via deep neural network. In: 14th IAPR international conference on machine vision applications, MVA 2015, Miraikan, Tokyo, Japan, 18-22 May, 2015. IEEE, pp 518–521

  27. Theriault C, Thome N, Cord M (2011) HMAX-S: deep scale representation for biologically inspired image categorization. In: 18th IEEE International Conference on Image Processing, ICIP 2011, Brussels, Belgium, September 11-14, 2011. IEEE, pp 1261–1264

  28. Theriault C, Thome N, Cord M (2013) Extended coding and pooling in the HMAX model. IEEE Trans Image Process 22(2):764–777

    Article  MathSciNet  Google Scholar 

  29. Tian D (2015) Support vector machine for automatic image annotation. Int J Hybrid Inf Technol 8(11):435–446

    Google Scholar 

  30. Tsai D, Jing Y, Liu Y, Rowley HA, Ioffe S, Rehg JM (2011) Large-scale image annotation using visual synset. In: IEEE International Conference on Computer Vision, ICCV 2011, Barcelona, Spain, November 6-13, 2011. IEEE Computer Society, pp 611–618

  31. Wang C, Huang K (2015) How to use bag-of-words model better for image classification. Image Vis Comput 38:65–74

    Article  Google Scholar 

  32. Wei Z, Luo X, Zhou F (2013) Ontology based automatic image annotation using multi-class SVM. In: Proceedings of the seventh international conference on image and graphics, ICIG 2013, Qingdao, China, July 26-28, 2013. IEEE Computer Society, pp 434–438

  33. Weston J, Bengio S, Usunier N (2010) Large scale image annotation: learning to rank with joint word-image embeddings. Mach Learn 81(1):21–35

    Article  MathSciNet  Google Scholar 

  34. Wu Z, Palmer M (1994) Verbs semantics and lexical selection. In: Proceedings of the 32nd annual meeting on association for computational linguistics, pp 133–138

  35. Zarka M, Ammar AB, Alimi AM (1391) Regimvid at imageclef 2015 scalable concept image annotation task: Ontology based hierarchical image annotation

  36. Zhang H, Lu Y, Kang T, Lim M (2016) B-HMAX: a fast binary biologically inspired model for object recognition. Neurocomputing 218:242–250

    Article  Google Scholar 

  37. Zou F, Liu Y, Wang H, Song J, Shao J, Zhou K, Zheng S (2016) Multi-view multi-label learning for image annotation. Multimed Tools Appl 75(20):12627–12644

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. ENSI, RIADI Laboratory, University of Manouba, Manouba, Tunisia

    Jalila Filali & Hajer Baazaoui Zghal

  2. ETIS UMR 8051, CY University, ENSEA, CNRS, F-59000, Cergy, France

    Hajer Baazaoui Zghal

  3. Polytech Nice Sophia Campus SophiaTech, Université Côte d’Azur/I3S/CNRS, Sophia-Antipolis, France

    Jean Martinet

Authors
  1. Jalila Filali
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Hajer Baazaoui Zghal
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Jean Martinet
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Jalila Filali.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Filali, J., Zghal, H.B. & Martinet, J. OntoAnnClass: ontology-based image annotation driven by classification using HMAX features. Multimed Tools Appl 80, 6823–6851 (2021). https://doi.org/10.1007/s11042-020-09864-9

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-020-09864-9

Keywords

Search

Navigation