Skip to main content
SpringerLink
Log in

Pattern graph-based image retrieval system combining semantic and visual features

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

In the literature, several image retrieval approaches that allow mapping between low-level features and high-level semantics have been proposed. Among these one can cite object recognition, ontologies, and relevance feedback. However, their main limitations concern their high dependence on reliable external resources (existing ontologies, learning sets, etc.) and lack of capacity to combine semantic and visual information and provide relevant results. This paper proposes a system aiming to improve image retrieval results. The proposed system is based on a pattern graph combining semantic and visual features. The idea is (1) to automatically build a modular ontology based on a learning step from textual corpus and terminological resource, (2) to organize visual features in a graph-based model where the combined module and graph represent a unique component called “pattern,” and (3) to build a pattern graph. To this end our system has been implemented. The obtained experimental results show that the pattern graph that we propose enables an improvement of retrieval task.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9

Similar content being viewed by others

Notes

  1. https://opennlp.apache.org/

  2. http://www.cis.uni-muenchen.de/∼schmid/tools/TreeTagger/

  3. http://nlp.stanford.edu/

  4. http://www.opencalais.com/

  5. http://babelnet.org/

  6. http://nlp.stanford.edu/software/relationExtractor.html

  7. http://knowitall.github.io/ollie/

  8. https://code.google.com/p/lire/

  9. http://imageclef.org/SIAPRdata

References

  1. Allani O, Mellouli N, Baazaoui H, Akdag H, Ben Ghezala H (2015) A relevant visual feature selection approach for image retrieval The international conference on computer vision theory and applications. VISAPP, p 2015

    Google Scholar 

  2. Arni T, Clough P, Sanderson M, Grubinger M (2008) Overview of the ImageCLEFphoto 2008 photographic retrieval task Workshop of the cross-language evaluation forum for European languages. Springer, Berlin Heidelberg, pp 500–511

    Google Scholar 

  3. Baker LD, Mccallum AK (1998) Distributional clustering of words for text classification Proceedings of the 21st annual international ACM SIGIR conference on research and development in information retrieval. ACM, pp 96–103

    Google Scholar 

  4. Bannour H, Hudelot C (2014) Building And using fuzzy multimedia ontologies for semantic image annotation. Multimedia Tools Appl 72(3):2107–2141

    Article  Google Scholar 

  5. Barbu T (2013) A Novel Image Similarity Metric using SIFT-based Characteristics Mathematical models in engineering and computer science: proceedings of the 2nd international conference on computers, digital communications and computing, ICDCC’13, pp 15–18

    Google Scholar 

  6. Besbes G, Baazaoui-Zghal H (2014) Modular ontologies and CBR-based hybrid system for web information retrieval. Multimedia Tools Appl 1–25

  7. Cheng Z, Shen J, Xie L, Zhu L (2017) Unsupervised visual hashing with semantic assistant for content-based image retrieval. IEEE Trans Knowl Data Eng 29:472–486

    Article  Google Scholar 

  8. Choi D, Kim J, Kim H, et al. (2012) A method for enhancing image retrieval based on annotation using modified wup similarity in wordnet Proceedings of the 11th WSEAS international conference on artificial intelligence, knowledge engineering and data bases AIKED, vol 2012, pp 83–87

  9. Crucianu M, Ferecatu M, Boujemaa N (2004) Relevance feedback for image retrieval: a short survey. Report of the DELOS2 European Network of Excellence (FP6)

  10. Cui J, Liu Y, Xu Y, Zhao H, Zha H (2013) Tracking generic human motion via fusion of low-and high-dimensional approaches. IEEE Trans Syst Man Cybern Syst 43(4):996–1002

    Article  Google Scholar 

  11. Dao MS, Boato G, DeNatale FG (2012) Discovering inherent event taxonomies from social media collections Proceedings of the 2nd ACM international conference on multimedia retrieval. ACM, p 48

  12. Demir B, Bruzzone L (2015) A novel active learning method in relevance feedback for content-based remote sensing image retrieval. IEEE Trans Geosci Remote Sens 53 (5):2323–2334

    Article  Google Scholar 

  13. D’aquin M, Sabou M, et Motta E (2006) Modularization: a key for the dynamic selection of relevant knowledge components

  14. Escalante HJ, Hernández CA, Gonzalez JA (2010) The segmented and annotated IAPR TC-12 benchmark. Comput Vis Image Underst 114(4):419–428

    Article  Google Scholar 

  15. Feng D., Siu W. C., Zhang H. J. (eds.) (2013) Multimedia information retrieval and management: technological fundamentals and applications. Springer Science and Business Media

  16. Fundel K, Küffner R., Zimmer R (2007) RelEx—Relation extraction using dependency parse trees. Bioinformatics 23(3):365–371

    Article  Google Scholar 

  17. Hammiche S, Benbernou S, et Vakali A (2005) A logic based approach for the multimedia data representation and retrieval 7th IEEE International symposium on multimedia. IEEE, p 8

    Google Scholar 

  18. Hernández-Gracidas CA, Sucar LE, Montes-Y-Gómez M (2013) Improving image retrieval by using spatial relations. Multimedia Tools Appl 62(2):479–505

    Article  Google Scholar 

  19. Khalid YIA, Noah SA (2011) A Framework for integrating DBpedia in a multi-modality ontology news image retrieval system 2011 International conference on semantic technology and information retrieval (STAIR). IEEE, pp 144–149

  20. Liqiang N, Meng W, Zheng-Jun Z, Tat-Seng C (2012) Oracle in image search: a content-based approach to performance prediction. ACM Trans Inf Syst

  21. Lin D (1998) Automatic retrieval and clustering of similar words Proceedings of the 17th international conference on computational linguistics-Volume 2. Association for Computational Linguistics, vol 1998, pp 768–774

  22. Liu AA, Nie WZ, Gao Y, Su YT (2016) Multi-modal clique-graph matching for view-based 3D model retrieval. IEEE Trans Image Process 25(5):2103–2116

    Article  MathSciNet  Google Scholar 

  23. Liu L, Cheng L, Liu Y, Jia Y, Rosenblum DS (2016) Recognizing complex activities by a probabilistic interval-based model AAAI, pp 1266–1272

    Google Scholar 

  24. Liu Y, Cui J, Zhao H, Zha H (2012) Fusion of low-and high-dimensional approaches by trackers sampling for generic human motion tracking 2012 21st International conference on pattern recognition (ICPR). IEEE, pp 898–901

  25. Liu Y, Liang Y, Liu S, Rosenblum DS, Zheng Y (2016) Predicting urban water quality with ubiquitous data. arXiv:1610.09462

  26. Liu Y, Nie L, Han L, Zhang L, Rosenblum DS (2016) Action2activity: recognizing complex activities from sensor data. arXiv:1611.01872

  27. Liu Y, Nie L, Liu L, Rosenblum DS (2016) From action to activity: sensor-based activity recognition. Neurocomputing 181:108–115

    Article  Google Scholar 

  28. Liu Y, Zhang D, Lu G, Ma W-Y (2007) A survey of content-based image retrieval with high-level semantics. Pattern Recogn 40(1):262–282

    Article  MATH  Google Scholar 

  29. Liu Y, Zhang L, Nie L, Yan Y, Rosenblum DS (2016) Fortune teller: predicting your career path AAAI, pp 201–207

    Google Scholar 

  30. Liu Y, Zhang X, Cui J, Wu C, Aghajan H, Zha H (2010) Visual analysis of child-adult interactive behaviors in video sequences 2010 16th International conference on virtual systems and multimedia (VSMM). IEEE, pp 26–33

  31. Liu Y, Zheng Y, Liang Y, Liu S, Rosenblum DS (2016) Urban water quality prediction based on multi-task multi-view learning Proceedings of the international joint conference on artificial intelligence

    Google Scholar 

  32. Lu Y, Wei Y, Liu L, Zhong J, Sun L, Liu Y (2016) Towards unsupervised physical activity recognition using smartphone accelerometers. Multimedia Tools Appl

  33. Meghini C, Sebastiani F, Straccia U (2001) A model of multimedia information retrieval. J ACM (JACM) 48(5):909–970

    Article  MathSciNet  MATH  Google Scholar 

  34. Mezaris V, Kompatsiaris I, Strintzis MG (2004) Region-based image retrieval using an object ontology and relevance feedback. Eurasip J Appl Signal Process, 2004 2004:886–901

    Article  Google Scholar 

  35. Minu RI, Thyagharajan KK (2012) Multimodal ontology search for semantic image retrieval. Submitted to International Journal of Computer System Science and Engineering for February, no 2012

  36. Moehrmann J, Heidemann G (2013) Semi-automatic image annotation International conference on computer analysis of images and patterns. Springer, Berlin Heidelberg, pp 266–273

    Chapter  Google Scholar 

  37. Moro A, Raganato A, Navigli R (2014) 2014 Entity linking meets word sense disambiguation: a unified approach. Transactions of the Association for Computational Linguistics (TACL) 2:231–244

    Google Scholar 

  38. Mustapha NB, Aufaure MA, Zghal HB, Ghezala HB (2012) Modular ontological warehouse for adaptative information search Model and Data Engineering. Springer, Berlin Heidelberg, pp 79–90

    Chapter  Google Scholar 

  39. Navigli R, Ponzetto SP (2012) BabelNet: the automatic construction, evaluation and application of a wide-coverage multilingual semantic network. Artif Intell 193:217–250

    Article  MathSciNet  MATH  Google Scholar 

  40. Nie L, Wang M, Gao Y, Zha ZJ, Chua TS (2013) Beyond text QA: multimedia answer generation by harvesting web information. IEEE Trans Multimedia 15(2):426–441

    Article  Google Scholar 

  41. Nie W, Liu A, Su Y (2016) Cross-domain semantic transfer from large-scale social media. Multimedia Systems 22(1):75–85

    Article  Google Scholar 

  42. Nie W, Liu A, Zhu X, Su Y (2016) Quality models for venue recommendation in location-based social network. Multimedia Tools and Appl 75(20):12521–12534

    Article  Google Scholar 

  43. Pham T-T, Maillot NE, Lim J-H, Chevallet J-P (2007) Latent semantic fusion model for image retrieval and annotation Proceedings of the 16th ACM conference on conference on information and knowledge management, CIKM’07. ACM, New York, pp 439–444

    Chapter  Google Scholar 

  44. Poslad S, Kesorn K (2014) A multi-modal incompleteness ontology model (MMIO) to enhance information fusion for image retrieval. Information Fusion 20:225–241

    Article  Google Scholar 

  45. Raoui Y, Bouyakhf EH, Devy M, Regragui F (2011) Global and local image descriptors for content based image retrieval and object recognition. Appl Math Sci 5(42):2109–2136

    MATH  Google Scholar 

  46. Rokach L, Oded M (2005) Clustering methods. Data Mining and Knowledge Discovery Handbook. Springer, USA, pp 321–352

    Book  MATH  Google Scholar 

  47. Salton G, McGill MJ (1986) Introduction to modern information retrieval. McGraw-Hill, Inc., New York

    MATH  Google Scholar 

  48. Smeulders AWM, Worring M, Santini S, Gupta A, Jain R (2000) Content-based image retrieval at the end of the early years. IEEE Trans Pattern Anal Mach Intell 22(12):1349–1380

    Article  Google Scholar 

  49. Smucker MD, Allan J, Carterette B (2007) A comparison of statistical significance tests for information retrieval evaluation. CIKM 2007:623–632

    Google Scholar 

  50. Straccia U, Visco G (2007) DLMedia: an ontology mediated multimedia information retrieval system Description logics

    Google Scholar 

  51. Yang Y, Nie F, Xu D, Luo J, Zhuang Y, Pan Y (2012) A multimedia retrieval framework based on semi-supervised ranking and relevance feedback. IEEE Trans Pattern Anal Mach Intell 34(4):723–742

    Article  Google Scholar 

  52. Zhang H, Shang X, Luan H, Wang M, Chua TS (2016) Learning from collective intelligence: feature learning using social images and tags. ACM Trans Multimed Comput Commun Appl

Download references

Author information

Authors and Affiliations

  1. RIADI Laboratory, ENSI, University of Manouba, Manouba, Tunisia

    Olfa Allani & Hajer Baazaoui Zghal

  2. LIASD Laboratory, Paris 8 University, Saint-Denis, France

    Olfa Allani, Nedra Mellouli & Herman Akdag

  3. IUT of Montreuil, Paris 8 University, Saint-Denis, France

    Nedra Mellouli

Authors
  1. Olfa Allani
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Hajer Baazaoui Zghal
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Nedra Mellouli
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Herman Akdag
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Hajer Baazaoui Zghal.

Appendix: Detailed and summarized algorithms

Appendix: Detailed and summarized algorithms

figure h
figure i

Concerning the optimization steps of the algorithm, some elements which allowed the improvement of the computing time and memory use are provided below:

  • During the intra-pattern search (figure1 step VI, algorithm 3 line 13), if the first obtained similarity measures on the randomly selected regions are judged weak (similarity measure less than 0.4), the pattern is left and we enchain with computing similarity to the next pattern. In fact this operation decreases the required search time. On one single pattern, the similarity decision time is reduced from 35214 s to 21539 s (+63.5%)

  • During the inter-pattern and intra-pattern search, treatments are parallelized using a multi-thread approach. A preliminary study has been conducted concerning the thread number which we have varied between 2 and 6 threads. We noticed that the most relevant value is obtained for 4 threads. These threads allow a reduction of the retrieval time from 2602514 s (for a sequential search) to 1865325 s (for a parallelized search).

  • Some elements are present in an important number of images such as sky, clouds, sun, etc. These elements are stored in a list and considered as visual stop words that do not bring interesting information to the retrieval process. That is why they are not considered during the retrieval process. Indeed, the application of this step reduces the number of compared regions to 2/3 in 65% of the query images and thus gradually improve the retrieval process cost.

The retrieval results are usually stored in a cash for future use. However if they are not used after a certain time, they are automatically deleted in order to constantly keep sufficient memory space. The memory space provided for this storage step is equal to 10 Mo and its use could reduce the retrieval time if the query image was treated before.

These steps contribute to the improvement of the computing time and the memory use.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Allani, O., Zghal, H.B., Mellouli, N. et al. Pattern graph-based image retrieval system combining semantic and visual features. Multimed Tools Appl 76, 20287–20316 (2017). https://doi.org/10.1007/s11042-017-4716-8

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-017-4716-8

Keywords

Search

Navigation