Abstract
In the literature, several image retrieval approaches that allow mapping between low-level features and high-level semantics have been proposed. Among these one can cite object recognition, ontologies, and relevance feedback. However, their main limitations concern their high dependence on reliable external resources (existing ontologies, learning sets, etc.) and lack of capacity to combine semantic and visual information and provide relevant results. This paper proposes a system aiming to improve image retrieval results. The proposed system is based on a pattern graph combining semantic and visual features. The idea is (1) to automatically build a modular ontology based on a learning step from textual corpus and terminological resource, (2) to organize visual features in a graph-based model where the combined module and graph represent a unique component called “pattern,” and (3) to build a pattern graph. To this end our system has been implemented. The obtained experimental results show that the pattern graph that we propose enables an improvement of retrieval task.









Similar content being viewed by others
Notes
References
Allani O, Mellouli N, Baazaoui H, Akdag H, Ben Ghezala H (2015) A relevant visual feature selection approach for image retrieval The international conference on computer vision theory and applications. VISAPP, p 2015
Arni T, Clough P, Sanderson M, Grubinger M (2008) Overview of the ImageCLEFphoto 2008 photographic retrieval task Workshop of the cross-language evaluation forum for European languages. Springer, Berlin Heidelberg, pp 500–511
Baker LD, Mccallum AK (1998) Distributional clustering of words for text classification Proceedings of the 21st annual international ACM SIGIR conference on research and development in information retrieval. ACM, pp 96–103
Bannour H, Hudelot C (2014) Building And using fuzzy multimedia ontologies for semantic image annotation. Multimedia Tools Appl 72(3):2107–2141
Barbu T (2013) A Novel Image Similarity Metric using SIFT-based Characteristics Mathematical models in engineering and computer science: proceedings of the 2nd international conference on computers, digital communications and computing, ICDCC’13, pp 15–18
Besbes G, Baazaoui-Zghal H (2014) Modular ontologies and CBR-based hybrid system for web information retrieval. Multimedia Tools Appl 1–25
Cheng Z, Shen J, Xie L, Zhu L (2017) Unsupervised visual hashing with semantic assistant for content-based image retrieval. IEEE Trans Knowl Data Eng 29:472–486
Choi D, Kim J, Kim H, et al. (2012) A method for enhancing image retrieval based on annotation using modified wup similarity in wordnet Proceedings of the 11th WSEAS international conference on artificial intelligence, knowledge engineering and data bases AIKED, vol 2012, pp 83–87
Crucianu M, Ferecatu M, Boujemaa N (2004) Relevance feedback for image retrieval: a short survey. Report of the DELOS2 European Network of Excellence (FP6)
Cui J, Liu Y, Xu Y, Zhao H, Zha H (2013) Tracking generic human motion via fusion of low-and high-dimensional approaches. IEEE Trans Syst Man Cybern Syst 43(4):996–1002
Dao MS, Boato G, DeNatale FG (2012) Discovering inherent event taxonomies from social media collections Proceedings of the 2nd ACM international conference on multimedia retrieval. ACM, p 48
Demir B, Bruzzone L (2015) A novel active learning method in relevance feedback for content-based remote sensing image retrieval. IEEE Trans Geosci Remote Sens 53 (5):2323–2334
D’aquin M, Sabou M, et Motta E (2006) Modularization: a key for the dynamic selection of relevant knowledge components
Escalante HJ, Hernández CA, Gonzalez JA (2010) The segmented and annotated IAPR TC-12 benchmark. Comput Vis Image Underst 114(4):419–428
Feng D., Siu W. C., Zhang H. J. (eds.) (2013) Multimedia information retrieval and management: technological fundamentals and applications. Springer Science and Business Media
Fundel K, Küffner R., Zimmer R (2007) RelEx—Relation extraction using dependency parse trees. Bioinformatics 23(3):365–371
Hammiche S, Benbernou S, et Vakali A (2005) A logic based approach for the multimedia data representation and retrieval 7th IEEE International symposium on multimedia. IEEE, p 8
Hernández-Gracidas CA, Sucar LE, Montes-Y-Gómez M (2013) Improving image retrieval by using spatial relations. Multimedia Tools Appl 62(2):479–505
Khalid YIA, Noah SA (2011) A Framework for integrating DBpedia in a multi-modality ontology news image retrieval system 2011 International conference on semantic technology and information retrieval (STAIR). IEEE, pp 144–149
Liqiang N, Meng W, Zheng-Jun Z, Tat-Seng C (2012) Oracle in image search: a content-based approach to performance prediction. ACM Trans Inf Syst
Lin D (1998) Automatic retrieval and clustering of similar words Proceedings of the 17th international conference on computational linguistics-Volume 2. Association for Computational Linguistics, vol 1998, pp 768–774
Liu AA, Nie WZ, Gao Y, Su YT (2016) Multi-modal clique-graph matching for view-based 3D model retrieval. IEEE Trans Image Process 25(5):2103–2116
Liu L, Cheng L, Liu Y, Jia Y, Rosenblum DS (2016) Recognizing complex activities by a probabilistic interval-based model AAAI, pp 1266–1272
Liu Y, Cui J, Zhao H, Zha H (2012) Fusion of low-and high-dimensional approaches by trackers sampling for generic human motion tracking 2012 21st International conference on pattern recognition (ICPR). IEEE, pp 898–901
Liu Y, Liang Y, Liu S, Rosenblum DS, Zheng Y (2016) Predicting urban water quality with ubiquitous data. arXiv:1610.09462
Liu Y, Nie L, Han L, Zhang L, Rosenblum DS (2016) Action2activity: recognizing complex activities from sensor data. arXiv:1611.01872
Liu Y, Nie L, Liu L, Rosenblum DS (2016) From action to activity: sensor-based activity recognition. Neurocomputing 181:108–115
Liu Y, Zhang D, Lu G, Ma W-Y (2007) A survey of content-based image retrieval with high-level semantics. Pattern Recogn 40(1):262–282
Liu Y, Zhang L, Nie L, Yan Y, Rosenblum DS (2016) Fortune teller: predicting your career path AAAI, pp 201–207
Liu Y, Zhang X, Cui J, Wu C, Aghajan H, Zha H (2010) Visual analysis of child-adult interactive behaviors in video sequences 2010 16th International conference on virtual systems and multimedia (VSMM). IEEE, pp 26–33
Liu Y, Zheng Y, Liang Y, Liu S, Rosenblum DS (2016) Urban water quality prediction based on multi-task multi-view learning Proceedings of the international joint conference on artificial intelligence
Lu Y, Wei Y, Liu L, Zhong J, Sun L, Liu Y (2016) Towards unsupervised physical activity recognition using smartphone accelerometers. Multimedia Tools Appl
Meghini C, Sebastiani F, Straccia U (2001) A model of multimedia information retrieval. J ACM (JACM) 48(5):909–970
Mezaris V, Kompatsiaris I, Strintzis MG (2004) Region-based image retrieval using an object ontology and relevance feedback. Eurasip J Appl Signal Process, 2004 2004:886–901
Minu RI, Thyagharajan KK (2012) Multimodal ontology search for semantic image retrieval. Submitted to International Journal of Computer System Science and Engineering for February, no 2012
Moehrmann J, Heidemann G (2013) Semi-automatic image annotation International conference on computer analysis of images and patterns. Springer, Berlin Heidelberg, pp 266–273
Moro A, Raganato A, Navigli R (2014) 2014 Entity linking meets word sense disambiguation: a unified approach. Transactions of the Association for Computational Linguistics (TACL) 2:231–244
Mustapha NB, Aufaure MA, Zghal HB, Ghezala HB (2012) Modular ontological warehouse for adaptative information search Model and Data Engineering. Springer, Berlin Heidelberg, pp 79–90
Navigli R, Ponzetto SP (2012) BabelNet: the automatic construction, evaluation and application of a wide-coverage multilingual semantic network. Artif Intell 193:217–250
Nie L, Wang M, Gao Y, Zha ZJ, Chua TS (2013) Beyond text QA: multimedia answer generation by harvesting web information. IEEE Trans Multimedia 15(2):426–441
Nie W, Liu A, Su Y (2016) Cross-domain semantic transfer from large-scale social media. Multimedia Systems 22(1):75–85
Nie W, Liu A, Zhu X, Su Y (2016) Quality models for venue recommendation in location-based social network. Multimedia Tools and Appl 75(20):12521–12534
Pham T-T, Maillot NE, Lim J-H, Chevallet J-P (2007) Latent semantic fusion model for image retrieval and annotation Proceedings of the 16th ACM conference on conference on information and knowledge management, CIKM’07. ACM, New York, pp 439–444
Poslad S, Kesorn K (2014) A multi-modal incompleteness ontology model (MMIO) to enhance information fusion for image retrieval. Information Fusion 20:225–241
Raoui Y, Bouyakhf EH, Devy M, Regragui F (2011) Global and local image descriptors for content based image retrieval and object recognition. Appl Math Sci 5(42):2109–2136
Rokach L, Oded M (2005) Clustering methods. Data Mining and Knowledge Discovery Handbook. Springer, USA, pp 321–352
Salton G, McGill MJ (1986) Introduction to modern information retrieval. McGraw-Hill, Inc., New York
Smeulders AWM, Worring M, Santini S, Gupta A, Jain R (2000) Content-based image retrieval at the end of the early years. IEEE Trans Pattern Anal Mach Intell 22(12):1349–1380
Smucker MD, Allan J, Carterette B (2007) A comparison of statistical significance tests for information retrieval evaluation. CIKM 2007:623–632
Straccia U, Visco G (2007) DLMedia: an ontology mediated multimedia information retrieval system Description logics
Yang Y, Nie F, Xu D, Luo J, Zhuang Y, Pan Y (2012) A multimedia retrieval framework based on semi-supervised ranking and relevance feedback. IEEE Trans Pattern Anal Mach Intell 34(4):723–742
Zhang H, Shang X, Luan H, Wang M, Chua TS (2016) Learning from collective intelligence: feature learning using social images and tags. ACM Trans Multimed Comput Commun Appl
Author information
Authors and Affiliations
Corresponding author
Appendix: Detailed and summarized algorithms
Appendix: Detailed and summarized algorithms


Concerning the optimization steps of the algorithm, some elements which allowed the improvement of the computing time and memory use are provided below:
-
During the intra-pattern search (figure1 step VI, algorithm 3 line 13), if the first obtained similarity measures on the randomly selected regions are judged weak (similarity measure less than 0.4), the pattern is left and we enchain with computing similarity to the next pattern. In fact this operation decreases the required search time. On one single pattern, the similarity decision time is reduced from 35214 s to 21539 s (+63.5%)
-
During the inter-pattern and intra-pattern search, treatments are parallelized using a multi-thread approach. A preliminary study has been conducted concerning the thread number which we have varied between 2 and 6 threads. We noticed that the most relevant value is obtained for 4 threads. These threads allow a reduction of the retrieval time from 2602514 s (for a sequential search) to 1865325 s (for a parallelized search).
-
Some elements are present in an important number of images such as sky, clouds, sun, etc. These elements are stored in a list and considered as visual stop words that do not bring interesting information to the retrieval process. That is why they are not considered during the retrieval process. Indeed, the application of this step reduces the number of compared regions to 2/3 in 65% of the query images and thus gradually improve the retrieval process cost.
The retrieval results are usually stored in a cash for future use. However if they are not used after a certain time, they are automatically deleted in order to constantly keep sufficient memory space. The memory space provided for this storage step is equal to 10 Mo and its use could reduce the retrieval time if the query image was treated before.
These steps contribute to the improvement of the computing time and the memory use.
Rights and permissions
About this article
Cite this article
Allani, O., Zghal, H.B., Mellouli, N. et al. Pattern graph-based image retrieval system combining semantic and visual features. Multimed Tools Appl 76, 20287–20316 (2017). https://doi.org/10.1007/s11042-017-4716-8
Received:
Revised:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11042-017-4716-8