Skip to main content
SpringerLink
Log in

Multi-level diversification approach of semantic-based image retrieval results

  • Regular Paper
  • Published:
Progress in Artificial Intelligence Aims and scope Submit manuscript

Abstract

With the increasing popularity of social photograph-sharing Web sites, a huge mass of digital images, associated with a set of tags voluntarily introduced by amateur photographers, is daily hosted and consequently, the Tag-based social Image Retrieval technique has been widely adopted. However, tag-based queries are often too ambiguous and abstract to be considered as an efficient solution for the retrieval of the most relevant images that meet the users’ needs. As an alternative, the Semantic-based social Image Retrieval technique has emerged for the purpose of retrieving the relevant images covering as much possible the topics that a given ambiguous query (q) may have. Actually, the diversification strategies are a great challenge for researchers. In this context, we jointly investigate two processes at the ambiguous query preprocessing and postprocessing levels. On the one hand, we propose a Tag-based Query Semantic Reformulation process, which aims at reformulating the tag-based users’ queries, according to multiple semantic facets of the different images’ views, by using a set of predefined ontological semantic rules. On the other hand, we propose a Multi-level Image Diversification process that can first perform a two-level-based image clustering offline, and second, filter and re-rank the image cluster retrieval results according to their pertinence versus the reformulated query online. The experimental results and statistical analysis performed on a collection of 25.000 socio-tagged images shared on Flickr demonstrate the effectiveness of the proposed technique, which is compared with the research technique based on one-level-based image clustering, tag-based image research technique and recent CBIR techniques.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16
Fig. 17
Fig. 18
Fig. 19
Fig. 20
Fig. 21
Fig. 22
Fig. 23
Fig. 24
Fig. 25
Fig. 26
Fig. 27
Fig. 28
Fig. 29
Fig. 30
Fig. 31
Fig. 32
Fig. 33
Fig. 34

Similar content being viewed by others

Notes

  1. https://www.google.com/.

  2. https://fr.yahoo.com/.

  3. https://www.flickr.com/.

  4. https://www.instagram.com/.

  5. http://www.culturecrossmedia.com/reseaux-sociauxevolution/.

  6. https://www.instagram.com/press/.

  7. http://www.kullin.net/2012/01/instagram-now-growing-faster-than-flickr/.

  8. http://www.owltouring.org/ Last visited 08/08/2016.

References

  1. Chathurani, N.W.U.D., Geva, S., Chandran, V., Cynthujah, V.: An effective content based image retrieval system based on global representation and multi-level searching. In: Proceedings of the 10th International Conference on Industrial and Information Systems (ICIIS), pp. 158–163 (2015)

  2. Wang, X.Y., Zhang, B.B., Yang, H.Y.: Content-based image retrieval by integrating color and texture features. Multimed. Tools Appl. 68(3), 545–569 (2014)

    Google Scholar 

  3. Atlam, H.F., Attiya, G., El-Fishawy, N.: Integration of color and texture features in CBIR system. Int. J. Comput. Appl. 164(3), 23–29 (2017)

    Google Scholar 

  4. Liu, G.H., Yang, J.Y.: Content-based image retrieval using color difference histogram. Pattern Recogn. 46(1), 188–198 (2013)

    Google Scholar 

  5. Chang, B.M., Tsai, H.H., Chou, W.L.: Using visual features to design a content-based image retrieval method optimized by particle swarm optimization algorithm. Eng. Appl. Artif. Intell. 26(10), 2372–2382 (2013)

    Google Scholar 

  6. Liu, G.H., Li, Z.Y., Zhang, L., Xu, Y.: Image retrieval based on micro-structure descriptor. Pattern Recogn. 44(9), 2123–2133 (2011)

    Google Scholar 

  7. ElAlami, M.E.: A new matching strategy for content based image retrieval system. Appl. Soft Comput. 14, 407–418 (2014)

    Google Scholar 

  8. Jiji, G.W., DuraiRaj, P.J.: Content-based image retrieval techniques for the analysis of dermatological lesions using particle swarm optimization technique. Appl. Soft Comput. 30, 650–662 (2015)

    Google Scholar 

  9. Wang, X.Y., Yu, Y.J., Yang, H.Y.: An effective image retrieval scheme using color, texture and shape features. Comput. Stand. Interfaces 33(1), 59–68 (2011)

    Google Scholar 

  10. Pass, G., Zabih, R.: Comparing images using joint histograms. Multimed. Syst. 7(3), 234–240 (1999)

    Google Scholar 

  11. Wang, X.Y., Li, Y.W., Yang, H.Y., Chen, J.W.: An image retrieval scheme with relevance feedback using feature reconstruction and SVM reclassification. Neurocomputing 127, 214–230 (2014)

    Google Scholar 

  12. Yousuf, M., Mehmood, Z., Habib, H.A., Mahmood, T., Saba, T., Rehman, A., Rashid, M.: A novel technique based on visual words fusion analysis of sparse features for effective content-based image retrieval. Math. Probl. Eng. 2018, 1–13 (2018)

    Google Scholar 

  13. Mehmood, Z., Abbas, F., Mahmood, T., Javid, M.A., Rehman, A., Nawaz, T.: Content-based image retrieval based on visual words fusion versus features fusion of local and global features. Arab. J. Sci. Eng. 43(12), 7265–7284 (2018)

    Google Scholar 

  14. Jabeen, S., Mehmood, Z., Mahmood, T., Saba, T., Rehman, A., Mahmood, M.T.: An effective content-based image retrieval technique for image visuals representation based on the bag-of-visual-words model. PLoS ONE 13(4), e0194526 (2018)

    Google Scholar 

  15. Sharif, U., Mehmood, Z., Mahmood, T., Javid, M.A., Rehman, A., Saba, T.: Scene analysis and search using local features and support vector machine for effective content-based image retrieval. Artif. Intell. Rev. 2018, 1–25 (2018)

    Google Scholar 

  16. Sarwar, A., Mehmood, Z., Saba, T., Qazi, K.A., Adnan, A., Jamal, H.: A novel method for content-based image retrieval to improve the effectiveness of the bag-of-words model using a support vector machine. J. Inf. Sci. 45(1), 117–135 (2019)

    Google Scholar 

  17. Mehmood, Z., Gul, N., Altaf, M., Mahmood, T., Saba, T., Rehman, A., Mahmood, M.T.: Scene search based on the adapted triangular regions and soft clustering to improve the effectiveness of the visual-bag-of-words model. EURASIP J. Image Video Process. 2018(1), 48 (2018)

    Google Scholar 

  18. Mehmood, Z., Anwar, S.M., Altaf, M.: A novel image retrieval based on rectangular spatial histograms of visual words. Kuwait J. Sci. 45(1), 54–69 (2018)

    MathSciNet  Google Scholar 

  19. Smeulders, A.W., Worring, M., Santini, S., Gupta, A., Jain, R.: Content-based image retrieval at the end of the early years. IEEE Trans. Pattern Anal. Mach. Intell. 22(12), 1349–1380 (2000)

    Google Scholar 

  20. Adebayo, S., McLeod, K., Tudose, I., Osumi-Sutherland, D., Burdett, T., Baldock, R., Parkinson, H.: PhenoImageShare: an image annotation and query infrastructure. J. Biomed. Semant. 7(1), 35–44 (2016)

    Google Scholar 

  21. Kurtz, C., Rubin, D.L.: Utilisation de relations ontologiques pour la comparaison d’images décrites par des annotations sémantiques. In: Proceedings of the (EGC), pp. 609–614 (2014)

  22. Lingutla, N.T., Preece, J., Todorovic, S., Cooper, L., Moore, L., Jaiswal, P.: AISO: annotation of image segments with ontologies. J. Biomed. Semant. 5(1), 50–54 (2014)

    Google Scholar 

  23. Hollink, L., Nguyen, G., Schreiber, G., Wielemaker, J., Wielinga, B., Worring, M.: Adding spatial semantics to image annotations. In: Proceedings of the 4th International Workshop on Knowledge Markup and Semantic Annotation (ISWC), pp. 31–40 (2004)

  24. Zhang, C., Chai, J., Jin, R.: User term feedback in interactive text-based image retrieval. In: Proceedings of the 28th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 51–58 (2005)

  25. Weinberger, K.Q., Slaney, M., Van Zwol, R.: Resolving tag ambiguity. In: Proceedings of the 16th ACM International Conference on Multimedia (MM), pp. 111–120 (2008)

  26. Ksibi, A., Ammar, A.B., Amar, C.B.: Adaptive diversification for tag-based social image retrieval. Int. J. Multimed. Inf. Retr. 3(1), 29–39 (2014)

    Google Scholar 

  27. Van Leuken, R.H., Garcia, L., Olivares, X., Van Zwol, R.: Visual diversification of image search results. In: Proceedings of the 9th International Conference on World Wide Web (WWW), pp. 341–350 (2009)

  28. Radlinski, F., Bennett, P.N., Carterette, B., Joachims, T.: Redundancy, diversity and interdependent document relevance. SIGIR Forum. 43(2), 46–52 (2009)

    Google Scholar 

  29. Hoque, E., Hoeber, O., Gong, M.: Evaluating the tradeoffs between diversity and precision for web image search using concept-based query expansion. In: Proceedings of the 11th International Conference on Web Intelligence and Intelligent Agent Technology (WI-IAT), vol. 3, pp. 130–133 (2011)

  30. Carpineto, C., Romano, G.: A survey of automatic query expansion in information retrieval. ACM Comput. Surv. 44(1), 1–50 (2012)

    MATH  Google Scholar 

  31. Lioma, C., Moens, M.F., Azzopardi, L.: Collaborative annotation for pseudo relevance feedback. In: Proceedings of Exploiting Semantic Annotation in Information Retrieval (ESAIR), pp. 25–35 (2008)

  32. Naouar, F., Hlaoua, L., Omri, M.N.: Relevance feedback in collaborative information retrieval based on validated annotation. In: Proceedings of the 1st International Conference on Reasoning and Optimization in Information Systems, vol. 13 (2013)

  33. Guo, Q., Liu, W., Lin, Y., Lin, H.: Query expansion based on user quality in folksonomy. In: Proceedings of Asia Information Retrieval Symposium (AIRS), pp. 396–405 (2012)

    Google Scholar 

  34. Kitanovski, I., Strezoski, G., Dimitrovski, I., Madjarov, G., Loskovska, S.: Multimodal medical image retrieval system. Multimed. Tools Appl. 76(2), 2955–2978 (2017)

    Google Scholar 

  35. Garg, N., Gatica-Perez, D.: Tagging and retrieving images with co-occurrence models: from corel to flickr. In: Proceedings of the 1st ACM Workshop on Large-Scale Multimedia Retrieval and Mining, pp. 105–112 (2009)

  36. Jin, S., Lin, H., Su, S.: Query expansion based on folksonomy tag co-occurrence analysis. In: Proceedings of the International Conference on Granular Computing, pp. 300–305 (2009)

  37. Broder, A., Ciccolo, P., Gabrilovich, E., Pang, B.: Domain-specific query augmentation using folksonomy tags: the case of contextual advertising. In: Proceedings of the 1st Workshop on Information Retrieval for Advertising (2008)

  38. Ksibi, A., Ammar, A.B., Amar, C.B.: Enhanced context-based query-to-concept mapping in social image retrieval. In: Proceedings of the 11th International Workshop on Content-Based Multimedia Indexing (CBMI), pp. 85–89 (2013)

  39. Wang, J., Davison, B.D.: Explorations in tag suggestion and query expansion. In: Proceeding of the 2008 ACM Workshop on Search in Social Media (SSM), pp. 43–50 (2008)

  40. Biancalana, C., Micarelli, A.: Social tagging in query expansion: a new way for personalized web search. In: Proceedings of the 12th IEEE International Conference on Computational Science and Engineering (CSE), pp. 1060–1065 (2009)

  41. Abbasi, R.: Query expansion in folksonomies. In: International Conference on Semantic and Digital Media Technologies (SAMT), pp. 1–16 (2010)

    Google Scholar 

  42. He, Y., Li, Y., Lei, J., Leung, C.H.: A framework of query expansion for image retrieval based on knowledge base and concept similarity. Neurocomputing 204, 26–32 (2016)

    Google Scholar 

  43. Otegi, A., Arregi, X., Ansa, O., Agirre, E.: Using knowledge-based relatedness for information retrieval. Knowl. Inf. Syst. 44(3), 689–718 (2015)

    Google Scholar 

  44. Hoque, E., Hoeber, O., Gong, M.: Balancing the trade-offs between diversity and precision for web image search using concept-based query expansion. J. Emerg. Technol. Web Intell. 4(1), 26–34 (2012)

    Google Scholar 

  45. Leong, C.W., Hassan, S., Ruiz, M.E., Mihalcea, R.: Improving query expansion for image retrieval via saliency and picturability. In: International Conference of the Cross-Language Evaluation Forum for European Languages (CLEF), pp. 137–142 (2011)

    Google Scholar 

  46. Yuvarani, M., Iyengar, N.C.S., Kannan, A.: Improved concept-based query expansion using Wikipedia. Int. J. Commun. Netw. Distrib. Syst. 11(1), 26–41 (2013)

    Google Scholar 

  47. Ayadi, Y., Amous, I., Gargouri, F.: Toward an automatic annotation approach based on ontological enrichment for advanced research. Int. J. Eng. Technol. 13(2), 80–89 (2013)

    Google Scholar 

  48. Alromima, W., Moawad, I.F., Elgohary, R., Aref, M.: Ontology-based query expansion for Arabic text retrieval. Int. J. Adv. Comput. Sci. Appl. 7(8), 223–230 (2016)

    Google Scholar 

  49. Devi, M.U., Gandhi, G.M.: Wordnet and ontology based query expansion for semantic information retrieval in sports domain. J. Comput. Sci. 11(2), 361–371 (2014)

    Google Scholar 

  50. Charhad, M., Zrigui, M., Quénot, G.: Une approche conceptuelle pour la modélisation et la structuration sémantique des documents vidéos. In: Proceedings of the 3rd International Conference: Sciences of Electronic, Technologies of Information and Telecommunications (SETIT), pp. 1–7 (2005)

  51. Bouchakwa, M., Ayadi, Y., Amous, I.: Modeling the semantic content of the socio-tagged images based on the extended conceptual graphs formalism. In: Proceedings of the 14th International Conference on Advances in Mobile Computing and Multi Media (MOMM), pp. 35–39 (2016)

  52. De Andrade, D.O.S., Maia, L.F., De Figueirêdo, H.F., Viana, W., Trinta, F., De Souza Baptista, C.: Photo annotation: a survey. Multimed. Tools Appl. 77(1), 423–457 (2018)

    Google Scholar 

  53. Van Leuken, R.H., Garcia, L., Olivares, X., Van Zwol, R.: Visual diversification of image search results. In: Proceedings of the 18th International Conference on World Wide Web (WWW), pp. 341–350 (2009)

  54. Wang, M., Yang, K., Hua, X.S., Zhang, H.J.: Towards a relevant and diverse search of social images. IEEE Trans. Multimed. 12(8), 829–842 (2010)

    Google Scholar 

  55. Sun, F., Wang, M., Wang, D., Wang, X.: Optimizing social image search with multiple criteria: relevance, diversity, and typicality. Neurocomputing 95, 40–47 (2012)

    Google Scholar 

  56. Carbonell, J., Goldstein, J.: The use of MMR, diversity-based re-ranking for reordering documents and producing summaries. In: Proceedings of the 21st Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 335–336 (1998)

  57. Agrawal, R., Gollapudi, S., Halverson, A., Ieong, S.: Diversifying search results. In: Proceedings of the 2nd ACM International Conference on Web Search and Data Mining (WSDM), pp. 5–14 (2009)

  58. Santos, R.L., Macdonald, C., Ounis, I.: Exploiting query reformulations for web search result diversification. In: Proceedings of the 19th International Conference on World Wide Web (WWW), pp. 881–890 (2010)

  59. Santos, R.L., Macdonald, C., Ounis, I.: Selectively diversifying web search results. In: Proceedings of the 19th ACM International Conference on Information and Knowledge Management (CIKM), pp. 1179–1188 (2010)

  60. Song, K., Tian, Y., Gao, W., Huang, T.: Diversifying the image retrieval results. In: Proceedings of the 14th ACM International Conference on Multimedia (ICM), pp. 707–710 (2006)

  61. Qian, X., Lu, D., Wang, Y., Zhu, L., Tang, Y.Y., Wang, M.: Image re-ranking based on topic diversity. IEEE Trans. Image Process. 26(8), 3734–3747 (2017)

    MathSciNet  MATH  Google Scholar 

  62. Yan, Y., Liu, G., Wang, S., Zhang, J., Zheng, K.: Graph-based clustering and ranking for diversified image search. Multimed. Syst. 23(1), 41–52 (2017)

    Google Scholar 

  63. Lashari, S.A., Ibrahim, R.: A framework for medical images classification using soft set. Procedia Technol. 11(2013), 548–556 (2013)

    Google Scholar 

  64. Wen, W., Hao, Z., Cai, R., Shao, Z.: Gaussian process learning for image classification based on low-level features. In: Proceedings of the 8th International Conference on Natural Computation (CNC), pp. 237–241 (2012)

  65. Cai, D., He, X., Li, Z., Ma, W.Y., Wen, J.R.: Hierarchical clustering of WWW image search results using visual, textual and link information. In: Proceedings of the 12th Annual ACM International Conference on Multimedia (MULTIMEDIA), pp. 952–959 (2004)

  66. Sabarad, A.K., Kankudti, M.H., Meena, S.M., Husain, M.: Color and texture feature extraction using Apache Hadoop framework. In: Proceedings of the 2015 International Conference on Computing Communication Control and Automation (ICCUBEA), pp. 585–588 (2015)

  67. Zhu, H., Shen, Z., Shang, L., Zhang, X.: Parallel image texture feature extraction under Hadoop cloud platform. In: Proceedings of the International Conference on Intelligent Computing (ICIC), pp. 459–465 (2014)

    Google Scholar 

  68. Qayyum, A., Anwar, S.M., Awais, M., Majid, M.: Medical image retrieval using deep convolutional neural network. Neurocomputing 266, 8–20 (2017)

    Google Scholar 

  69. Li, Y.D., Hao, Z.B., Lei, H.: Survey of convolutional neural network. Int. J. Comput. Appl. 36(9), 2508–2515 (2016)

    Google Scholar 

  70. Lyndon, D., Kumar, A., Kim, J., Leong, P.H. W., Feng, D.: Convolutional neural networks for medical clustering. In: CLEF (Working Notes) (2015)

  71. Mehmood, Z., Rashid, M., Rehman, A., Saba, T., Dawood, H., Dawood, H.: Effect of complementary visual words versus complementary features on clustering for effective content-based image search. J. Intell. Fuzzy Syst. 2019, 1–14 (2019)

    Google Scholar 

  72. Cha, G.H.: Kernel principal component analysis for content based image retrieval. In: Proceedings of the Pacific-Asia Conference on Knowledge Discovery and Data Mining (PAKDD), pp. 844–849 (2005)

    Google Scholar 

  73. Wang, S., Jing, F., He, J., Du, Q., Zhang, L.: Igroup: presenting web image search results in semantic clusters. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI), pp. 587–596 (2007)

  74. Ramage, D., Heymann, P., Manning, C.D., Garcia-Molina, H.: Clustering the tagged web. In: Proceedings of the 2nd ACM International Conference on Web Search and Data Mining (WSDM), pp. 54–63 (2009)

  75. Yin, Z., Li, R., Mei, Q., Han, J.: Exploring social tagging graph for web object classification. In: Proceedings of the 15th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD), pp. 957–966 (2009)

  76. Wang, X.J., Zhang, L., Jing, F., Ma, W.-Y.: Annosearch: image auto-annotation by search. Proc. Comput. Vis. Pattern Recogn. 2, 1483–1490 (2006)

    Google Scholar 

  77. Bouchakwa, M., Ayadi, Y., Amous, I.: Semantic pattern-based automatic annotation process of images shared on social networks. In: Proceedings of the 30th IBIMA Conference (IBIMA), pp. 19–38 (2017)

  78. Sowa, J.F.: Conceptual structures: information processing in mind and machine. Addison-Wesley publishing Company, Reading (1984)

    MATH  Google Scholar 

  79. Cohen, J.: A coefficient of agreement for nominal scales. Educ. Psychol. Meas. 20(1), 37–46 (1960)

    Google Scholar 

  80. MacQueen, J.: Some methods for classification and analysis of multivariate observations. In: Proceedings of the 5th Berkeley Symposium On Mathematical Statistics and Probabilities, vol. 1, pp. 281–296 (1967)

  81. Thorndike, R.L.: Who belongs in the family? Psychometrika 18(4), 267–276 (1953)

    Google Scholar 

  82. Rousseeuw, P.J., Kaufman, L.: Finding groups in data. Ser. Probab. Math. Stat. 34(1), 111–112 (1990)

    MATH  Google Scholar 

  83. Tibshirani, R., Walther, G., Hastie, T.: Estimating the number of clusters in a data set via the gap statistic. J R. Stat. Soc. 63(2), 411–423 (2001)

    MathSciNet  MATH  Google Scholar 

  84. Bhusare, B.B., Bansode, S.M.: Centroids initialization for K-means clustering using improved pillar algorithm. Int. J. Adv. Res. Comput. Eng. Technol. 3(4), 1317–1322 (2014)

    Google Scholar 

  85. Chris, B., Ellen, M.: Evaluating evaluation measure stability. In: Proceedings of the 28th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 33–40 (2000)

  86. Ellen M.V.: The philosophy of information retrieval evaluation. In: CLEFEL, pp. 355–370 (2001)

  87. Im, D.H., Park, G.D.: Linked tag: image annotation using semantic relationships between image tags. Int. J. Multimed. Tools Appl. 74(7), 2273–2287 (2015)

    Google Scholar 

  88. Ricardo, B., Berthier, R.: Modern Information Retrieval. ACM Press, NewYork (1999)

    Google Scholar 

  89. Mehmood, Z., Anwar, S.M., Ali, N., Habib, H.A., Rashid, M.: A novel image retrieval based on a combination of local and global histograms of visual words. Math. Probl. Eng. 2016, 12 (2016)

    Google Scholar 

  90. Tian, X., Jiao, L., Liu, X., Zhang, X.: Feature integration of EODH and Color-SIFT: application to image retrieval based on codebook. Signal Process. Image Commun. 29(4), 530–545 (2014)

    Google Scholar 

  91. Ali, N., Bajwa, K.B., Sablatnig, R., Mehmood, Z.: Image retrieval by addition of spatial information based on histograms of triangular regions. Comput. Electr. Eng. 54, 539–550 (2016)

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. MIRACL Laboratory, Technopole of Sfax, University of Sfax, P.O. Box 242, 3031, Sfax, Tunisia

    Mariam Bouchakwa, Yassine Ayadi & Ikram Amous

Authors
  1. Mariam Bouchakwa
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Yassine Ayadi
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Ikram Amous
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Mariam Bouchakwa.

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Bouchakwa, M., Ayadi, Y. & Amous, I. Multi-level diversification approach of semantic-based image retrieval results. Prog Artif Intell 9, 1–30 (2020). https://doi.org/10.1007/s13748-019-00195-x

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s13748-019-00195-x

Keywords

Search

Navigation