Skip to main content
SpringerLink
Log in

Pseudo-relevance feedback diversification of social image retrieval results

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

In this paper we introduce a novel pseudo-relevance feedback (RF) perspective to social image search results diversification. Traditional RF techniques introduce the user in the processing loop by harvesting feedback about the relevance of the query results. This information is used for recomputing a better representation of the needed data. The novelty of our work is in exploiting the automatic generation of user feedback in a completely unsupervised diversification scenario, where positive and negative examples are used to generate better representations of visual classes in the data. First, user feedback is simulated automatically by selecting positive and negative examples from the initial query results. Then, an unsupervised hierarchical clustering is used to re-group images according to their content. Diversification is finally achieved with a re-ranking approach of the previously achieved clusters. Experimental validation on real-world data from Flickr shows the benefits of this approach achieving very promising results.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8

Similar content being viewed by others

Notes

  1. https://www.flickr.com/

  2. https://www.google.com/imghp

  3. An implementation of the used blur indicators is available at http://www.mathworks.com/ matlabcentral/fileexchange/27314-focus-measure/content/fmeasure/fmeasure.m

References

  1. Bosch A, Zisserman A, Munoz X (2007) Image classifcation using random forests and ferns. In: Proceedings of IEEE International Conference on Computer Vision, Rio de Janeiro, Brazil, 14-21. ISSN 1550-5499, pp 1–8

  2. Boteanu B, Mironică I, Ionescu B (2015) Hierarchical clustering pseudo-relevance feedback for social image search result diversification. In: Proceedings of 13th International Workshop on Content-Based Multimedia Indexing CBMI 2015, Prague, Czech Republic, 10-12, pp 1–6

  3. Cao G, Nie JY, Gao J, Robertson S (2008) Selecting Good Expansion Terms for Pseudo- Relevance Feedback. In: ACM SIGIR Conference on Research and Development in Information Retrieval, Singapore, 20-24, pp 243–250

  4. Cao L, Ji R, Liu W, Gao Y, Duan LY, Men C (2012) Weakly supervised topic grouping of youtube search results. In: Proceedings of the 19th IEEE International Conference on Image Processing (ICIP), Coronado Springs - Disney World Orlando, FL, USA, pp 2885–2888

  5. Carbonell J, Yang RFY, Brown RD, Geng Y, Lee D (1997) Translingual information retrieval: a comparative evaluation. In: Proceedings of 15th International Joint Conference on Artificial Intelligence (IJCAI), Aichi, Japan, 23-29, pp 708–715

  6. Cai D, He X, Li Z, Ma W-Y, Wen JR (2004) Hierarchical clustering of WWW image search results using visual, textual and link information. In: Proceedings of the 12th Annual ACM International Conference on Multimedia, NY, USA, 10-16, pp 952–959

  7. Castellanos A, Benavent X, Serrano AG, De Ves E, Cigarrn J UNED-UV @ retrieving diverse social images task. In: Proceedings of the MediaEval 2015 Workshop, Wurzen, Germany, 14-15, 2015, CEUR-WS.org, ISSN 1613-0073, 1436, http://ceur-ws.org/Vol-1436/

  8. Calumby RT, do IBA, Araujo C, Santana VP, Munoz JAV, Penatti OAB, Li LT, Almeida J, Chiachia G, Gonalves MA, Da S. Torres R Recod @ MediaEval 2015: diverse social images retrieval. In: Proceedings of the MediaEval 2015 Workshop, Wurzen, Germany, 14-15, 2015, CEUR-WS.org, ISSN 1613-0073, 1436, http://ceur-ws.org/Vol-1436/

  9. Castellanos A, GarcEDa-serrano A, Recuero JMC UNED @ retrieving diverse social images task. In: Proceedings of the MediaEval 2014 Workshop, Barcelona, Spain, 16-17, 2014, CEUR-WS.org, ISSN 1613-0073, 1263, http://ceur-ws.org/Vol-1263/

  10. Calumby RT, Santana VP, Cordeiro FS, Penatti OAB, Li LT, Chiachia G, Da S. Torres R Recod @ MediaEval 2014: diverse social images retrieval. In: Proceedings of the MediaEval 2014 Workshop, Barcelona, Spain, 16-17, 2014, CEUR-WS.org, ISSN 1613-0073, 1263, http://ceur-ws.org/Vol-1263/

  11. Datta R, Joshi D, Li J, Wang JZ (2008) Image Retrieval: ideas, Influences, and Trends of the New Age. ACM Comput Surv 40(2):1–60

    Article  Google Scholar 

  12. Dang V, Croft WB (2012) Diversity by proportiona lity: an election-based approach to search result diversification. In: Proceedings of the 35th International ACM SIGIR Conference on Research and Development in Information Retrieval, Oregon, USA, 12-16, pp 65–74

  13. Dang-Nguyen D-T, Piras L, Giacinto G, Boato G, De natale F (2015) A hybrid approach for Retrieving Diverse Social Images of Landmarks. In: Proceedings of IEEE International Conference on Multimedia and Expo (ICME), Torino, Italy, pp 1–6

  14. Denis F (1998) PAC learning from positive statistical queries. In: Proceedings of 9th International Conference on Algorithmic Learning Theory, Lecture Notes in Artificial Intelligence, 1501. Springer - Verlag, pp 112–126

  15. De Weijer V, Schmid C, Verbeek J, Larlus D (2009) Learning color names for real-world applications. IEEE Trans Image Process 18(7):1512–1523

    Article  MathSciNet  Google Scholar 

  16. Garcia Seco de Herrera A, Kalpathy-Cramer J, Demner Fushman D, Antani S, Müller H (2013) Overview of the ImageCLEF 2013 medical tasks. In: Working Notes of Cross Language Evaluation Forum (CLEF), Spain, pp 23–26

  17. Giacinto G (2207) A nearest-neighbor approach to relevance feedback in content-based image retrieval. In: Proceedings of the 6th ACM Conf. on Image and Video Retrieval, Amsterdam, Netherlands, 09-11, pp 456–463

  18. Ionescu B, Popescu A, Radu A-L, Müller H (2014) Result diversification in social image retrieval: a benchmarking framework

  19. Ionescu B, Popescu A, Lupu M, Ginscă AL, Boteanu B, Müller H (2015) Div150Cred: a social image retrieval result diversification with user tagging credibility dataset. In: ACM Multimedia Systems (MMSys), Oregon, USA, 18-20, pp 207–212

  20. Ionescu B, Radu A-L, Menéndez M, Müller H, Popescu A, Loni B (2014) Div400: a social image retrieval result diversification dataset. In: ACM Multimedia Systems (MMSys), Singapore, 19-20, pp 29–34

  21. Ionescu B, Ginscă AL, Boteanu B, Popescu A, Lupu M, Müller H Retrieving diverse social images at MediaEval 2015: challenge, dataset and evaluation. In: Proceedings of the MediaEval 2015 Workshop, Wurzen, Germany, 14-15, 2015, CEUR-WS.org, ISSN 1613-0073, 1436, http://ceur-ws.org/Vol-1436/

  22. Ionescu B, Popescu A, Lupu M, Ginscă AL, Müller H Retrieving diverse social images at MediaEval 2014: challenge, dataset and evaluation. In: Proceedings of the MediaEval 2014Workshop, Barcelona, Spain, 16-17, 2014, CEUR-WS.org, ISSN 1613-0073, 1263, http://ceur-ws.org/Vol-1263/

  23. Jones S, Shao L (2013) Content-based retrieval of human actions from realistic video databases. Inf Sci 236:56–65

    Article  Google Scholar 

  24. Jiang L, Mitamura T, Yu SI, Hauptmann AG (2014) Zero-example Event Search Using Multimodal Pseudo Relevance Feedback. In: Proceedings of International Conference on Multimedia Retrieval, Glasgow, U.K., 01-04, pp 297–304

  25. Jain AK, Murty MN, Flynn PJ (1999) Data clustering: a review. ACM Comput Surv (CSUR) 31(3):264–323

    Article  Google Scholar 

  26. Ksibi A, Ammar AB, Amar CB (2014) Adaptive diversification for tag-based social image retrieval. Int J Multimed Inf Retr 3(1):29–39

    Article  Google Scholar 

  27. King B (1967) Step-wise clustering procedures. J Am Stat Assoc 62(317):86–101

    Article  Google Scholar 

  28. Liang S, Sun Z (2008) Sketch retrieval and relevance feedback with biased SVM classification. Pattern Recogn Lett 29(12):1733–1741

    Article  Google Scholar 

  29. Liu B, Lee WS, Yu PS, Li X (2002) Partially supervised classification of text documents. In: Proceedings of 19th International Conference on Machine Learning, Sydney, Australia, 08-12, pp 387?-394

  30. Ludwig O, Delgado D, Goncalves V, Nunes U (2009) Trainable Classifier-Fusion Schemes: An Application To Pedestrian Detection. In: Proceedings of the 12th Int. IEEE Conf. on Intelligent Transportation Systems, Missouri, USA, 04-07, pp 1–6

  31. Manjunath BS, Ohm JR, Vasudevan VV, Yamada A (2001) Color and texture descriptors. IEEE Trans Circ Syst Video Technol 11(6):703–715

    Article  Google Scholar 

  32. Over P, Awad G, Michel M, Fiscus J, Sanders G, Kraaij W, Smeaton AF, Queenot G (2013) TRECVID 2013 — An Overview of the Goals, Tasks, Data, Evaluation Mechanisms and Metrics. TRECVID Workshop, NIST, USA. http://www-nlpir.nist.gov/projects/tvpubs/tv13.papers/tv13overview.pdf

    Google Scholar 

  33. Ojala T, Pietikinen M, Harwood D (1994) Performance evaluation of texture measures with classification based on kullback discrimination of distributions. In: Proceedings of the 12th IAPR International Conference on Pattern Recognition, 1, pp 582–585

  34. Priyatharshini R, Chitrakala S (2013) Association based image retrieval: a survey. In: Mobile Communication and Power Engineering, Springer Communications, Computer and Information Science, 296, pp 17–26

  35. Paramita ML, Sanderson M, Clough P (2009) Diversity in photo retrieval: overview of the ImageCLEF photo task 2009. In: Multilingual Information Access Evaluation II. Multimedia Experiments, pp 45?-59

  36. Pedronette DCG, Da S. Torres R, Calumby RT (2014) Using contextual spaces for image re-ranking and rank aggregation. Multimed Tools Appl 69(3):689–716

    Article  Google Scholar 

  37. Popescu A CEA LIST’s participation at MediaEval 2013 placing task. In: Proceedings of the MediaEval 2014 Workshop, Barcelona, Spain, 16-17, 2014, CEUR-WS.org, ISSN 1613-0073, 1263, http://ceur-ws.org/Vol-1263/

  38. Rudinac S, Hanjalic A, Larson MA (2013) Generating visual summaries of geographic areas using Community-Contributed images. IEEE Trans Multimed 15 (4):921–932

    Article  Google Scholar 

  39. Rocchio J (1971) Relevance feedback in information retrieval. In: The Smart Retrieval System Experiments in Automatic Document Processing. Prentice Hall, Englewood Cliffs, NJ, USA, pp 313– 323

  40. Rui Y, Huang T, Chang S-F (1999) Image retrieval: current techniques, promising directions and open issues. Vis Commun Image Represent 10(1):39–62

    Article  Google Scholar 

  41. Ravindranath SS, Gygli M, Gool LV ETH-CVL @ MediaEval 2015: learning objective functions for improved image retrieval. In: Proceedings of the MediaEval 2015 Workshop, Wurzen, Germany, 14-15, 2015, CEUR-WS.org, ISSN 1613-0073, 1436, http://ceur-ws.org/Vol-1436/

  42. Smeulders AWM, Worring M, Santini S, Gupta A, Jain R (2000) Content-based image retrieval at the end of the early years. IEEE Trans Pattern Anal Mach Intell 22(12):1349–1380

    Article  Google Scholar 

  43. Sarac MI, Duygulu P (2014) Bilkent-RETINA at retrieving diverse social images task of mediaEval 2014. In: Proceedings of the MediaEval 2014 Workshop. CEUR-WS.org, ISSN 1613-0073, 1263, http://ceur-ws.org/Vol-1263/, Barcelona, Spain

  44. Spyromitros-Xioufis E, Papadopoulos S, Ginscă AL, Popescu A, Kompatsiaris I, Vlahavas I (2015) Improving diversity in image search via supervised relevance scoring. In: Proceedings of ACM International Conference on Multimedia Retrieval, Shanghai, China, pp 323–330

  45. Soleymani M, Larson M (2010) Crowd-sourcing for affective annotation of video: development of a viewer-reported boredom corpus. SIGIR Workshop on Crowd-sourcing for Search Evaluation, Geneva, Switzerland, p 23

  46. Stricker M, Orengo M (1995) Similarity of color image. In: IS&T/SPIE’s Symposium on Electronic Imaging: Science & Technology, pp 381–392

  47. Sun A, Bhowmick SS (2009) Image Tag Clarity: in Search of Visual-Representative Tags for Social Images. In: Proceedings of the 1st SIGMM Workshop on Social Media, ACM, Beijing, China, 19-24, pp 19–26

  48. Sneath PHA, Sokal RR (1973) Numerical taxonomy. Freeman, London, UK

    MATH  Google Scholar 

  49. Spampinato C, Palazzo S PeRCeiVe Lab@UNICT at MediaEval 2014 diverse images: random forests for diversity-based clustering. In: Proceedings of the MediaEval 2014 Workshop, Barcelona, Spain, 16-17, 2014, CEUR-WS.org, ISSN 1613-0073, 1263, http://ceur-ws.org/Vol-1263/

  50. Taneva B, Kacimi M, Weikum G (2010) Gathering and Ranking Photos of Named Entities with High Precision, High Recall, and Diversity. In: Proceedings of the 3rd ACM International Conference on Web Search and Data Mining, New York, USA, 03-06, pp 431–440

  51. Tang X (1998) Texture information in Run-Length matrices. IEEE Trans Image Process 7(11):1602–1609

    Article  Google Scholar 

  52. Van Leuken RH, Garcia L, Olivares X, Van Zwol R (2009) Visual Diversification of Image Search Results. In: Proceedings of the 18th International Conference on World Wide Web, Madrid, Spain, 20-24, pp 341–350

  53. Vee E, Srivastava U, Shanmugasundaram J, Bhat P, Yahia SA (2008) Efficient computation conference=of Diverse Query Results, Proceedings of 24th IEEE International Conference on Data Engineering, Cancun, Mexico, 07-12, pp 228–236

  54. Vieira MR, Razente HL, Barioni MCN, Hadjieleftheriou M, Srivastava D, Traina Jr C, Tsotras VJ (2011) On query result diversification. In: Proceedings of IEEE International Conference on Data Engineering, Hannover, Germany, 11-16, pp 1163–1174

  55. Viola P, Jones MJ (2004) Robust Real-Time face detection. Int J Comput Vis 57(2):137–154

    Article  Google Scholar 

  56. Van Brummelen GR (2013) Heavenly Mathematics: The forgotten art of spherical trigonometry. Princeton University

  57. Wang XY, Zhang BB, Yang HY (2013) Active SVM-based relevance feedback using multiple classifiers ensemble and features reweighting. Eng Appl Artif Intell 26 (1):368–381

    Article  Google Scholar 

  58. Wu HC, Luk RWP, Wong KF, Kwok KL (2008) Interpreting TFIDF term weights as making relevance decisions. ACM Trans Inf Syst 26(3):1–37

    Article  Google Scholar 

  59. Ward Jr JH (1963) Hierarchical grouping to optimize an objective function. J Am Stat Assoc 58(301):236–244

  60. Yang Y, Nie F, Xu D, Luo J, Zhuang Y, Pan Y (2012) A multimedia retrieval framework based on semi-supervised ranking and relevance feedback. IEEE Trans Pattern Anal Mach Intell 34(4):723– 742

    Article  Google Scholar 

  61. Yu J, Lu Y, Xu Y, Sebe N, Tian Q (2007) Integrating relvance feedback in boosting for content-based image retrieval. In: IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Hawaii, USA, 15-20, pp 965–968

  62. Yan R, Hauptmann A, Jin R (2003) Multimedia search with pseudo-relevance feedback. In: Proceedings of International Conference on Image and Video Retrieval (CIVR), Illinois, USA, 24-25, pp 238– 247

  63. Zhu X, Goldberg A, Gael JV, Andrzejewski D (2007) Improving diversity in ranking using absorbing random walks, pp 97–104

  64. Zaharieva M, Diem L MIS @ retrieving diverse social images task 2015. In: Proceedings of the MediaEval 2015 Workshop, Wurzen, Germany, 14-15, 2015, CEUR-WS.org, ISSN 1613-0073, 1436, http://ceur-ws.org/Vol-1436/

  65. Zaharieva M, Schwab P A unified framework for retrieving diverse social images. In: Proceedings of the MediaEval 2014 Workshop, Barcelona, Spain, 16-17, 2014, CEURWS. org, ISSN 1613-0073, 1263, http://ceur-ws.org/Vol-1263/

Download references

Acknowledgments

This work has been funded by the Ministry of European Funds through the Financial Agreement POSDRU 187/1.5/S/155420 PROSCIENCE, and by the ESF POSDRU/159/ 1.5/S/132395 InnoRESEARCH programme. We acknowledge also the MediaEval Benchmarking Initiative for Multimedia Evaluation (http://www.multimediaeval.org) for providing the data.

Author information

Authors and Affiliations

  1. LAPI, University “Politehnica” of Bucharest, 061071, Bucharest, Romania

    Bogdan Boteanu, Ionuţ Mironică & Bogdan Ionescu

Authors
  1. Bogdan Boteanu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Ionuţ Mironică
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Bogdan Ionescu
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Bogdan Boteanu.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Boteanu, B., Mironică, I. & Ionescu, B. Pseudo-relevance feedback diversification of social image retrieval results. Multimed Tools Appl 76, 11889–11916 (2017). https://doi.org/10.1007/s11042-016-3678-6

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-016-3678-6

Keywords

Search

Navigation