Abstract
CLEF first launched a multi-lingual visual information retrieval task in 2003 as part of the ImageCLEF track. Several such tasks subsequently followed, encompassing both the medical and non-medical domains. The main aim of such ad hoc image retrieval tasks was to investigate the effectiveness of retrieval approaches that exploit textual and visual evidence in the context of large and heterogeneous collections of images that are searched for by users with diverse information needs. This chapter presents an overview of the image retrieval activities within ImageCLEF from 2003 to 2011, focusing on the non-medical domain and, in particular, on the photographic retrieval and Wikipedia image retrieval tasks. We review the available test collections built in the context of these activities, present the main evaluation results, and summarise the contributions and lessons learned.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
Similar content being viewed by others
References
Agrawal R, Gollapudi S, Halverson A, Ieong S (2009) Diversifying search results. In: Proceedings of the second ACM international conference on web search and data mining, WSDM‘09. ACM, New York, pp 5–14. http://doi.acm.org/10.1145/1498759.1498766
Arni T, Clough P, Sanderson M, Grubinger M (2009) Overview of the ImageCLEFphoto 2008 photographic retrieval task. In: Peters C, Deselaers T, Ferro N, Gonzalo J, Jones GJF, Kurimo M, Mandl T, Peñas A (eds) Evaluating systems for multilingual and multimodal information access: ninth workshop of the cross–language evaluation forum (CLEF 2008). Revised Selected Papers, Lecture notes in computer science (LNCS), vol 5706. Springer, Heidelberg, pp 500–511
Boididou C, Papadopoulos S, Kompatsiaris Y, Schifferes S, Newman N (2014) Challenges of computational verification in social multimedia. In: Proceedings of the 23rd international conference on world wide web, WWW‘14 companion. ACM, New York, pp 743–748. http://doi.acm.org/10.1145/2567948.2579323
Buckley C, Voorhees EM (2000) Evaluating evaluation measure stability. In: Proceedings of the 23rd annual international ACM SIGIR conference on research and development in information retrieval, SIGIR‘00. ACM, New York, pp 33–40. http://doi.acm.org/10.1145/345508.345543
Buckley C, Voorhees EM (2004) Retrieval evaluation with incomplete information. In: Proceedings of the 27th annual international ACM SIGIR conference on research and development in information retrieval, SIGIR‘04. ACM, New York, pp 25–32. http://doi.acm.org/10.1145/1008992.1009000
Clarke CL, Kolla M, Cormack GV, Vechtomova O, Ashkan A, Büttcher S, MacKinnon I (2008) Novelty and diversity in information retrieval evaluation. In: Proceedings of the 31st annual international ACM SIGIR conference on research and development in information retrieval, SIGIR‘08. ACM, New York, pp 659–666. http://doi.acm.org/10.1145/1390334.1390446
Clough P, Sanderson M (2004) The CLEF 2003 cross language image retrieval track. In: Peters C, Braschler M, Gonzalo J, Kluck M (eds) Comparative evaluation of multilingual information access systems: fourth workshop of the cross–language evaluation forum (CLEF 2003), revised selected papers. Lecture notes in computer science (LNCS), vol 3237. Springer, Heidelberg, pp 581–593
Clough P, Müller H, Sanderson M (2005) The CLEF 2004 cross-language image retrieval track. In: Peters C, Clough P, Gonzalo J, Jones GJF, Kluck M, Magnini B (eds) Multilingual information access for text, speech and images: fifth workshop of the cross–language evaluation forum (CLEF 2004). Revised selected papers. Lecture notes in computer science (LNCS), vol 3491. Springer, Heidelberg, pp 597–613
Clough P, Grubinger M, Deselaers T, Hanbury A, Müller H (2006a) Overview of the ImageCLEF 2006 photographic retrieval and object annotation tasks. In: Nardi A, Peters C, Vicedo JL, Ferro N (eds) CLEF 2006 working notes. CEUR workshop proceedings (CEUR-WS.org). ISSN 1613-0073. http://ceur-ws.org/Vol-1172/
Clough PD, Sanderson M, Reid N (2006b) The eurovision St Andrews collection of photographs. SIGIR Forum 40(1):21–30. http://doi.acm.org/10.1145/1147197.1147199
Clough P, Grubinger M, Deselaers T, Hanbury A, Müller H (2007) Overview of the ImageCLEF 2006 photographic retrieval and object annotation tasks. In: Peters C, Clough P, Gey FC, Karlgren J, Magnini B, Oard DW, de Rijke M, Stempfhuber M (eds) Evaluation of multilingual and multi-modal information retrieval: seventh workshop of the cross–language evaluation forum (CLEF 2006). Revised selected papers, Lecture notes in computer science (LNCS), vol 4730. Springer, Heidelberg, pp 223–256
Del Bimbo A (1999) Visual information retrieval. Morgan Kaufmann, San Francisco
Grubinger M, Clough P, Leung C (2006) The IAPR TC-12 benchmark for visual information search. IAPR Newsl 28(2):10–12
Grubinger M, Clough P, Hanbury A, Müller H (2008) Overview of the ImageCLEFphoto 2007 photographic retrieval task. In: Peters C, Jijkoun V, Mandl T, Müller H, Oard DW, Peñas A, Petras V, Santos D (eds) Advances in multilingual and multimodal information retrieval: eighth workshop of the cross–language evaluation forum (CLEF 2007). Revised selected papers, Lecture notes in computer science (LNCS), vol 5152. Springer, Heidelberg, pp 433–444
Husain SS, Bober M (2017) Improving large-scale image retrieval through robust aggregation of local descriptors. IEEE Trans Pattern Anal Mach Intell 39(9):1783–1796. https://doi.org/10.1109/TPAMI.2016.2613873
Müller H, Deselaers T, Grubinger M, Clough P, Hanbury A, Hersh W (2007) Problems with running a successful multimedia retrieval benchmark. In: Proceedings of the third MUSCLE/ImageCLEF workshop on image and video retrieval evaluation
Müller H, Kalpathy-Cramer J, Eggel I, Bedrick S, Radhouani S, Bakke B, Khan CE, Jr, Hersh WR (2010) Overview of the CLEF 2009 medical image retrieval track. In: Peters C, Tsikrika T, Müller H, Kalpathy-Cramer J, Jones GJF, Gonzalo J, Caputo B (eds) Multilingual information access evaluation vol. ii multimedia experiments – tenth workshop of the cross–language evaluation forum (CLEF 2009). Revised selected papers, Lecture notes in computer science (LNCS). Springer, Heidelberg, pp 72–84
Paramita ML, Grubinger M (2010) Photographic image retrieval. In: Müller H, Clough PD, Deselaers T, Caputo B (eds) ImageCLEF, experimental evaluation in visual information retrieval. Springer, Berlin, pp 141–162. https://doi.org/10.1007/978-3-642-15181-1_8
Paramita ML, Sanderson M, Clough P (2009a) Developing a test collection to support diversity analysis. In: Proceedings of redundancy, diversity, and IDR workshop-SIGIR, vol 9, pp 39–45
Paramita ML, Sanderson M, Clough P (2009b) Diversity in photo retrieval: overview of the imageclefphoto task 2009. In: Workshop of the cross-language evaluation forum for European languages. Springer, Berlin pp 45–59
Popescu A, Tsikrika T, Kludas J (2010) Overview of the wikipedia retrieval task at ImageCLEF 2010. In: Braschler M, Harman DK, Pianta E, Ferro N (eds) CLEF 2010 working notes, CEUR workshop proceedings (CEUR-WS.org). ISSN 1613-0073. http://ceur-ws.org/Vol-1176/
Russakovsky O, Deng J, Su H, Krause J, Satheesh S, Ma S, Huang Z, Karpathy A, Khosla A, Bernstein M, Berg AC, Fei-Fei L (2015) Imagenet large scale visual recognition challenge. Int J Comput Vis 115(3):211–252. https://doi.org/10.1007/s11263-015-0816-y
Santos RLT, Macdonald C, Ounis I (2015) Search result diversification. Found Trends Inf Retr 9(1):1–90. http://dx.doi.org/10.1561/1500000040
Tsikrika T, Kludas J (2009) Overview of the WikipediaMM task at ImageCLEF 2008. In: Peters C, Deselaers T, Ferro N, Gonzalo J, Jones GJF, Kurimo M, Mandl T, Peñas A (eds) Evaluating systems for multilingual and multimodal information access: ninth workshop of the cross–language evaluation forum (CLEF 2008). Revised selected papers, Lecture notes in computer science (LNCS), vol 5706. Springer, Heidelberg, pp 539–550
Tsikrika T, Kludas J (2010) Overview of the WikipediaMM task at ImageCLEF 2009. In: Peters C, Tsikrika T, Müller H, Kalpathy-Cramer J, Jones GJF, Gonzalo J, Caputo B (eds) Multilingual information access evaluation vol. ii multimedia experiments – tenth workshop of the cross–language evaluation forum (CLEF 2009). Revised selected papers, Lecture notes in computer science (LNCS). Springer, Heidelberg, pp 60–71
Tsikrika T, Garcia Seco de Herrera A, Müller H (2011a) Assessing the scholarly impact of ImageCLEF. In: Forner P, Gonzalo J, Kekäläinen J, Lalmas M, de Rijke M (eds) Multilingual and multimodal information access evaluation. Proceedings of the second international conference of the cross-language evaluation forum (CLEF 2011). Lecture notes in computer science (LNCS), vol 6941. Springer, Heidelberg, pp 95–106
Tsikrika T, Popescu A, Kludas J (2011b) Overview of the Wikipedia image retrieval task at ImageCLEF 2011. In: Petras V, Forner P, Clough P, Ferro N (eds) CLEF 2011 working notes. CEUR workshop proceedings (CEUR-WS.org). ISSN 1613-0073. http://ceur-ws.org/Vol-1177/
Tsikrika T, Kludas J, Popescu A (2012) Building reliable and reusable test collections for image retrieval: the wikipedia task at imageclef. IEEE MultiMedia 19(3):24–33
Tsikrika T, Larsen B, Müller H, Endrullis S, Rahm E (2013) The scholarly impact of CLEF (2000–2009). In: Forner P, Müller H, Paredes R, Rosso P, Stein B (eds) Information access evaluation meets multilinguality, multimodality, and visualization. Proceedings of the fourth international conference of the CLEF initiative (CLEF 2013). Lecture notes in computer science (LNCS), vol 8138. Springer, Heidelberg, pp 1–12
Wan J, Wang D, Hoi SCH, Wu P, Zhu J, Zhang Y, Li J (2014) Deep learning for content-based image retrieval: a comprehensive study. In: Proceedings of the 22Nd ACM international conference on multimedia, MM‘14. ACM, New York, pp 157–166. http://doi.acm.org/10.1145/2647868.2654948
Westerveld T, van Zwol R (2007) Multimedia retrieval at inex 2006. ACM SIGIR Forum 41(1):58–63
Zhai CX, Cohen WW, Lafferty J (2003) Beyond independent relevance: methods and evaluation metrics for subtopic retrieval. In: Proceedings of the 26th annual international ACM SIGIR conference on research and development in information retrieval, SIGIR‘03. ACM, New York, pp 10–17. http://doi.acm.org/10.1145/860435.860440
Acknowledgements
We thank all those involved in helping to coordinate and organise ImageCLEF over the years and all those who have engaged with the track, providing submissions and presenting at CLEF.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2019 Springer Nature Switzerland AG
About this chapter
Cite this chapter
Clough, P., Tsikrika, T. (2019). Multi-Lingual Retrieval of Pictures in ImageCLEF. In: Ferro, N., Peters, C. (eds) Information Retrieval Evaluation in a Changing World. The Information Retrieval Series, vol 41. Springer, Cham. https://doi.org/10.1007/978-3-030-22948-1_9
Download citation
DOI: https://doi.org/10.1007/978-3-030-22948-1_9
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-22947-4
Online ISBN: 978-3-030-22948-1
eBook Packages: Computer ScienceComputer Science (R0)