Abstract
Content-based image retrieval (CBIR) algorithms have been seen as a promising access method for digital photograph collections. Unfortunately, we have very little evidence of the usefulness of these algorithms in real user needs and contexts. In this paper, we introduce a test collection for the evaluation of CBIR algorithms. In the test collection, the performance testing is based on photograph similarity perceived by end-users in the context of realistic illustration tasks and environment. The building process and the characteristics of the resulting test collection are outlined, including a typology of similarity criteria expressed by the subjects judging the similarity of photographs. A small-scale study on the consistency of similarity assessments is presented. A case evaluation of two CBIR algorithms is reported. The results show clear correlation between the subjects' similarity assessments and the functioning of feature parameters of the tested algorithms.
Article PDF
Similar content being viewed by others
References
Armitage L and Enser P (1997) Analysis of user needs in image archives. Journal of Information Science, 23: 287-299.
Belongie S, Carson C, Greenspan H and Malik J (1998) Color-and texture-based image segmentation using the expectation-maximization algorithm and its application to content-based image retrieval. In: Sixth International Conference on Computer Vision, pp. 675-682. Available URL: http://www.cs.berkeley.edu/projects/vision/ publications.html.
Das M, Manmatha R, Greenspan H and Malik J (1999) Indexing flowers by color names using domain knowledgedriven segmentation. IEEE Intelligent Systems, 14(5): 24-343.
Del Bimbo A (1999) Visual Information Retrieval. Morgan Kaufmann, San Francisco.
Eakins J (1996) Automatic image content retrieval-are we getting anywhere? In: Proceedings of the Third International Conference on Electronic Library and Visual Information Research, De Montfort University, Milton Keynes, May 1996, pp. 123-135.
Enser P (1995) Pictorial information retrieval. Journal of Documentation, 51: 126-170.
Forsyth DA and Fleck MM (1997) Body plans. In: Proceeding of IEEE Conference on Computer Vision and Pattern Recognition Puerto Rico, June 1997.
Forsyth DA, Malik J, Fleck MM, Greenspan H, Leung T, Belongie S, Carson C and Bregler C (1996) Finding pictures of objects in large collections of images. In: Proceedings of the International Workshop on Object Recognition, Cambridge, April 1996. Available URL: http://www.cs.berkeley.edu/~daf/.
Gong Y (1998) Intelligent Image Databases: Towards Advanced Image Retrieval. Kluwer Academic Publishers, Boston.
Gudivada V and Raghavan V (1997) Modeling and retrieving images by content. Information Processing & Management, 33(4): 427-452.
Gupta A and Jain R (1997) Visual information retrieval. Communications of the ACM, 40: 71-79.
Harman D (1993) The First Text Retrieval Conference (TREC-1). National Institute of Standards and Technology, Gaithersburg. (NIST Special Publication 500-207).
Iivonen M (1995) Consistency in the selection of search concepts and search terms. Information Processing & Management, 31(2): 173-190.
Keister L (1994) User Types and queries: Impact on image access systems. In: Fidel R, Hahn T, Rasmussen E and Smith P, Eds., Challenges in Indexing Electronic Text and Images. Learned Information, Medford, New Jersey, pp. 7-22.
Kekäläinen J (1999) The effects of query complexity, expansion and structure on retrieval performance in propabilistic text retrieval. Doctoral Thesis, University of Tampere, Tampere. Acta Electronica Universitatis Tamperensis, ISBN 951-44-4596-1.
Markkula M and Sormunen E (1998) Searching for photos-journalists' practices in pictorial IR. In: Eakins J, Harper D and Jose J, Eds., The Challenge of Image Retrieval. Electronic Workshops in Computing (eWIC), 1998. URL: http://www.ewic.org.uk/ewic/workshop/view.cfm/CIR-98.
Markkula M and Sormunen E (2000) End-user searching challenges indexing practices in the digital photograph archive. Information Retrieval, 1(4): 259-285.
Panofsky E (1970) Meaning in the Visual Arts. Penguin, London.
Picard R, Minka T and Szummer M (1996) Modeling user subjectivity in image libraries. M.I.T. Media Laboratory, Perceptual Computing Section Technical Report No. 382, 1996. (also IEEE Int. Conf. On Image Proc., Lausanne, Sept. 1996). Available at URL: http://picard.www.media.mit.edu/cgi-bin/tr pagemaker.
Rasmussen E (1997) Indexing images. In: Williams M, Ed., Annual Reviewof Information Science and Technology 32. Information Today, Medford, New Jersey, pp. 169-196.
Saracevic T (1984) Measuring the degree of agreement between searchers. In: Flood B, Witiak J and Hogan H, Eds., ASIS 84: Proceedings of the American Society for Information Science 47th Annual Meeting, Vol. 28. White Plans, Knowledge Industry Publications, NY, pp. 227-230.
Shatford S (1986) Analyzing the subject of a picture: A theoretical approach. Cataloguing and Classification Quarterly, 6: 39-62.
Sormunen E (2000) A method for measuring wide range performance of boolean queries in full-text databases. Doctoral Thesis, University of Tampere, Tampere. Acta Electronica Universitatis Tamperensis, ISBN: 951-44-4732-8, 231 p. URL: http://acta.uta.fi/pdf/951-44-4732-8.pdf.
Sormunen E, Markkula M and Järvelin K (1999) The perceived similarity of photos-seeking a solid basis for the evaluation of content-based retrieval algorithms. In: Draper S. et al., Eds., Mira 99: Evaluating Interactive Information Retrieval. Glasgow, UK, 4-16 April, 1999. Electronic Workshops in Computing. URL: http://www.ewic.org.uk/ewic/workshop/fetch.cfm/MIRA-99/.
Swain MJ and Ballard H (1991) Color indexing. International Journal of Computer Vision, 7(1): 11-32.
Tague-Sutcliffe J (1992) The pragmatics of information retrieval experimentation, revisited. Information Processing & Management, 28(4): 467-490.
Tico M, Haverinen T and Kuosmanen P (1999a) An unsupervised method of rough color image segmentation. In: Proceedings of the 33th Asilomar Conference on Signals, Systems and Computers, Vol. 1. Pacific Grove, California, Oct. 24-27, 1999, pp. 58-62.
Tico M, Haverinen T and Kuosmanen P (2000) A method of color histogram creation for image retrieval. In: Proceedings of the Nordic Signal Processing Symposium (NORSIG'2000), Kolmarden, Sweden, June 13-15, 2000, pp. 157-160.
Tico M and Kuosmanen P (1999b) An efficient sparse data filtering method for image histogram comparison. In: Proceedings of the 11th Scandinavian Conference on Image Analysis (SCIA'99), Kangerlussuaq, Greenland, June 1999, pp. 715-722.
Voorhees E and Harman D (1997) The Fifth Text REtrieval Conference (TREC-5). National Institute of Standards and Technology, Gaithersburg. (NIST Special Publication 500-238).
Author information
Authors and Affiliations
Rights and permissions
About this article
Cite this article
Markkula, M., Tico, M., Sepponen, B. et al. A Test Collection for the Evaluation of Content-Based Image Retrieval Algorithms—A User and Task-Based Approach. Information Retrieval 4, 275–293 (2001). https://doi.org/10.1023/A:1011954407169
Issue Date:
DOI: https://doi.org/10.1023/A:1011954407169