Abstract
In this paper we proposed the use of spatial relations as a way of improving annotation-based image retrieval. We analyzed different types of spatial relations and selected the most adequate ones for image retrieval. We developed an image comparison and retrieval method based on conceptual graphs, which incorporates spatial relations. Additionally, we proposed an alternative term-weighting scheme and explored the use of more than one sample image for retrieval using several late fusion techniques. Our methods were evaluated with a rich and complex image dataset, based on the 39 topics developed for the ImageCLEF 2008 photo retrieval task. Results show that: (i) incorporating spatial relations produces a significant increase in performance, (ii) the label weighting scheme we proposed obtains better results than other traditional schemes, and (iii) the combination of several sample images using late fusion produces an additional improvement in retrieval according to several metrics.
Similar content being viewed by others
References
Arni T, Sanderson M, Clough P, Grubinger M (2008) Overview of the imageclef 2008 photographic retrieval task. Working Notes of the CLEF
Aslam JA, Montague M (2001) Models for metasearch. In: Proceedings of the 24th annual international ACM SIGIR conference on research and development in information retrieval, pp 276–284
Bekhatir M, Chiaramella Y, Mulhem P (2005) A signal/semantic framework for image retrieval. In: JCDL 005: proceedings of the 5th ACM/IEEE-CS joint conference on digital libraries. ACM, pp 368–368
Berretti S, Bimbo AD, Vicario E (2003) Weighted walkthroughs between extended entities for retrieval by spatial arrangement. IEEE Trans Multimedia 5(1):52–70
Berretti S, Del Bimbo A, Vicario E (2001) Efficient matching and indexing of graph models in content-based retrieval. IEEE Trans Pattern Anal Mach Intell 23(10):1089–1105. doi:10.1109/34.954600
Carneiro G, Chan AB, Moreno PJ, Vasconcelos N (2007) Supervised learning of semantic classes for image annotation and retrieval. IEEE Trans Pattern Anal Mach Intell 29:2007
Chang SK, Shi QY, Yan CW (1987) Iconic indexing by 2-d strings. IEEE Trans Pattern Anal Mach Intell 9(3):413–428
Chen J, Li Z, Li C, Gold CM (1998) Describing topological relations with voronoi-based 9-intersection model. Int Arch Photogramm Remote Sens 32(4):99–104
Datta R, Joshi D, Li J, Wang JZ (2008) Image retrieval: ideas, influences, and trends of the new age. ACM Comput Surv 39:1–60
Duygulu P, Barnard K, de Freitas JFG, Forsyth DA (2002) Object recognition as machine translation: learning a lexicon for a fixed image vocabulary. In: Proceedings of the seventh European conference on computer vision, vol 4, pp 97–112. citeseer.ist.psu.edu/duygulu02object.html
Egenhofer MJ (1989) A formal definition of binary topological relationships. In: Proceedings of the 3rd international conference, FODO 1989 on foundations of data organization and algorithms, pp 457–472
Egenhofer MJ (1993) Definitions of line-line relations for geographic databases. IEEE Data Eng Bull 16(3):40–45
Egenhofer MJ, Franzosa RD (1991) Point-set topological spatial relations. Int J Geogr Inf Syst 5(2):161–174
Escalante H, Hernández-Gracidas C, González J, López A, Montes M, Morales E, Sucar L, Nor LV, Grubinger M (2010) The segmented and annotated iapr-tc12 benchmark. Comput Vis Image Underst 114(4):419–428
Frank AU (1992) Qualitative spatial reasoning about distances and directions in geographic space. J Visual Lang Comput 3(4):343–371
Goodrum A (2000) Image information retrieval: an overview of current research. Inf Sci 3:2000
Goyal RK, Egenhofer MJ (2000) Cardinal directions between extended spatial objects. IEEE Trans Knowl Data Eng
Grubinger M (2007) Analysis and evaluation of visual information systems performance. PhD thesis, School of Computer Science and Mathematics, Faculty of Health, Engineering and Science, Victoria University, Melbourne, Australia
Hernández-Gracidas C, Sucar LE (2007) Markov random fields and spatial information to improve automatic image annotation. In: Proceedings of the 2nd Pacific Rim conference on advances in image and video technology, PSIVT’07. Springer, Berlin, Heidelberg, pp 879–892
Hernández-Gracidas C, Sucar LE, Montes-y Gómez M (2009) Modeling spatial relations for image retrieval by conceptual graphs. In: Proceedings of the first Chilean workshop on pattern recognition
Hernández-Gracidas C, Juárez A, Sucar LE, Montes-y Gómez M, Villaseñor L (2010) Data fusion and label weighting for image retrieval based on spatio-conceptual information. In: Proceedings of the 9th international conference on adaptivity, personalization and fusion of heterogeneous information (RIAO)
Jones KS (1972) A statistical interpretation of term specificity and its application in retrieval. J Doc 28:11–21
Kanji GK (1993) 100 statistical tests / Gopal K. Kanji. Sage Publications, London, Newbury Park, California
Lee JH (1997) Analyses of multiple evidence combination. In: SIGIR ’97: proceedings of the 20th annual international ACM SIGIR conference on research and development in information retrieval. ACM, New York, pp 267–276
Mechkour M, Berrut C, Chiaramella Y (1995) Using a conceptual graph framework for image retrieval. In: Proc. of the MMM95 (multimedia modeling) conference, pp 127–142
Millard DE, Gibbins NM, Michaelides DT, Weal MJ (2005) Mind the semantic gap. In: HYPERTEXT ’05: proceedings of the sixteenth ACM conference on hypertext and hypermedia. ACM, New York, pp 54–62
Montes M, Gelbukh A, López-López A, Baeza-Yates R (2000) Comparison of conceptual graphs. In: Proceedings of the Mexican international conference on articial intelligence, MICAI 2000
Paul K (1995) Decision level data fusion for routing of documents in the trec3 context: a best case analysis of worst case results. In: Proceedings of the third text retrieval conference (TREC-3)
Picard RW, Minka TP (1995) Vision texture for annotation. In: Multimedia systems
Rathi V, Majumdar AK (2002) Content based image search over the world wide web. In: Proceedings of the Indian conference on computer vision, graphics and image processing
Ren W, Singh M, Singh S (2002) Image retrieval using spatial context. In: Proceedings of the 9th international workshop on systems, signals and image processing, pp 44–49
Rui Y, Huang TS, Chang S (1999) Image retrieval: current techniques, promising directions and open issues. J Vis Commun Image Represent 10(1):39–62
Shaw JA, Fox EA, Shaw JA, Fox EA (1994) Combination of multiple searches. In: Proceedings of the second text retrieval conference (TREC-2), pp 243–252
Sistla AP, Yu C, Haddack R (1994) Reasoning about spatial relationships in picture retrieval systems. In: Proceedings of the 20th international conference on very large data bases, pp 570–581
Smeulders AWM, Member S, Worring M, Santini S, Gupta A, Jain R (2000) Content-based image retrieval at the end of the early years. IEEE Trans Pattern Anal Mach Intell 22:1349–1380
Sowa JF (1984) Conceptual structures: information processing in mind and machine. Addison-Wesley Longman Publishing Co., Inc., Boston
Vogt CC, Cottrell GW (1999) Fusion via a linear combination of scores. Inf Retr 1(3):151–173
Wu S, McClean S (2006) Performance prediction of data fusion for information retrieval. Inf Process Manag 42(4):899–91
Yuan J, Li J, Zhang B (2007) Exploiting spatial context constraints for automatic image region annotation. In: MULTIMEDIA ’07: proceedings of the 15th international conference on multimedia. ACM, New York, pp 595–604
Zhang Q, Yau SS (2004) On intractability of spatial relationships in content-based image database systems. Commun Inf Syst 4(2):181–190
Zhou X, Chen J, Li Z, Zhao R, Zhu J (2005) A model for topological relationships based on euler number. In: Proceedings of the ISPRS Hangzhou workshop
Acknowledgements
This research was partially supported by CONACyT under project grant CB-2008-01-103878 and postdoctoral fellowship 10216. Thanks to Dr. Antonio Juárez who kindly processed our lists with his fusion methods to obtain results for fuzzy Borda count and combMNZ.
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Hernández-Gracidas, C.A., Sucar, L.E. & Montes-y-Gómez, M. Improving image retrieval by using spatial relations. Multimed Tools Appl 62, 479–505 (2013). https://doi.org/10.1007/s11042-011-0911-1
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11042-011-0911-1