Skip to main content
SpringerLink
Log in

Relevance feedback based saliency adaptation in CBIR

  • Regular Paper
  • Published:
Multimedia Systems Aims and scope Submit manuscript

Abstract

Content-based image retrieval (CBIR) has been under investigation for a long time, with many systems built to meet different application demands. However, in all systems there is still a gap between user expectations and system retrieval capabilities. Therefore, user interaction is an essential component of any CBIR system. Interaction up to now has mostly focused on changing global image features or similarities between images. We consider the interaction with salient details in an image, i.e., points, lines, and regions. Interactive salient detail definition goes further than summarizing the image into a set of salient details. We aim to dynamically update the user- and context-dependent definition of saliency based on relevance feedback. To that end, we propose an interaction framework for salient details from the perspective of the user. A number of instantiations of the framework are presented. Finally, we apply our approach for query refinement in a detail-based image retrieval system with salient points and regions. Experimental results prove the effectiveness of adapting the saliency from user feedback in the retrieval process.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Similar content being viewed by others

References

  1. Alter, T., Basri, R.: Extracting salient curves from images: an analysis of saliency network. Int. J. Comput. Vis. 27(1), 51–69 (1998)

    Article  Google Scholar 

  2. Barnard, K., Duygulu, P., Guru, R., Gabbur, P., Forsyth, D.: The effects of segmentation and feature choice in a translation model of object recognition. In: International Conference on Pattern Recognition (2003)

  3. Bres, S., Jolion, J.: Detection of interest points for image indexation. In: 3rd International Conference on Visual Information Systems, pp. 427–434 (1999)

  4. Canny, J.: A computational approach to edge detection. IEEE Trans. Pattern Anal. Mach. Intell. 8(6), 679–698 (1986)

    Google Scholar 

  5. Carson, C., Belongie, S., Greenspan, H., Malik, J.: Blobworld: image segmentation using expectation-maximization and its application to image querying. IEEE Trans. Pattern Anal. Mach. Intell. 24(8), 1026–1038 (2002)

    Article  Google Scholar 

  6. Cinque, L., Lecca, F., Levialdi, S., Tanimoto, S.: Retrieval of images using rich region descriptions. In: 14th International Conference on Pattern Recognition, vol. 1 (1998)

  7. Dimai, A.: Unsupervised extraction of salient region-descriptors for content based image retrieval. In: Proceedings of the 10th International Conference on Image Analysis and Processing (1998)

  8. Fleck, M.: Some defects in finite-difference edge finders. IEEE Trans. Pattern Anal. Mach. Intell. 14(3), 337–345 (1992)

    Article  Google Scholar 

  9. Flickner, M., Sawhney, H., Niblack, W., Ashley, J., Huang, Q., Dom, B.: Query by image and video content: the qbic system. IEEE Comput. 28(9), 23–32 (1995)

    Google Scholar 

  10. Gevers, T., Smeulders, A.: Pictoseek: combining color and shape invariant features for image retrieval. IEEE Trans. Image Process. 9(1), 102–119 (2000)

    Article  Google Scholar 

  11. Hoang, M., Geusbroek, J., Smeulders, A.: Color texture measurement and segmentation. In: Signal Processing (2003)

  12. Iqbal, Q., Aggarwal, J.: Retrieval by classification of images containing large manmade objects using perceptual grouping. Pattern Recognit. 35, 1463–1479 (2001)

    Google Scholar 

  13. Itti, L., Koch, C., Niebur, E.: A model of saliency-based visual attention for rapid scene analysis. IEEE Trans. Pattern Anal. Mach. Intell. 20(11), 1254–1259 (1998)

    Article  Google Scholar 

  14. Jacobs, C., Finkelstein, A., Salesin, D.: Fast multiresolution image querying. In: Proceedings of the 22nd Annual Conference on Computer Graphics and Interactive Techniques, pp. 277–286 (1995)

  15. Kam, A.: A general multiscale scheme for unsupervised image segmentation. Phd thesis, University of Cambridge (2000)

  16. Kovesi, P.: http://www.csse.uwa.edu.au/~pk/research/matlabfns/

  17. MacArthur, S., Brodley, C., Kak, A., Broderick, L.: Interactive content-based image retrieval using relevance feedback. Comput. Vis. Image Understand. 88(2), 55–75 (2002)

    Article  Google Scholar 

  18. Mahamud, S., Williams, L., Thornber, K., Xu, K.: Segmentation of multiple salient closed contours from real images. In: IEEE Trans. Pattern Anal. Mach. Intell. 25(4), 891–897 (1999)

    Google Scholar 

  19. Marques, O., Costa, F.M., Furht, B.: Content-based image search and retrieval using relevance feedback: the muse project. In: Proceedings of the International Association of Science and Technology for Development (2000)

  20. Martinez, A., Serra, J.: A new approach to object-related image retrieval. J. Vis. Lang. Comput. 11, 345–363 (2000)

    Google Scholar 

  21. Mikolajczyk, K., Schmid, C.: Indexing based on scale invariant interest points. In: International Conference on Computer Vision, pp. 525–531 (2001)

  22. Mikolajczyk, K., Schmid, C.: An affine invariant interest point detector. In: European Conference on Computer Vision, pp. 128–142 (2002)

  23. Minka, T., Picard, R.: Interactive learning using a ‘society of models’. In: International Conference on Computer Vision and Pattern Recognition (1996)

  24. Muller, H., Muller, W., Maillet, S., Pun, T., Squire, D.: Strategies for positive and negative relevance feedback in image retrieval. In: Proceedings of the 15th International Conference on Pattern Recognition (2000)

  25. Pauwels, E., Frederix, G.: Fiding salient regions in images—nonparametric clustering for image segmentation and grouping. Comput. Vis. Image Understand. 75(1/2), 73–85 (1999)

    Google Scholar 

  26. Rosin, P.: Edges: saliency measures and automatic thresholding. Mach. Vis. Appl. 9, 139–159 (1997)

    Google Scholar 

  27. Rui, Y., Huang, T.: Optimizing learning in image retrieval. In: International Conference on Computer Vision and Pattern Recognition (2000)

  28. Rui, Y., Huang, T., Chang, S.: Image retrieval: current techniques, promising directions and open issues. J. Vis. Commun. Image Represent. 10, 39–62 (1999)

    Google Scholar 

  29. Rui, Y., Huang, T., Mehrotra, S.: Relevance feedback: a power tool for interactive content-based image retrieval. IEEE Trans. Circuits Video Technol. 8(5), 644–655 (1998)

    Google Scholar 

  30. Salah, A., Alpaydin, E., Akarun, L.: A selective attention-based method for visual pattern recognition with application to handwritten digit recognition and face recognition. IEEE Trans. Pattern Anal. Mach. Intell. 24(3), 420–425 (2002)

    Article  Google Scholar 

  31. Santini, S., Gupta, A., Jain, R.: Emergent semantics through interaction in image databases. In: IEEE Transactions on Knowledge and Data Engineering (2001)

  32. Schmid, C., Mohr, R.: Local grayvalue invariants for image retrieval. IEEE Trans. Pattern Anal. Mach. Intell. 15(5), 530–535 (1997)

    Google Scholar 

  33. Schreiber, A., Dubbeldam, B., Wielemaker, J., Wielinga, B.: Ontology-based photo annotation. In: IEEE Intelligent Systems (2001)

  34. Sebe, N., Tian, Q., Loupias, E., Lew, M., Huang, T.: Salient points for content based image retrieval. In: International Conference on Computer Vision and Pattern Recognition (2001)

  35. Smeulders, A., Worring, M., Santini, S., Gupta, A., Jain, R.: Content-based image retrieval at the end of early years. IEEE Trans. Pattern Anal. Mach. Intell. 22(12), 1349–1380 (2000)

    Article  Google Scholar 

  36. Stollnitz, E., DeRose, T., Salesin, D.: Wavelets for computer graphics: a primer, part 2. IEEE Comput. Graph. Appl. 15(4), 75–85 (1995)

    Article  Google Scholar 

  37. Stricker, M., Orengo, M.: Similarity of color images. Proc. SPIE Stor. Retrieval Image Video Databases 2420, 381–392 (1995)

    Google Scholar 

  38. Swain, M., Ballard, D.: Color indexing. Int. J. Comput. Vis. 7(1), 11–32 (1991)

    Article  Google Scholar 

  39. Vendrig, J., Worring, M., Smeulders, A.: Filter image browsing: interactive image retrieval by using database overviews. Multimedia Tools Appl. 15(1), 83–103 (2001)

    Google Scholar 

  40. Walker, K., Cootes, T., Taylor, C.: Locating salient object features. In: 9th British Machine Vision Conference 2, 557–566 (1998)

    Google Scholar 

  41. Walker, K., Cootes, T.F., Taylor, C.J.: Locating salient facial features using image invariants. In: 3rd International Conference on Automatic Face and Gesture Recognition, pp. 242–247 (1998)

Download references

Author information

Authors and Affiliations

  1. Intelligent Sensory Information Systems, Faculty of Science, University of Amsterdam, Kruislaan 403, 1098SJ, Amsterdam, The Netherlands

    Giang P. Nguyen & Marcel Worring

Authors
  1. Giang P. Nguyen
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Marcel Worring
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Giang P. Nguyen.

Additional information

Categories and subject descriptors

H.3.3 [Information storage and retrieval:] Information search and retrieval—query formulation, relevance feedback

General terms

Image retrieval

Rights and permissions

Reprints and permissions

About this article

Cite this article

Nguyen, G.P., Worring, M. Relevance feedback based saliency adaptation in CBIR. Multimedia Systems 10, 499–512 (2005). https://doi.org/10.1007/s00530-005-0178-3

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00530-005-0178-3

Keywords

Search

Navigation