Skip to main content
SpringerLink
Log in

RST-SHELO: sketch-based image retrieval using sketch tokens and square root normalization

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

Sketch-based image retrieval (SBIR) is an emergent research area with a variety of applications, specially when an example image is not available for querying. Moreover, making a sketch has become a very attractive and simple task due to the already ubiquitous touch-screen and mobile technologies. Although a sketch is a natural way for representing the structure of a thought object, it may easily get confused in a dataset with high variability turning the retrieval task a quite challenging problem. Indeed, the state-of-the-art methods still show low performance on diverse evaluation datasets. Thereby, a robust sketch descriptor together with a better strategy for representing regular images as sketches are demanded. In this work, we present RST-SHELO, and improved version of SHELO (Soft Histogram of Edge Logal Orientations), an efficient state-of-the-art method for describing sketches. The proposed improvements comes from two aspects: a better technique for obtaining sketch-like representations and a better normalization strategy of SHELO. For the first case, we propose to use the sketch token approach [21], aiming to detect image contours by means of mid-level features. For the second case, we demonstrate that a square root normalization positively affect the effectiveness on the retrieval task. Based on our improvements, we present new state-of-the-art performance. To validate our achievements, we have conducted diverse experiments using two public datasets, Flickr15K and Saavedra’s. Our results show an effectiveness gain of 62 % in the first and 5 % in the second dataset.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16

Similar content being viewed by others

References

  1. Arandjelovic R, Zisserman A (2012) Three things everyone should know to improve object retrieval. In: Proceedings of the 2012 IEEE conference on computer vision and pattern recognition (CVPR), CVPR ’12, pp 2911–2918

  2. Arbelaez P, Maire M, Fowlkes C, Malik J (2011) Contour detection and hierarchical image segmentation. IEEE Transactions on Pattern Analysis and Machine Intelligence 33(5):898–916

    Article  Google Scholar 

  3. Belongie S, Malik J, Puzicha J (2002) Shape matching and object recognition using shape contexts. IEEE Transactions on Pattern Analysis and Machine Intelligence 24(4):509–522

    Article  Google Scholar 

  4. Borgefors G (1988) Hierarchical chamfer matching: a parametric edge matching algorithm. IEEE Transactions on Pattern Analysis and Machine Intelligence 10(6):849–865

    Article  Google Scholar 

  5. Cao Y, Wang C, Zhang L, Zhang L (2011) Edgel index for large-scale sketch-based image search. In: Proceedings of the 2011 IEEE conference on computer vision and pattern recognition. IEEE Computer Society, pp 761–768

  6. Chalechale A, Naghdy G, Mertins A (2005) Sketch-based image matching using angular partitioning. IEEE Trans Syst Man Cybern Syst Hum 35(1):28–41

    Article  Google Scholar 

  7. Chen T, Cheng MM, Tan P, Shamir A, Hu SM (2009) Sketch2photo: internet image montage. ACM Transactions on Graphics 28(5):124:1–124:10

    Google Scholar 

  8. Dalal N, Triggs B (2005) Histograms of oriented gradients for human detection. In: Proceedings of the 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR’05) - Volume 1 - Volume 01. IEEE Computer Society, pp 886–893

  9. Del Bimbo A, Pala P (1997) Visual image retrieval by elastic matching of user sketches. IEEE Transactions on Pattern Analysis and Machine Intelligence 19 (2):121–132

    Article  Google Scholar 

  10. Eitz M, Hildebrand K, Boubekeur T, Alexa M (2009) A descriptor for large scale image retrieval based on sketched feature lines. In: Proceedings of the 6th Eurographics symposium on sketch-based interfaces and modeling, pp 29–36

  11. Eitz M, Hildebrand K, Boubekeur T, Alexa M (2009) Photosketch: a sketch based image query and compositing system. In: SIGGRAPH 2009: Talks, SIGGRAPH ’09, pp 60:1–60:1

  12. Eitz M, Hildebrand K, Boubekeur T, Alexa M (2011) Sketch-based image retrieval: Benchmark and bag-of-features descriptors. IEEE Trans Vis Comput Graph 17(11):1624–1636

    Article  Google Scholar 

  13. Fei-Fei L, Fergus R, Perona P (2004) Lening generative visual models from few training examples: an incremental bayesian approach tested on 101 object categories

  14. Felzenszwalb P, David M, Deva R (2008) A discriminatively trained, multiscale, deformable part model. In: International conference on computer vision and pattern recognition

  15. Gonzalez R, Woods R (2008) Digital Image Processing, 3rd edn. Pearson Prentice Hall, New Jersey

    Google Scholar 

  16. Hu R, Collomosse J (2013) A performance evaluation of gradient field hog descriptor for sketch based image retrieval. Comp Vision Image Underst 117(7):790–806

    Article  Google Scholar 

  17. Hu R, Barnard M, Collomosse J (2010) Gradient field descriptor for sketch based retrieval and localization. In: 17th IEEE International Conference on Image Processing (ICIP), pp 1025–1028

  18. Hu R, Wang T, Collomosse J (2011) A bag-of-regions approach to sketch-based image retrieval. In: 18th IEEE international conference on image processing (ICIP), pp 3661–3664

  19. Lazebnik S, Schmid C, Ponce J (2006) Beyond bags of features: Spatial pyramid matching for recognizing natural scene categories. In: 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, vol 2, pp 2169–2178

  20. Li B, Schreck T, Godil A, Alexa M, Boubekeur T, Bustos B, Chen J, Eitz M, Furuya T, Hildebrand K, Huang S, Johan H, Kuijper A, Ohbuchi R, Richter R, Saavedra JM, Scherer M, Yanagimachi T, Yoon GJ, Yoon SM (2012) SHREC’12 track: sketch-based 3D shape retrieval. In: Eurographics workshop on 3D object retrieval, pp 109–118

  21. Lim J, Zitnick C, Dollar P (2013) Sketch tokens: a learned mid-level representation for contour and object detection. In: 2013 IEEE conference on computer vision and pattern recognition (CVPR), pp 3158–3165

  22. Lowe DG (2004) Distinctive image features from scale-invariant keypoints. Int J Comput Vis 60(2):91–110

    Article  Google Scholar 

  23. Mikolajczyk K, Schmid C (2005) A performance evaluation of local descriptors. IEEE Transactions on Pattern Analysis and Machine Intelligence 27(10):1615–1630

    Article  Google Scholar 

  24. Saavedra JM (2014) Sketch based image retrieval using a soft computation of the histogram of edge local orientations (s-helo). In: International conference on image processing, ICIP’2014 (To appear)

  25. Saavedra J, Bustos B (2010) An improved histogram of edge local orientations for sketch-based image retrieval. In: Pattern recognition, lecture notes in computer science, vol 6376, pp 432–441

  26. Saavedra JM, Bustos B (2013) Sketch-based image retrieval using keyshapes

  27. Sun Won C, Kwon Park D, Park SJ (2002) Efficient use of MPEG-7 edge histogram descriptor. Electronic and Telecomunications Research Institute Journal 24:23–30

    Google Scholar 

  28. Tola E, Lepetit V, Fua P (2010) Daisy: an efficient dense descriptor applied to wide-baseline stereo. In: IEEE Transactions on Pattern Analysis and Machine Intelligence, vol 32

Download references

Acknowledgments

We are grateful for financial support from two chilean institutions: CONICYT, through the projects PAI-781204025 and 14STIC-01, and CORFO-INNOVA, through the project 15ITE2-38948.

Author information

Authors and Affiliations

  1. Orand Research, Orand S.A., Estado 360 Of. 702, Santiago, Chile

    Jose M. Saavedra

Authors
  1. Jose M. Saavedra
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Jose M. Saavedra.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Saavedra, J.M. RST-SHELO: sketch-based image retrieval using sketch tokens and square root normalization. Multimed Tools Appl 76, 931–951 (2017). https://doi.org/10.1007/s11042-015-3076-5

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-015-3076-5

Keywords

Search

Navigation