Skip to main content
SpringerLink
Log in

Combining pixel domain and compressed domain index for sketch based image retrieval

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

Sketch-based image retrieval (SBIR) lets one express a precise visual query with simple and widespread means. In the SBIR approaches, the challenge consists in representing the image dataset features in a structure that allows one to efficiently and effectively retrieve images in a scalable system. We put forward a sketch-based image retrieval solution where sketches and natural image contours are represented and compared, in both, the compressed-domain of wavelet and in the pixel domain. The query is efficiently performed in the wavelet domain, while effectiveness refinements are achieved using the pixel domain to verify the spatial consistency between the sketch strokes and the natural image contours. Also, we present an efficient scheme of inverted lists for sketch-based image retrieval using the compressed-domain of wavelets. Our proposal of indexing presents two main advantages, the amount of the data to compute the query is smaller than the traditional method while it presents a better effectiveness.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8

Similar content being viewed by others

Notes

  1. Visual Geometry Group – http://www.robots.ox.ac.uk/~vgg/data/parisbuildings/index.html

  2. Visual Geometry Group – http://www.robots.ox.ac.uk/~vgg/data/parisbuildings/index.html

  3. Paris sketches and ground-truth – https://sites.google.com/site/sketchretrieval/

  4. ImageNet – http://www.imagenet.org/

References

  1. Arbelaez P, Maire M, Fowlkes C, Malik J (2011) Contour detection and hierarchical image segmentation. IEEE Trans Pattern Anal Mach Intell 33(5):898–916. doi:10.1109/TPAMI.2010.161

    Article  Google Scholar 

  2. Brogefors G (1988) Hierarchical chamfer matching: a parametric edge matching algorithm. IEEE Trans Pattern Anal Mach Intell 10(6):849–865. doi:10.1109/34.9107

    Article  Google Scholar 

  3. Bui T, Collomosse J (2015) Scalable sketch-based image retrieval using color gradient features. In: The IEEE international conference on computer vision (ICCV) workshops

  4. Cao Y, Wang C, Zhang L, Zhang L (2011) Edgel index for large-scale sketch-based image search. In: CVPR, pp 761–768

  5. Dalal N, Triggs B (2005) Histograms of oriented gradients for human detection. In: Proceedings of the 2005 IEEE computer society conference on computer vision and pattern recognition (CVPR’05) - Volume 1 - Volume 01, CVPR ’05. IEEE Computer Society, Washington, pp 886–893. doi:10.1109/CVPR.2005.177

  6. Davis J, Goadrich M (2006) The relationship between precision-recall and roc curves. ACM, New York

    Book  Google Scholar 

  7. Del Bimbo A, Pala P (1997) Visual image retrieval by elastic matching of user sketches. IEEE Trans Pattern Anal Mach Intell 19(2):121–132. doi:10.1109/34.574790

    Article  Google Scholar 

  8. DeVore RA, Jawerth BD, Lucier BJ (1992) Image compression through wavelet transform coding. IEEE Trans Inf Theory 38(2):719–746

    Article  MathSciNet  MATH  Google Scholar 

  9. Eitz M, Hays J, Alexa M (2012) How do humans sketch objects?. ACM Trans Graph 31(4):44:1–44:10. doi:10.1145/2185520.2185540

    Google Scholar 

  10. Eitz M, Hildebrand K, Boubekeur T, Alexa M (2011) Sketch-based image retrieval: benchmark and bag-of-features descriptors. IEEE Trans Vis Comput Graph 17(11):1624–1636

    Article  Google Scholar 

  11. Filho CAF, Araujó AA, Crucianu M, Gouet-Brunet V (2013) Sketch-finder: efficient and effective sketch-based retrieval for large image collections. In: Proceedings of the 2013 XXVI conference on graphics, patterns and Images, SIBGRAPI ’13. IEEE Computer Society, Washington, pp 234–241. doi:10.1109/SIBGRAPI.2013.40

  12. Flickner M, Sawhney H, Niblack W, Ashley J, Huang Q, Dom B, Gorkani M, Hafner J, Lee D, Petkovic D, Steele D, Yanker P (1995) Query by image and video content: the qbic system. Computer 28(9):23–32. doi:10.1109/2.410146

    Article  Google Scholar 

  13. Hu R, Collomosse J (2013) A performance evaluation of gradient field hog descriptor for sketch based image retrieval. Comput Vis Image Underst 117(7):790–806. doi:10.1016/j.cviu.2013.02.005

    Article  Google Scholar 

  14. Jacobs CE, Finkelstein A, Salesin DH (1995) Fast multiresolution image querying. In: Proceedings of SIGGRAPH 95, pp 277–286

  15. Jain R (2008) The art of computer systems performance analysis:. Wiley India pvt limited

  16. Kurita T, Otsu N, Hirata K (1992) A sketch retrieval method for full color image database-query by visual example. In: 1992 IEEE Computer society conference on computer vision and applications (IAPR 1992). IEEE, pp 530–533

  17. Lee YJ, Grauman K (2009) Shape discovery from unlabeled image collections. 2013 IEEE Conference on Computer Vision and Pattern Recognition 0:2254–2261. doi:10.1109/CVPRW.2009.5206698

    Google Scholar 

  18. Liu MY, Tuzel O, Veeraraghavan A, Chellappa R (2010) Fast directional chamfer matching. In: CVPR. IEEE Computer society

  19. Lowe DG (2004) Distinctive image features from scale-invariant keypoints. Int J Comput Vision 60(2):91–110. doi:10.1023/B:VISI.0000029664.99615.94

    Article  Google Scholar 

  20. Ma C, Yang X, Zhang C, Ruan X, Yang MH (2016) Sketch retrieval via local dense stroke features. Image Vis Comput 46:64–73. doi:10.1016/j.imavis.2015.11.007. http://www.sciencedirect.com/science/article/pii/S0262885615001389

    Article  Google Scholar 

  21. Polsley S, Ray J, Hammond T (2017) Sketchseeker: Finding similar sketches. IEEE Transactions on Human-Machine Systems 47(2):194–205. doi:10.1109/THMS.2017.2649684

    Article  Google Scholar 

  22. Qi Y, Song Y, Zhang H, Liu J (2016) Sketch-based image retrieval via siamese convolutional neural network. In: 2016 IEEE international conference on image processing, ICIP 2016, Phoenix, AZ, USA, September 25-28, 2016, pp 2460–2464. doi:10.1109/ICIP.2016.7532801

  23. Qian X, Tan X, Zhang Y, Hong R, Wang M (2016) Enhancing sketch-based image retrieval by re-ranking and relevance feedback. IEEE Trans Image Process 25(1):195–208. doi:10.1109/TIP.2015.2497145

    Article  MathSciNet  Google Scholar 

  24. Robertson S (2004) Understanding inverse document frequency: on theoretical arguments for idf. J Doc 60:2004

    Article  Google Scholar 

  25. Robertson S, Zaragoza H, Taylor M (2004) Simple bm25 extension to multiple weighted fields. In: Proceedings of the thirteenth ACMinternational conference on information and knowledge management, CIKM ’04. ACM, New York, pp 42–49, doi:10.1145/1031171.1031181

  26. Saavedra J, Bustos B (2010) An improved histogram of edge local orientations for sketch-based image retrieval. In: Goesele M, Roth S, Kuijper A, Schiele B, Schindler K (eds) Pattern recognition, lecture notes in computer science, vol 6376. Springer, Berlin Heidelberg, pp 432–441

  27. Saavedra JM (2014) Sketch based image retrieval using a soft computation of the histogram of edge local orientations (s-HELO). In: 2014 IEEE International conference on image processing, ICIP 2014, Paris, France, October 27-30, 2014, pp 2998–3002. doi:10.1109/ICIP.2014.7025606

  28. Saavedra JM (2015) Rst-shelo: sketch-based image retrieval using sketch tokens and square root normalization. Multimedia Tools Appl 1–21

  29. Saavedra JM, Barrios JM (2015) Sketch based image retrieval using learned keyshapes (LKS). In: Proceedings of the british machine vision conference 2015, BMVC, 2015, Swansea, UK, September 7-10, 2015, pp 164.1–164.11. doi:10.5244/C.29.164

  30. Salton G, McGill MJ (1986) Introduction to modern information retrieval. McGraw-Hill, Inc., New York

    MATH  Google Scholar 

  31. Shannon C (1948) A mathematical theory of communication. Bell Syst Tech J 27(379-423):623–656

    Article  MathSciNet  MATH  Google Scholar 

  32. Sivic J, Zisserman A (2003) Video google: a text retrieval approach to object matching in videos. In: Proceedings of the 9th IEEE international conference on computer vision - vol. 2, ICCV ’03. IEEE Computer Society, Washington, p 1470

  33. Smeulders AWM, Worring M, Santini S, Gupta A, Jain R (2000) Content-based image retrieval at the end of the early years. IEEE Trans Pattern Anal Mach Intell 22(12):1349–1380. doi:10.1109/34.895972

    Article  Google Scholar 

  34. Srinivas M, Patnaik LM (1994) Adaptive probabilities of crossover and mutation in genetic algorithms. IEEE Trans Syst Man Cybern 24(4):656–667

    Article  Google Scholar 

  35. Stenger BDR (2004) Model-based hand tracking using a hierarchical bayesian filter

  36. Sun X, Wang C, Xu C, Zhang L (2013) Indexing billions of images for sketch-based retrieval. In: Proceedings of the 21st ACM international conference on multimedia, MM ’13. ACM, New York, pp 233-242. doi:10.1145/2502081.2502281

  37. Tseng KY, Lin YL, Chen YH, Hsu WH (2012) Sketch-based image retrieval on mobile devices using compact hash bits. In: Proceedings of the 20th ACM International Conference on Multimedia, MM ’12. ACM, New York, pp 913–916

  38. Venters CC, Hartley RJ, Hewitt WT (2005) Content-based image retrieval query paradigms. In: Encyclopedia of information science and technology (i), pp 556–563

  39. Xiao C, Wang C, Zhang L, Zhang L (2015) Sketch-based image retrieval via shape words. In: Proceedings of the 5th ACM on international conference on multimedia retrieval, Shanghai, China, June 23-26, 2015, pp 571–574. doi:10.1145/2671188.2749360

Download references

Acknowledgments

The authors are grateful to CAPES/COFECUB, FAPEMIG (PPM-006-16), CNPq (307062/2016-3) and PUC Minas for the financial support to this work.

Author information

Authors and Affiliations

  1. Department of Computer Science/VIPLAB, Pontifical Catholic University of Minas Gerais (PUC Minas), Belo Horizonte, Brazil

    Carlos Alberto Fraga Pimentel Filho & Silvio Jamil Ferzoli Guimarães

  2. Department of Computer Science, University of Chile, Santiago, Chile

    Benjamin Bustos

  3. Department of Computer Science/NPDI, Federal University of Minas Gerais (UFMG), Belo Horizonte, Brazil

    Carlos Alberto Fraga Pimentel Filho & Arnaldo de Albuquerque Araújo

Authors
  1. Carlos Alberto Fraga Pimentel Filho
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Benjamin Bustos
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Arnaldo de Albuquerque Araújo
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Silvio Jamil Ferzoli Guimarães
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Silvio Jamil Ferzoli Guimarães.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Pimentel Filho, C.A.F., Bustos, B., Araújo, A.d.A. et al. Combining pixel domain and compressed domain index for sketch based image retrieval. Multimed Tools Appl 76, 22019–22042 (2017). https://doi.org/10.1007/s11042-017-4758-y

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-017-4758-y

Keywords

Search

Navigation