Skip to main content
SpringerLink
Log in

A survey of sketch-based image retrieval

  • Original Paper
  • Published:
Machine Vision and Applications Aims and scope Submit manuscript

Abstract

Sketch-based image retrieval (SBIR) has been studied since the early 1990s and has drawn more and more interest recently. Yet, a comprehensive review of the SBIR field is still absent. This survey tries to fill in this gap by reviewing the representative papers studying the SBIR problem. More importantly, this survey tries to answer two important questions which are generally not well discussed: what are the objectives of SBIR, and what is the general methodology of SBIR? The reviewed papers are organized in a chronological way and analyzed by answering these two important questions. As a novel trend, fine-grained SBIR has become the main topic for the recent research. The discussion on it is also integrated. From this survey, we hope that different perspectives can be observed, common values can be discovered and new ideas can be inspired.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2

Similar content being viewed by others

References

  1. Abdulbaqi, H.A., Sulong, G., Hashem, S.H.: A sketch based image retrieval: a rreview of literature. J. Theor. Appl. Inf. Technol. 63(1), 158–167 (2014)

    Google Scholar 

  2. Belongie, S., Malik, J., Puzicha, J.: Shape matching and object recognition using shape contexts. IEEE Trans. Pattern Anal. Mach. Intell. 24(4), 509–522 (2002)

    Article  Google Scholar 

  3. Bentley, J.L.: Multidimensional binary search trees used for associative searching. Commun. ACM 18(9), 509–517 (1975)

    Article  Google Scholar 

  4. Birari, D.R., Shinde, J.: Survey on sketch based image retrieval. Int. J. Adv. Res. Comput. Commun. Eng. 4(12), 513–516 (2015)

    Google Scholar 

  5. Candemir, S., Borovikov, E., Santosh, K.C., Antani, S.K., Thoma, G.R.: Rsilc: rotation- and scale-invariant, line-based color-aware descriptor. Image Vis. Comput. 42, 1–12 (2015)

    Article  Google Scholar 

  6. Canny, J.: A computational approach to edge detection. IEEE Trans. Pattern Anal. Mach. Intell. 8(6), 679–698 (1986)

    Article  Google Scholar 

  7. Cao, Y., Wang, C., Zhang, L., Zhang, L.: Edgel index for large-scale sketch-based image search. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 761–768 (2011)

  8. Chalechale, A., Naghdy, G., Mertins, A.: Sketch-based image matching using angular partitioning. IEEE Trans. Syst. Man Cybern. 35(1), 28–41 (2005)

    Article  Google Scholar 

  9. Chans, Y., Lei, Z., Lopresti, D.P., Kung, S.Y.: A feature-based approach for image retrieval by sketch. In: SPIE International Symposium on Voice, Video and Data Communications, pp. 220–231 (1997)

  10. Chopra, S., Hadsell, R., LeCun, Y.: Learning a similarity metric discriminatively, with application to face verification. In: Proceedings of the IEEE Computer Conference on Computer Vision and Pattern Recognition, pp. 539–546 (2005)

  11. Dalal, N., Triggs, B.: Histograms of oriented gradients for human detection. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 886–893 (2005)

  12. Datta, R., Joshi, D., Li, J., Wang, J.Z.: Image retrieval: ideas, influences, and trends of the new age. ACM Comput. Surv. 40(2), 5:1–5:60 (2008)

    Article  Google Scholar 

  13. Del Bimbo, A., Pala, P.: Visual image retrieval by elastic matching of user sketches. IEEE Trans. Pattern Anal. Mach. Intell. 19(2), 121–132 (1997)

    Article  Google Scholar 

  14. Del Bimbo, A., Pala, P., Santini, S.: Visual image retrieval by elastic deformation of object sketches. In: IEEE Symposium on Visual Languages, pp. 216–223 (1994)

  15. Del Bimbo, A., Pala, P., Santini, S.: Image retrieval by elastic matching of shapes and image patterns. In: IEEE International Conference on Multimedia Computing and Systems, pp. 215–218 (1996)

  16. Eitz, M., Hays, J., Alexa, M.: How do humans sketch objects? ACM Trans. Graph. (Proceedings of SIGGRAPH) 31(4), 44:1–44:10 (2012)

    Google Scholar 

  17. Eitz, M., Hildebrand, K., Boubekeur, T., Alexa, M.: A descriptor for large scale image retrieval based on sketched feature lines. In: Eurographics Symposium on Sketch-Based Interfaces and Modeling, pp. 29–36 (2009)

  18. Eitz, M., Hildebrand, K., Boubekeur, T., Alexa, M.: An evaluation of descriptors for large-scale image retrieval from sketched feature lines. Comput. Graph. 34(5), 482–498 (2010)

    Article  Google Scholar 

  19. Eitz, M., Hildebrand, K., Boubekeur, T., Alexa, M.: Sketch-based image retrieval: Benchmark and bag-of-features descriptors. IEEE Trans. Vis. Comput. Graph. 17(11), 1624–1636 (2011)

    Article  Google Scholar 

  20. Everingham, M., Van Gool, L., Williams, C.K.I., Winn, J., Zisserman, A.: The PASCAL Visual Object Classes Challenge 2010 (VOC2010) Results. http://www.pascal-network.org/challenges/VOC/voc2010/workshop/index.html

  21. Faloutsos, C., Barber, R., Flickner, M., Hafner, J., Niblack, W., Petkovic, D., Equitz, W.: Efficient and effective querying by image content. J. Intell. Inf. Syst. 3(3–4), 231–262 (1994)

    Article  Google Scholar 

  22. Felzenszwalb, P.F., Girshick, R.B., McAllester, D., Ramanan, D.: Object detection with discriminatively trained part based models. IEEE Trans. Pattern Anal. Mach. Intell. 32(9), 1627–1645 (2010)

    Article  Google Scholar 

  23. Fukunage, K., Narendra, P.M.: A branch and bound algorithm for computing k-nearest neighbors. IEEE Trans. Comput. 24(7), 750–753 (1975)

    Article  Google Scholar 

  24. Gaidhani, P.A., Bagal, S.: Survey paper on sketch based and content based image retrieval. Int. J. Sci. Res. 4(12) (2015)

  25. Gao, Y., Leung, M.K.: Face recognition using line edge map. IEEE Trans. Pattern Anal. Mach. Intell. 24(6), 764–779 (2002)

    Article  Google Scholar 

  26. Hirata, K., Kato, T.: Query by visual example—content based image retrieval. In: International Conference on Extending Database Technology: Advances in Database Technology, pp. 56–71 (1992)

  27. Hu, R., Barnard, M., Collomosse, J.: Gradient field descriptor for sketch based retrieval and localization. In: IEEE International Conference on Image Processing, pp. 1025–1028 (2010)

  28. Hu, R., Collomosse, J.: A performance evaluation of gradient field hog descriptor for sketch based image retrieval. Comput. Vis. Image Underst. 117, 790–806 (2013)

    Article  Google Scholar 

  29. Hu, R., Wang, T., Collomosse, J.: A bag-of-regions approach to sketch based image retrieval. In: IEEE International Conference on Image Processing, pp. 3661–3664 (2011)

  30. Joachims, T.: Optimizing search engines using clickthrough data. In: Proceedings of the ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 133–142 (2002)

  31. Kato, T., Kurita, T., Otsu, N., Kyoji, H.: A sketch retrieval method for full color image database-query by visual example. In: IAPR International Conference on Computer Vision and Applications, pp. 530–533 (1992)

  32. Kendall, M.G.: A new measure of rank correlation. Biometrika 30(1/2), 81–93 (1938)

    Article  Google Scholar 

  33. Knutsson, H.: Representing local structure using tensors. In: Scandinavian Conference on Image Analysis, pp. 244–251 (1989)

  34. Krizhevsky, A., Sutskever, I., Hinton, G.E.: Imagenet classification with deep convolutional neural networks. In: Advances in Neural Information Processing Systems, pp. 1097–1105 (2012)

  35. Lazebnik, S., Schmid, C., Ponce, J.: Beyond bags of features: spatial pyramid matching for recognizing natural scene categories. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 2169–2178 (2006)

  36. Li, Y., Hospedales, T.M., Song, Y.Z., Gong, S.: Fine-grained sketch-based image retrieval by matching deformable part models. In: In British Machine Vision Conference (BMVC) (2014)

  37. Li, Y., Song, Y.Z., Hospedales, T., Gong, S.: Free-hand sketch synthesis with deformable stroke models. Int. J. Comput. Vis. 122(1), 169–190 (2017)

    Article  MathSciNet  Google Scholar 

  38. Lowe, D.G.: Object recognition from local scale-invariant features. In: IEEE International Conference on Computer Vision, 20–25 September, 1999, Kerkyra, Corfu, Greece, Proceedings, vol. 2, pp. 1150–1157 (1999)

  39. Lowe, D.G.: Distinctive image features from scale-invariant keypoints. Int. J. Comput. Vis. 60(2), 91–110 (2004)

    Article  MathSciNet  Google Scholar 

  40. Manjunath, B.S., Salembier, P., Sikora, t.: Introduction to MPEG-7: Multimedia Content Description Interface. Wiley, New York (2002)

  41. Muja, M., Lowe, D.G.: Fast approximate nearest neighbors with automatic algorithm configuration. In: IEEE International Conference on Computer Vision Theory and Applications, pp. 331–340 (2009)

  42. Niblack, W., Barber, R., Equitz, W., Flickner, M., Glasman, E.H., Petkovic, D., Yanker, P., Faloutsos, C., Taubin, G.: The qbic project: querying images by content, using color, texture, and shape. In: Storage and Retrieval for Image and Video Databases (SPIE), pp. 173–187 (1993)

  43. Opelt, A., Pinz, A., Zisserman, A.: Learning an alphabet of shape and appearance for multi-class object detection. Int. J. Comput. Vis. 80(1), 16–44 (2008)

    Article  Google Scholar 

  44. Parui, S., Mittal, A.: Similarity-invariant sketch-based image retrieval in large databases. In: European Conference on Computer Vision, pp. 398–414 (2014)

    Google Scholar 

  45. Rajendran, R.K., Chang, S.F.: Image retrieval with sketches and compositions. In: IEEE International Conference on Multimedia and Expo, pp. 717–720 (2000)

  46. Rui, Y., Huang, T.S., Chang, S.F.: Image retrieval: current techniques, promising directions, and open issues. J. Vis. Commun. Image Represent. 10(1), 39–62 (1999)

    Article  Google Scholar 

  47. Sangkloy, P., Burnell, N., Ham, C., Hays, J.: The sketchy database: Learning to retrieve badly drawn bunnies. In: SIGGRAPH (2016)

  48. Santosh, K., Lamiroy, B., Wendling, L.: DTW-radon-based shape descriptor for pattern recognition. Int. J. Pattern Recogn. Artif. Intell. 27(3) (2013)

    Article  MathSciNet  Google Scholar 

  49. Santosh, K.C., Lamiroy, B., Wendling, L.: Symbol recognition using spatial relations. Pattern Recogn. Lett. 33(3), 331–341 (2012)

    Article  Google Scholar 

  50. Shechtman, E., Irani, M.: Matching local self-similarities across images and videos. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 1–8 (2007)

  51. Sivic, J., Zisserman, A.: Video Google: a text retrieval approach to object matching in videos. In: IEEE International Conference on Computer Vision, pp. 1470–1477 (2003)

  52. Sobel, I., Feldman, G.: An Isotropic 3x3 Image Gradient Operator for Image Processing. In: Pattern Classification and Scene Analysis, pp. 271–272 (1973)

  53. Song, J., Song, Y., Xiang, T., Hospedales, T., Ruan, X.: Deep multi-task attribute-based ranking for fine-grained sketch-based image retrieval. In: British Machine Vision Conference (2016)

  54. Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., Rabinovich, A.: Going deeper with convolutions. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 1–9 (2015)

  55. Tang, X., Wang, X.: Face sketch recognition. IEEE Trans. Circ. Syst. Video Technol. 14(1), 50–57 (2004)

    Article  Google Scholar 

  56. Thayananthan, A., Stenger, B., Torr, P.H.S., Cipolla, R.: Shape context and chamfer matching in cluttered scenes. In: IEEE Conference on Computer Vision and Pattern Recognition (2003)

  57. Wang, J., Song, Y., Leung, T., Rosenberg, C., Wang, J., Philbin, J., Chen, B., Wu, Y.: Learning fine-grained image similarity with deep ranking. In: Proceedings of the 2014 IEEE Conference on Computer Vision and Pattern Recognition, pp. 1386–1393 (2014)

  58. Yu, Q., Liu, F., Song, Y.Z., Xiang, T., Hospedales, T.M., Loy, C.C.: Sketch me that shoe. In: IEEE Conference on Computer Vision and Pattern Recognition (2016)

  59. Yu, Q., Yang, Y., Song, Y., Xiang, T., Hospedales, T.: Sketch-a-net that beats humans. In: British Machine Vision Conference, pp. 7.1–7.12 (2015)

  60. Zobel, J., Moffat, A.: Inverted files for text search engines. ACM Comput. Surv. 38(2), 6.1–6.56 (2006)

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. OSRAM Corporate Technology, Munich, Germany

    Yi Li

  2. Queen Mary University of London, London, UK

    Wenzhao Li

Authors
  1. Yi Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Wenzhao Li
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Yi Li.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Li, Y., Li, W. A survey of sketch-based image retrieval. Machine Vision and Applications 29, 1083–1100 (2018). https://doi.org/10.1007/s00138-018-0953-8

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00138-018-0953-8

Keywords

Search

Navigation