Skip to main content
SpringerLink
Log in

OneSketch: learning high-level shape features from simple sketches

  • Original article
  • Published:
The Visual Computer Aims and scope Submit manuscript

Abstract

Humans use simple sketches to convey complex concepts and abstract ideas in a concise way. Just a few abstract pencil strokes can carry a large amount of semantic information that can be used as meaningful representation for many applications. In this work, we explore the power of simple human strokes denoted to capture high-level 2D shape semantics. For this purpose, we introduce OneSketch, a crowd-sourced dataset of abstract one-line sketches depicting high-level 2D object features. To construct the dataset, we formulate a human sketching task with the goal of differentiating between objects with a single minimal stroke. While humans are rather successful at depicting high-level shape semantics and abstraction, we investigate the ability of deep neural networks to convey such traits. We introduce a neural network which learns meaningful shape features from our OneSketch dataset. Essentially, the model learns sketch-to-shape relations and encodes them in an embedding space which reveals distinctive shape features. We show that our network is applicable for differentiating and retrieving 2D objects using very simple one-stroke sketches with good accuracy.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11

Similar content being viewed by others

References

  1. Bell, S., Bala, K.: Learning visual similarity for product design with convolutional neural networks. ACM Trans. Graph., 34(4), 1–10, 2015

    Article  Google Scholar 

  2. I. Berger, A. Shamir, M. Mahler, E. Carter, and J. Hodgins, Style and abstraction in portrait sketching. ACM Trans. Graph., 32(4), 1–12, 2013

    Article  Google Scholar 

  3. Bhunia, A.K., Yang, Y., Hospedales, T.M., Xiang, T., Song, Y.-Z.: Sketch less for more: on-the-fly fine-grained sketch-based image retrieval. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9779–9788 (2020)

  4. Bui, Tu., Ribeiro, Leonardo, Ponti, Moacir, Collomosse, John: Sketching out the details: Sketch-based image retrieval using convolutional neural networks with multi-stage regression. Computers & Graphics 71, 77–87 (2018)

    Article  Google Scholar 

  5. Cao, X., Zhang, H., Liu, S., Guo, X., Lin, L.: Sym-fish: a symmetry-aware flip invariant sketch histogram shape descriptor. In: 2013 IEEE International Conference on Computer Vision, pp. 313–320 (2013)

  6. Cao, Y., Wang, C., Zhang, L., Zhang, L.: Edgel index for large-scale sketch-based image search. In: CVPR 2011, pp. 761–768 (2011)

  7. Chang, A.X., Funkhouser, T., Guibas, L., Hanrahan, P., Huang, Q., Li, Z., Savarese, S., Savva, M., Song, S., Su, H., Xiao, J., Yi, L., Yu, F.: Shapenet: An information-rich 3d model repository (2015)

  8. Chen, S.-Y., Liu, F.-L., Lai, Y.-K., Rosin, P.L., Li, C., Fu, H., Gao, L.: Deepfaceediting: Deep face generation and editing with disentangled geometry and appearance control. ACM Trans. Graph., 40(4), 1–15, 2021

    Google Scholar 

  9. Chen, S.-Y., Su, W., Gao, L., Xia, S., Fu, H.: Deepfacedrawing: Deep generation of face images from sketches. ACM Trans. Graph., 39(4) (2020) 72

    Article  Google Scholar 

  10. Chopra, S., Hadsell, R., LeCun, Y.: Learning a similarity metric discriminatively, with application to face verification. In: 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR’05), vol. 1, pp. 539–546 (2005)

  11. Devlin, J., Chang, M.-W., Lee, K., Toutanova, K.: BERT: Pre-training of deep bidirectional transformers for language understanding. In: Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pp. 4171–4186, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics (2019)

  12. Eitz, Mathias, Hays, James, Alexa, Marc: How do humans sketch objects? ACM Transactions on graphics (TOG) 31(4), 1–10 (2012)

    Google Scholar 

  13. Eitz, Mathias, Hildebrand, Kristian, Boubekeur, Tamy, Alexa, Marc: Sketch-based image retrieval: benchmark and bag-of-features descriptors. IEEE Trans. Visual Comput. Graph. 17(11), 1624–1636 (2011)

    Article  Google Scholar 

  14. Ghosh, A., Zhang, R., Dokania, P.K., Wang, O., Efros, A.A., Torr, P.H.S., Shechtman, E.: Interactive sketch and fill: Multiclass sketch-to-image translation. In: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) (2019)

  15. Ha, D., Eck, D.: A neural representation of sketch drawings. In: International Conference on Learning Representations (2018)

  16. Hadsell, R., Chopra, S., LeCun, Y.: Dimensionality reduction by learning an invariant mapping. In: 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR’06), vol. 2, pp. 1735–1742 (2006)

  17. Rui, Hu., Collomosse, John: A performance evaluation of gradient field hog descriptor for sketch based image retrieval. Comput. Vis. Image Underst. 117(7), 790–806 (July 2013)

    Article  Google Scholar 

  18. Jiao, J., Cao, Y., Lau, M., Lau, R.: Tactile sketch saliency. In: Proceedings of the 28th ACM International Conference on Multimedia, MM’20. Association for Computing Machinery, New York, NY, USA, pp. 3072–3080 (2020)

  19. Kato, T., Kurita, T., Otsu, N., Hirata, K.: A sketch retrieval method for full color image database-query by visual example. In: [1992] Proceedings of the 11th IAPR International Conference on Pattern Recognition, pp. 530–533 (1992)

  20. Lamb, A., Ozair, S., Verma, V., Ha, D.: Sketchtransfer: A new dataset for exploring detail-invariance and the abstractions learned by deep networks. In: Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 963–972 (2020)

  21. Li, Y., Su, H., Qi, C.R., Fish, N., Cohen-Or, D., Guibas, L.J.: Joint embeddings of shapes and images via cnn image purification. ACM Trans. Graph. 34(6), 1–12 (2015)

    Article  Google Scholar 

  22. Li, Y., Hospedales, T., Song, Y.-Z., Gong, S.: Fine-grained sketch-based image retrieval by matching deformable part models. In: BMVC 2014—Proceedings of the British Machine Vision Conference 2014 (2014)

  23. Limpaecher, A., Feltman, N., Treuille, A., Cohen, M.: Real-time drawing assistance through crowdsourcing. ACM Trans. Graph., 32(4), 2013, 1–8

    Article  Google Scholar 

  24. Lin, H., Fu, Y., Xue, X., Jiang, Y.-G.: Sketch-bert: learning sketch bidirectional encoder representation from transformers by self-supervised learning of sketch gestalt. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6758–6767 (2020)

  25. Lin, T.-Y., Cui, Y., Belongie, S., Hays, J.: Learning deep representations for ground-to-aerial geolocalization. In: 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 5007–5015 (2015)

  26. Liu, F., Deng, X., Lai, Y.-K., Liu, Y.-J., Ma, C., Wang, H.: Sketchgan: Joint sketch completion and recognition with generative adversarial network. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2019)

  27. Muhammad, U.R., Yang, Y., Song, Y.-Z., Xiang, T., Hospedales, T.M.: Learning deep sketch abstraction. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 8014–8023 (2018)

  28. Muhammad, U.R., Yang, Y., Hospedales, T.M., Xiang, T., Song, Y.-Z..: Goal-driven sequential data abstraction. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 71–80 (2019)

  29. Pang, K., Song, Y.-Z., Xiang, T., Hospedales, T.M.: Cross-domain generative learning for fine-grained sketch-based image retrieval. In: BMVC, pp. 1–12 (2017)

  30. Ribeiro, L.S.F., Bui, T., Collomosse, J., Ponti, M.: Sketchformer: transformer-based representation for sketched structure. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14153–14162 (2020)

  31. Saavedra, J.M., Barrios, J.M.: Sketch based image retrieval using learned keyshapes (lks). In: Proceedings of the British Machine Vision Conference (BMVC), pp. 164.1–164.11 (2015)

  32. Sain, A., Bhunia, A., Yang, Y., Xiang, T., Song, Y.-Z.: Cross-modal hierarchical modelling for fine-grained sketch based image retrieval (2020)

  33. Sangkloy, P., Burnell, N., Ham, C., Hays, J.: The sketchy database: learning to retrieve badly drawn bunnies. ACM Trans. Graph. 35(4), 1–12 (2016)

    Article  Google Scholar 

  34. Sarvadevabhatla, R.K., Venkatesh B.R.: Eye of the dragon: Exploring discriminatively minimalist sketch-based abstractions for object categories. In: Proceedings of the 23rd ACM International Conference on Multimedia, MM’15. Association for Computing Machinery, New York, NY, USA, pp. 271–280 (2015)

  35. Shrivastava, A., Malisiewicz, T., Gupta, A., Efros, A.A.: Data-driven visual similarity for cross-domain image matching. In: Proceedings of the 2011 SIGGRAPH Asia Conference. Association for Computing Machinery, New York, NY, USA (2011)

  36. Song, J., Yu, Q., Song, Y.-Z., Xiang, T., Hospedales, T.M.: Deep spatial-semantic attention for fine-grained sketch-based image retrieval. In: Proceedings of the IEEE international conference on computer vision, pp. 5551–5560 (2017)

  37. Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J., Wojna, Z.: Rethinking the inception architecture for computer vision. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2818–2826 (2016)

  38. Van der Maaten, L., Hinton, G.: Visualizing data using t-sne. Journal of Machine Learning Research, 9(11), 2008 2579-2605

    MATH  Google Scholar 

  39. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, Ł, Polosukhin, I.: Attention is all you need. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems, vol. 30. Curran Associates, New York (2017)

    Google Scholar 

  40. Wang, F., Kang, L., Li, Y.: Sketch-based 3d shape retrieval using convolutional neural networks. In: 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1875–1883 (2015)

  41. Wang, J., Song, Y., Leung, T., Rosenberg, C., Wang, J., Philbin, J., Chen, B., Wu, Y.: Learning fine-grained image similarity with deep ranking. In: 2014 IEEE Conference on Computer Vision and Pattern Recognition, pp. 1386–1393 (2014)

  42. Xuemiao, Xu., Xie, Minshan, Miao, Peiqi, Wei, Qu., Xiao, Wenpeng, Zhang, Huaidong, Liu, Xueting, Wong, Tien-Tsin.: Perceptual-aware sketch simplification based on integrated vgg layers. IEEE Trans. Vis. Comput. Graph. 27(1), 178–189 (2021)

    Article  Google Scholar 

  43. Yu, Q., Liu, F., Song, Y.-Z., Xiang, T., Hospedales, T.M., Loy, C.-C.: Sketch me that shoe. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 799–807 (2016)

Download references

Author information

Authors and Affiliations

  1. Ben-Gurion University of the Negev, Be’er Sheva, Israel

    Eyal Reisfeld & Andrei Sharf

Authors
  1. Eyal Reisfeld
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Andrei Sharf
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Andrei Sharf.

Ethics declarations

Conflict of interest

The authors declare that they have no conflict of interest.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Reisfeld, E., Sharf, A. OneSketch: learning high-level shape features from simple sketches. Vis Comput 39, 2811–2822 (2023). https://doi.org/10.1007/s00371-022-02494-2

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00371-022-02494-2

Keywords

Search

Navigation