Skip to main content
SpringerLink
Log in

SketchFormer: transformer-based approach for sketch recognition using vector images

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

Sketches have been employed since the ancient era of cave paintings for simple illustrations to represent real-world entities and communication. The abstract nature and varied artistic styling make automatic recognition of these drawings more challenging than other areas of image classification. Moreover, the representation of sketches as a sequence of strokes instead of raster images introduces them at the correct abstract level. However, dealing with images as a sequence of small information makes it challenging. In this paper, we propose a Transformer-based network, dubbed as AttentiveNet, for sketch recognition. This architecture incorporates ordinal information to perform the classification task in real-time through vector images. We employ the proposed model to isolate the discriminating strokes of each doodle using the attention mechanism of Transformers and perform an in-depth qualitative analysis of the isolated strokes for classification of the sketch. Experimental evaluation validates that the proposed network performs favorably against state-of-the-art techniques.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7

Similar content being viewed by others

Notes

  1. Official TensorFlow implementation of RNN for QuickDraw

References

  1. Abualigah L (2020) Multi-verse optimizer algorithm: a comprehensive survey of its results, variants, and applications. Neural Comput Applic pp 1–21

  2. Abualigah L, Diabat A (2020) A novel hybrid antlion optimization algorithm for multi-objective task scheduling problems in cloud computing environments. Clust Comput pp 1–19

  3. Abualigah LM, Khader AT (2017) Unsupervised text feature selection technique based on hybrid particle swarm optimization algorithm with genetic operators for the text clustering. J Supercomput 73(11):4773–4795

    Article  Google Scholar 

  4. Abualigah LM, Khader AT, Hanandeh ES (2018) A combination of objective functions and hybrid krill herd algorithm for text document clustering analysis. Eng Appl Artif Intell 73:111–125

    Article  Google Scholar 

  5. Abualigah LM, Khader AT, Hanandeh ES (2018) Hybrid clustering analysis using improved krill herd algorithm. Appl Intell 48(11):4047–4071. https://doi.org/10.1007/s10489-018-1190-6

    Article  Google Scholar 

  6. Abualigah LM, Khader AT, Hanandeh ES (2018) A new feature selection method to improve the document clustering using particle swarm optimization algorithm. J Comput Sci 25:456–466

    Article  Google Scholar 

  7. Abualigah LMQ (2018) Feature selection and enhanced krill herd algorithm for text document clustering, 1st edn, Springer Publishing Company, Incorporated

  8. Arandjelović R, Sezgin TM (2011) Sketch recognition by fusion of temporal and image-based features. Pattern Recogn 44(6):1225–1234

    Article  Google Scholar 

  9. Bahdanau D, Cho K, Bengio Y (2014) Neural machine translation by jointly learning to align and translate. arXiv:1409.0473

  10. Dehghani M, Gouws S, Vinyals O, Uszkoreit J, Kaiser Ł (2018) Universal transformers. arXiv:1807.03819

  11. Eitz M, Hays J, Alexa M (2012) How do humans sketch objects?. ACM Trans Graph 31(4):44–1

    Google Scholar 

  12. Eitz M, Richter R, Boubekeur T, Hildebrand K, Alexa M (2012) Sketch-based shape retrieval. ACM Trans Graph (TOG) 31(4):31

    Google Scholar 

  13. Graves A (2013) Generating sequences with recurrent neural networks. arXiv:1308.0850

  14. Ha D, Eck D (2017) A neural representation of sketch drawings. arXiv:1704.03477

  15. He K, Zhang X, Ren S, Sun J (2016) Deep residual learning for image recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 770–778

  16. Huang Z, Fu H, Lau RW (2014) Data-driven segmentation and labeling of freehand sketches. ACM Trans Graph (TOG) 33(6):175

    Article  Google Scholar 

  17. Krizhevsky A, Sutskever I, Hinton GE (2012) Imagenet classification with deep convolutional neural networks. In: Advances in neural information processing systems, pp 1097–1105

  18. LaViola JJ Jr, Zeleznik RC (2004) Mathpad 2: a system for the creation and exploration of mathematical sketches. ACM Trans Graph (TOG) 23 (3):432–440

    Article  Google Scholar 

  19. Li K, Pang K, Song J, Song YZ, Xiang T, Hospedales TM, Zhang H (2018) Universal sketch perceptual grouping. In: Proceedings of the european conference on computer vision (ECCV), pp 582–597

  20. Li L, Fu H, Tai CL (2018) Fast sketch segmentation and labeling with deep learning. IEEE Comput Graph Appl 39(2):38–51

    Article  Google Scholar 

  21. Li Y, Hospedales TM, Song YZ, Gong S (2015) Free-hand sketch recognition by multi-kernel feature learning. Comput Vis Image Underst 137:1–11

    Article  Google Scholar 

  22. Liu P, Yu H, Cang S (2019) Adaptive neural network tracking control for underactuated systems with matched and mismatched disturbances. Nonlinear Dynam 98(2):1447–1464

    Article  Google Scholar 

  23. Lu T, Tai CL, Su F, Cai S (2005) A new recognition model for electronic architectural drawings. Comput Aided Des 37(10):1053–1069

    Article  Google Scholar 

  24. Ouyang TY, Davis R (2011) Chemink: a natural real-time recognition system for chemical drawings. In: Proceedings of the 16th international conference on Intelligent user interfaces, ACM , pp 267–276

  25. Sangkloy P, Burnell N, Ham C, Hays J (2016) The sketchy database: learning to retrieve badly drawn bunnies. ACM Trans Graph (TOG) 35 (4):119

    Article  Google Scholar 

  26. Sarvadevabhatla RK, Babu RV (2015) Freehand sketch recognition using deep features. arXiv:1502.00254

  27. Sarvadevabhatla RK, Surya S, Mittal T, Babu RV (2018) Game of sketches: deep recurrent models of pictionary-style word guessing. In: Thirty-second AAAI conference on artificial intelligence

  28. Schneider RG, Tuytelaars T (2014) Sketch classification and classification-driven analysis using fisher vectors. ACM Trans Graph (TOG) 33(6):174

    Article  Google Scholar 

  29. Seddati O, Dupont S, Mahmoudi S (2015) Deepsketch: deep convolutional neural networks for sketch recognition and similarity search. In: 2015 13th international workshop on content-based multimedia indexing (CBMI), IEEE, pp 1–6

  30. Seddati O, Dupont S, Mahmoudi S (2017) Deepsketch 3. Multimed Tools Appl 76(21):22,333–22,359

    Article  Google Scholar 

  31. Sert M, Boyacı E (2019) Sketch recognition using transfer learning. Multimed Tools Appl 78(12):17,095–17,112

    Article  Google Scholar 

  32. Sezgin TM, Davis R (2008) Sketch recognition in interspersed drawings using time-based graphical models. Comput Graph 32(5):500–510

    Article  Google Scholar 

  33. Song J, Pang K, Song YZ, Xiang T, Hospedales TM (2018) Learning to sketch with shortcut cycle consistency. In: Proceedings of the IEEE conference on computer vision and pattern recognition , pp 801–810

  34. Sun L, Zhao C, Yan Z, Liu P, Duckett T, Stolkin R (2018) A novel weakly-supervised approach for rgb-d-based nuclear waste object detection. IEEE Sensors J 19(9):3487–3500

    Article  Google Scholar 

  35. Sun Z, Wang C, Zhang L, Zhang L (2012) Free hand-drawn sketch segmentation. In: European conference on computer vision. Springer, New York, pp 626–639

  36. Sutherland IE (1964) Sketchpad a man-machine graphical communication system. Simulation 2(5) R–3

  37. Szegedy C, Liu W, Jia Y, Sermanet P, Reed S, Anguelov D, Erhan D, Vanhoucke V, Rabinovich A (2015) Going deeper with convolutions. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1–9

  38. Tang Z, Yu H, Lu C, Liu P, Jin X (2019) Single-trial classification of different movements on one arm based on erd/ers and corticomuscular coherence. IEEE Access 7:128,185–128,197

    Article  Google Scholar 

  39. Tang ZC, Li C, Wu JF, Liu PC, Cheng SW (2019) Classification of eeg-based single-trial motor imagery tasks using a b-csp method for bci. Front Inform Technol Electron Eng 20(8):1087–1098

    Article  Google Scholar 

  40. Vaswani A, Shazeer N, Parmar N, Uszkoreit J, Jones L, Gomez AN, Kaiser Ł, Polosukhin I (2017) Attention is all you need. In: Advances in neural information processing systems, pp 5998–6008

  41. Wang F, Kang L, Li Y (2015) Sketch-based 3d shape retrieval using convolutional neural networks. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 1875–1883

  42. Wang X, Chen X, Zha Z (2018) Sketchpointnet: a compact network for robust sketch recognition. In: 2018 25th IEEE international conference on image processing (ICIP), IEEE, pp 2994–2998

  43. Xu P, Huang Y, Yuan T, Pang K, Song YZ, Xiang T, Hospedales TM, Ma Z, Guo J (2018) Sketchmate: deep hashing for million-scale human sketch retrieval. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 8090–8098

  44. Yang Y, Hospedales TM (2015) Deep neural networks for sketch recognition. arXiv:1501.07873 1(2), 3

  45. Yanık E, Sezgin TM (2015) Active learning for sketch recognition. Comput Graph 52:93–105

    Article  Google Scholar 

  46. Yu Q, Yang Y, Liu F, Song YZ, Xiang T, Hospedales TM (2017) Sketch-a-net: a deep neural network that beats humans. Int J Comput Vision 122(3):411–425

    Article  MathSciNet  Google Scholar 

  47. Zhang H, Liu S, Zhang C, Ren W, Wang R, Cao X (2016) Sketchnet: sketch classification with web images. In: Proceedings of the IEEE conference on computer vision and pattern recognition , pp 1105–1113

  48. Zhang J, Chen Y, Li L, Fu H, Tai CL (2018) Context-based sketch classification. In: Proceedings of the joint symposium on computational aesthetics and sketch-based interfaces and modeling and non-photorealistic animation and rendering, ACM, p 3

  49. Zhang X, Huang Y, Zou Q, Pei Y, Zhang R, Wang S (2019) A hybrid convolutional neural network for sketch recognition. Pattern Recognition Letters

  50. Zhao P, Liu Y, Lu Y, Xu B (2019) A sketch recognition method based on transfer deep learning with the fusion of multi-granular sketches. Multimed Tools Appl 78(24):35,179–35,193

    Article  Google Scholar 

  51. Zou C, Yu Q, Du R, Mo H, Song YZ, Xiang T, Gao C, Chen B, Zhang H (2018) Sketchyscene: richly-annotated scene sketches. In: Proceedings of the european conference on computer vision (ECCV), pp 421–436

Download references

Author information

Authors and Affiliations

  1. Machine Learning Research Laboratory, Department of Computer Science and Engineering, Delhi Technological University, New Delhi, 110042, India

    Anil Singh Parihar, Gaurav Jain, Shivang Chopra & Suransh Chopra

Authors
  1. Anil Singh Parihar
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Gaurav Jain
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Shivang Chopra
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Suransh Chopra
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Anil Singh Parihar.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Parihar, A.S., Jain, G., Chopra, S. et al. SketchFormer: transformer-based approach for sketch recognition using vector images. Multimed Tools Appl 80, 9075–9091 (2021). https://doi.org/10.1007/s11042-020-09837-y

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-020-09837-y

Keywords

Search

Navigation