research-article

Context-based sketch classification

Authors:

Chiew-Lan TaiAuthors Info & Claims

Expressive '18: Proceedings of the Joint Symposium on Computational Aesthetics and Sketch-Based Interfaces and Modeling and Non-Photorealistic Animation and Rendering

Article No.: 3, Pages 1 - 10

https://doi.org/10.1145/3229147.3229154

Published: 17 August 2018 Publication History

Abstract

We present a novel context-based sketch classification framework using relations extracted from scene images. Most of existing methods perform sketch classification by considering individually sketched objects and often fail to identify their correct categories, due to the highly abstract nature of sketches. For a sketched scene containing multiple objects, we propose to classify a sketched object by considering its surrounding context in the scene, which provides vital cues for alleviating its recognition ambiguity. We learn such context knowledge from a database of scene images by summarizing the inter-object relations therein, such as co-occurrence, relative positions and sizes. We show that the context information can be used for both incremental sketch classification and sketch co-classification. Our method outperforms a state-of-the-art single-object classification method, evaluated on a new dataset of sketched scenes.

Supplementary Material

ZIP File (03-008.zip)

Supplemental files.

Download
32.54 MB

References

[1]

Christine Alvarado and Randall Davis. 2004. SketchREAD: A Multi-domain Sketch Recognition Engine. In Proc. ACM UIST. ACM, 10.

Digital Library

[2]

Relja Arandjelović and Tevfk Metin Sezgin. 2011. Sketch recognition by fusion of temporal and image-based features. Pattern Recognition 44, 6 (2011), 1225--1234.

Digital Library

[3]

T Bui, L Ribeiro, M Ponti, and John Collomosse. 2017. Compact descriptors for sketch-based image retrieval using a triplet loss convolutional neural network. Computer Vision and Image Understanding 164 (2017), 27--37.

[4]

Xiaochun Cao, Hua Zhang, Si Liu, Xiaojie Guo, and Liang Lin. 2013. SYM-FISH: A Symmetry-Aware Flip Invariant Sketch Histogram Shape Descriptor. In Proc. IEEE ICCV.

Digital Library

[5]

Tao Chen, Ming-Ming Cheng, Ping Tan, Ariel Shamir, and Shi-Min Hu. 2009. Sketch2Photo: Internet Image Montage. ACM TOG 28, 5, Article 124 (Dec. 2009), 10 pages.

Digital Library

[6]

N. Donmez and K. Singh. 2012. Concepture: A Regular Language Based Framework for Recognizing Gestures with Varying and Repetitive Patterns. In Proc. SBIM. Eurographics Association, 9.

Digital Library

[7]

Mathias Eitz, James Hays, and Marc Alexa. 2012. How Do Humans Sketch Objects? ACM TOG 31, 4, Article 44 (July 2012), 10 pages.

Digital Library

[8]

M. Eitz, K. Hildebrand, T. Boubekeur, and M. Alexa. 2011. Sketch-Based Image Retrieval: Benchmark and Bag-of-Features Descriptors. IEEE TVCG 17, 11 (Nov 2011), 1624--1636.

Digital Library

[9]

P. F. Felzenszwalb, R. B. Girshick, D. McAllester, and D. Ramanan. 2010. Object Detection with Discriminatively Trained Part-Based Models. IEEE TPAMI 32, 9 (Sept 2010), 1627--1645.

Digital Library

[10]

Matthew Fisher and Pat Hanrahan. 2010. Context-based Search for 3D Models. ACM TOG 29, 6, Article 182 (Dec. 2010), 10 pages.

Digital Library

[11]

Carolina Galleguillos and Serge Belongie. 2010. Context Based Object Categorization: A Critical Survey. CVIU 114, 6 (June 2010), 712--722.

Digital Library

[12]

David Ha and Douglas Eck. 2017. A Neural Representation of Sketch Drawings. CoRR abs/1704.03477 (2017).

[13]

Rui Hu and John Collomosse. 2013. A performance evaluation of gradient feld hog descriptor for sketch based image retrieval. Computer Vision and Image Understanding 117, 7 (2013), 790--806.

Digital Library

[14]

Zhe Huang, Hongbo Fu, and Rynson W. H. Lau. 2014. Data-driven Segmentation and Labeling of Freehand Sketches. ACM TOG 33, 6, Article 175 (Nov. 2014), 10 pages.

Digital Library

[15]

Levent Burak Kara and Thomas F. Stahovich. 2004. Hierarchical Parsing and Recognition of Hand-sketched Diagrams. In Proc. ACM UIST. ACM, 13--22.

Digital Library

[16]

R. Krishna, Y. Zhu, O. Groth, J. Johnson, K. Hata, J. Kravitz, S. Chen, Y. Kalantidis, L.-J. Li, D. A. Shamma, M. S. Bernstein, and Li Fei-Fei. 2017. Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations. IJCV 123, 1 (01 May 2017), 32--73.

Digital Library

[17]

Joseph J. LaViola, Jr. and Robert C. Zeleznik. 2004. MathPad2: A System for the Creation and Exploration of Mathematical Sketches. ACM TOG 23, 3 (Aug. 2004), 432--440.

Digital Library

[18]

Bo Li, Yijuan Lu, and R. Fares. 2013. Semantic sketch-based 3D model retrieval. In ICMEW. 1--4.

[19]

L. J. Li, R. Socher, and L. Fei-Fei. 2009. Towards total scene understanding: Classifcation, annotation and segmentation in an automatic framework. In Proc. IEEE CVPR. 2036--2043.

[20]

Yi Li, Timothy M. Hospedales, Yi-Zhe Song, and Shaogang Gong. 2015. Free-hand sketch recognition by multi-kernel feature learning. CVIU 137 (2015), 1 -- 11.

Digital Library

[21]

Tong Lu, Chiew-Lan Tai, Feng Su, and Shijie Cai. 2005. A New Recognition Model for Electronic Architectural Drawings. CAD 37, 10 (2005), 1053 -- 1069.

Digital Library

[22]

Zhaoliang Lun, Changqing Zou, Haibin Huang, Evangelos Kalogerakis, Ping Tan, Marie-Paule Cani, and Hao Zhang. 2017. Learning to group discrete graphical patterns. ACM Transactions on Graphics (TOG) 36, 6 (2017), 225.

Digital Library

[23]

Tomasz Malisiewicz and Alyosha Efros. 2009. Beyond categories: The visual memex model for reasoning about object relationships. In Advances in neural information processing systems. 1222--1230.

Digital Library

[24]

R. Mottaghi, X. Chen, X. Liu, N. G. Cho, S. W. Lee, S. Fidler, R. Urtasun, and A. Yuille. 2014. The Role of Context for Object Detection and Semantic Segmentation in the Wild. In Proc. IEEE CVPR. 891--898.

Digital Library

[25]

Tom Y. Ouyang and Randall Davis. 2011. ChemInk: A Natural Real-time Recognition System for Chemical Drawings. In Proc. ACM IUI. ACM, 10.

Digital Library

[26]

Brandon Paulson and Tracy Hammond. 2008. PaleoSketch: Accurate Primitive Sketch Recognition and Beautifcation. In Proc. ACM IUI. ACM, 10.

Digital Library

[27]

Tiziano Portenier, Qiyang Hu, Paolo Favaro, and Matthias Zwicker. 2017. SmartSketcher: Sketch-based Image Retrieval with Dynamic Semantic Re-ranking. In Proc. SBIM. ACM, Article 1, 12 pages.

Digital Library

[28]

Yonggang Qi, Yi-Zhe Song, Honggang Zhang, and Jun Liu. 2016. Sketch-based image retrieval via siamese convolutional neural network. In Image Processing (ICIP), 2016 IEEE International Conference on. IEEE, 2460--2464.

[29]

X. Qian, X. Tan, Y. Zhang, R. Hong, and M. Wang. 2016. Enhancing Sketch-Based Image Retrieval by Re-Ranking and Relevance Feedback. IEEE TIP 25, 1 (Jan 2016), 195--208.

[30]

A. Rabinovich, A. Vedaldi, C. Galleguillos, E. Wiewiora, and S. Belongie. 2007. Objects in Context. In Proc. IEEE ICCV. 1--8.

[31]

Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. 2015. ImageNet Large Scale Visual Recognition Challenge. IJCV 115, 3 (01 Dec 2015), 211--252.

Digital Library

[32]

Patsorn Sangkloy, Nathan Burnell, Cusuh Ham, and James Hays. 2016. The Sketchy Database: Learning to Retrieve Badly Drawn Bunnies. ACM TOG 35, 4, Article 119 (July 2016), 12 pages.

Digital Library

[33]

Rosália G. Schneider and Tinne Tuytelaars. 2014. Sketch Classifcation and Classifcation-driven Analysis Using Fisher Vectors. ACM TOG 33, 6, Article 174 (Nov. 2014), 9 pages.

Digital Library

[34]

Tevfk Metin Sezgin and Randall Davis. 2005. HMM-based efcient sketch recognition. In Proceedings of the 10th international conference on Intelligent user interfaces. ACM, 281--283.

Digital Library

[35]

Tevfk Metin Sezgin and Randall Davis. 2008. Sketch recognition in interspersed drawings using time-based graphical models. Computers & Graphics 32, 5 (2008), 500--510.

Digital Library

[36]

Tevfk Metin Sezgin, Thomas Stahovich, and Randall Davis. 2001. Sketch Based Interfaces: Early Processing for Sketch Understanding. In Proceedings of Workshop on Perceptive User Interfaces. ACM, 8.

Digital Library

[37]

HyoJong Shin and Takeo Igarashi. 2007. Magic Canvas: Interactive Design of a 3-D Scene Prototype from Freehand Sketches. In Proceedings of Graphics Interface. ACM, 63--70.

Digital Library

[38]

Oana Sidi, Oliver van Kaick, Yanir Kleiman, Hao Zhang, and Daniel Cohen-Or. 2011. Unsupervised Co-segmentation of a Set of Shapes via Descriptor-space Spectral Clustering. ACM TOG 30, 6, Article 126 (Dec. 2011), 10 pages.

Digital Library

[39]

J. Sivic and A. Zisserman. 2003. Video Google: a Text Retrieval Approach to Object Matching in Videos. In Proc. IEEE ICCV.

Digital Library

[40]

Zhenbang Sun, Changhu Wang, Liqing Zhang, and Lei Zhang. 2012. Free hand-drawn sketch segmentation. In European Conference on Computer Vision. Springer, 626--639.

Digital Library

[41]

Ivan E. Sutherland. 1964. Sketchpad: A Man-machine Graphical Communication System. In DAC. ACM, 6.329--6.346.

Digital Library

[42]

C. Szegedy, W. Liu, Y. Jia, P. Sermanet, S. Reed, D. Anguelov, D. Erhan, V. Vanhoucke, and A. Rabinovich. 2015. Going Deeper With Convolutions. In Proc. IEEE CVPR.

[43]

J. Xiao, J. Hays, K. A. Ehinger, A. Oliva, and A. Torralba. 2010. SUN database: Largescale scene recognition from abbey to zoo. In Proc. IEEE CVPR.

[44]

Kun Xu, Kang Chen, Hongbo Fu, Wei-Lun Sun, and Shi-Min Hu. 2013. Sketch2Scene: Sketch-based Co-retrieval and Co-placement of 3D Models. ACM TOG 32, 4, Article 123 (July 2013), 15 pages.

Digital Library

[45]

Kemal Tugrul Yesilbek and T Metin Sezgin. 2017. Sketch recognition with few examples. Computers & Graphics 69 (2017), 80--91.

Digital Library

[46]

Qian Yu, Feng Liu, Yi-Zhe Song, Tao Xiang, Timothy M. Hospedales, and Chen-Change Loy. 2016. Sketch Me That Shoe. In Proc. IEEE CVPR.

[47]

Qian Yu, Yongxin Yang, Feng Liu, Yi-Zhe Song, Tao Xiang, and Timothy M. Hospedales. 2017. Sketch-a-Net: A Deep Neural Network that Beats Humans. IJCV 122, 3 (01 May 2017), 411--425.

Digital Library

[48]

H. Zhang, S. Liu, C. Zhang, W. Ren, R. Wang, and X. Cao. 2016. SketchNet: Sketch Classifcation with Web Images. In Proc. IEEE CVPR.

Cited By

Ge CSun HSong YMa ZLiao J(2022)Exploring Local Detail Perception for Scene Sketch Semantic SegmentationIEEE Transactions on Image Processing10.1109/TIP.2022.314251131(1447-1461)Online publication date: 2022
https://doi.org/10.1109/TIP.2022.3142511
Jain GChopra SChopra SParihar A(2022)Attention-Net: An Ensemble Sketch Recognition Approach Using Vector ImagesIEEE Transactions on Cognitive and Developmental Systems10.1109/TCDS.2020.302305514:1(136-145)Online publication date: Mar-2022
https://doi.org/10.1109/TCDS.2020.3023055
Chen SZhang JZhao YRosin PLai YGao L(2022)A review of image and video colorization: From analogies to deep learningVisual Informatics10.1016/j.visinf.2022.05.0036:3(51-68)Online publication date: Sep-2022
https://doi.org/10.1016/j.visinf.2022.05.003
Show More Cited By

Index Terms

Context-based sketch classification
1. Computing methodologies
  1. Computer graphics
    1. Shape modeling
      1. Shape analysis
2. Human-centered computing
  1. Human computer interaction (HCI)

Recommendations

Automatic understanding of sketch maps using context-aware classification

We present the first comprehensive system for offline classifying sketch map objects.We created a database of labeled sketch maps for training and evaluation purposes.Context-awareness improves the classification of sketch map objects greatly. Sketching ...
MobileNet-SA: Lightweight CNN with Self Attention for Sketch Classification
Image and Video Technology
Abstract
Sketch classification plays a crucial role across diverse domains, including image retrieval, artistic style analysis, and content-based image retrieval. While CNNs have demonstrated remarkable success in various image-related tasks, the ...
Sketch-based modeling
EG '16: Proceedings of the 37th Annual Conference of the European Association for Computer Graphics: Tutorials

Sketching is one of the most natural ways to exchange ideas. It has been used by human beings since prehistory. Research has shown that human beings have an inherent ability to understand sketches. This is why sketch-based interfaces for 3D modeling are ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

Expressive '18: Proceedings of the Joint Symposium on Computational Aesthetics and Sketch-Based Interfaces and Modeling and Non-Photorealistic Animation and Rendering

August 2018

200 pages

ISBN:9781450358927

DOI:10.1145/3229147

General Chairs:
Brian Wyvill
University of Victoria, Canada
,
Hongbo Fu
City University of Hong Kong, Hong Kong

Copyright © 2018 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGGRAPH: ACM Special Interest Group on Computer Graphics and Interactive Techniques

In-Cooperation

EUROGRAPHICS: The European Association for Computer Graphics

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 17 August 2018

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

Expressive '18

Sponsor:

SIGGRAPH

Expressive '18: Joint Symposium on Computational Aesthetics and Sketch Based Interfaces and Modeling and Non-Photorealistic Animation and Rendering

August 17 - 19, 2018

British Columbia, Victoria, Canada

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

8
Total Citations
View Citations
439
Total Downloads

Downloads (Last 12 months)14
Downloads (Last 6 weeks)0

Reflects downloads up to 28 Feb 2025

Other Metrics

View Author Metrics

Citations

Cited By

Ge CSun HSong YMa ZLiao J(2022)Exploring Local Detail Perception for Scene Sketch Semantic SegmentationIEEE Transactions on Image Processing10.1109/TIP.2022.314251131(1447-1461)Online publication date: 2022
https://doi.org/10.1109/TIP.2022.3142511
Jain GChopra SChopra SParihar A(2022)Attention-Net: An Ensemble Sketch Recognition Approach Using Vector ImagesIEEE Transactions on Cognitive and Developmental Systems10.1109/TCDS.2020.302305514:1(136-145)Online publication date: Mar-2022
https://doi.org/10.1109/TCDS.2020.3023055
Chen SZhang JZhao YRosin PLai YGao L(2022)A review of image and video colorization: From analogies to deep learningVisual Informatics10.1016/j.visinf.2022.05.0036:3(51-68)Online publication date: Sep-2022
https://doi.org/10.1016/j.visinf.2022.05.003
Li LZou CZheng YSu QFu HTai C(2021) Sketch-R2CNN : An RNN-Rasterization-CNN Architecture for Vector Sketch Recognition IEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2020.298762627:9(3745-3754)Online publication date: 1-Sep-2021
https://doi.org/10.1109/TVCG.2020.2987626
Arnia FAisyah SSaddami KMunadi K(2021)The Performances of Pre-trained Convolutional Neural Networks in Clothing Sketch Classification2021 18th International Conference on Electrical Engineering/Electronics, Computer, Telecommunications and Information Technology (ECTI-CON)10.1109/ECTI-CON51831.2021.9454854(107-111)Online publication date: 19-May-2021
https://doi.org/10.1109/ECTI-CON51831.2021.9454854
Yang ZDai QZhang J(2021)Visual perception driven collage synthesisComputational Visual Media10.1007/s41095-021-0226-88:1(79-91)Online publication date: 27-Oct-2021
https://doi.org/10.1007/s41095-021-0226-8
Parihar AJain GChopra SChopra S(2020)SketchFormer: transformer-based approach for sketch recognition using vector imagesMultimedia Tools and Applications10.1007/s11042-020-09837-yOnline publication date: 7-Nov-2020
https://doi.org/10.1007/s11042-020-09837-y
Pan CHuang JGong JChen C(2020)Teach machine to learn: hand-drawn multi-symbol sketch recognition in one-shotApplied Intelligence10.1007/s10489-019-01607-0Online publication date: 2-Mar-2020
https://doi.org/10.1007/s10489-019-01607-0

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten