Abstract
Sketching provides the most natural way to provide a visual search query for visual object search. However, how to draw 3D sketches in a three-dimensional space and how to use a hand-drawn 3D sketch to search similar 3D models are not only interesting and novel, but also challenging research topics. In this paper, we try to answer them by initiating a novel study on 3D sketching and build a 3D sketching system which allows users to freely draw 3D sketches in the air and demonstrate its promising potentials in related applications such as collecting 3D sketch data and conducting 3D sketch-based 3D model retrieval. By utilizing the 3D sketching system, we collect a 3D sketch dataset, build a 3D sketch-based 3D model retrieval benchmark, and organize a Eurographics Shape Retrieval Contest (SHREC) track on 3D sketch-based shape retrieval based on the benchmark. We investigate 3D sketch and model matching problems and propose a novel 3D sketch-based model retrieval algorithm CNN-SBR based on Convolutional Neural Networks (CNNs) and achieve the best performance in the SHREC track. We wish that the 3D sketching system, the 3D sketch-based model retrieval benchmark, and the proposed 3D sketch-based model retrieval algorithm CNN-SBR will further promote sketch-based shape retrieval and its applications. We have made all of these publicly available on the project homepage: http://orca.st.usm.edu/~bli/SBR16/project.html.
Similar content being viewed by others
References
Ankerst M, Kastenmüller G, Kriegel H, Seidl T (1999) 3D shape histograms for similarity search and classification in spatial databases. In: Advances in spatial databases, 6th international symposium, SSD’99, Hong Kong, China, July 20-23, 1999, Proceedings, pp 207–226
Araújo C, Cabiddu D, Attene M, Livesu M, Vining N, Sheffer A (2019) Surface2Volume: surface segmentation conforming assemblable volumetric partition. ACM Trans Graph 38(4):80,1–80,16
Aurenhammer F (1991) Voronoi diagrams - a survey of a fundamental geometric data structure. ACM Comput Surv 23(3):345–405
Bae S, Balakrishnan R, Singh K (2008) Ilovesketch: as-natural-as-possible sketching system for creating 3D curve model. In: UIST. ACM, pp 151–160
Beecks C, Grass A (2018) Efficient point-based pattern search in 3D motion capture databases. In: Younas M, Disso JP (eds) 6th IEEE international conference on future internet of things and cloud, FiCloud 2018, Barcelona, Spain, August 6-8, 2018. IEEE Computer Society, pp 230–235
Beecks C, Hassani M, Brenger B, Hinnell J, Schüller D, Mittelberg I, Seidl T (2016) Efficient query processing in 3D motion capture gesture databases. Int J Semantic Comput 10(1):5–26
Berger I, Shamir A, Mahler M, Carter EJ, Hodgins JK (2013) Style and abstraction in portrait sketching. ACM Trans Graph 32(4):55,1–55,12
Chatfield K, Simonyan K, Vedaldi A, Zisserman A (2014) Return of the devil in the details: delving deep into convolutional nets. In: Valstar MF, French AP, Pridmore TP (eds) British machine vision conference, BMVC 2014, Nottingham, UK, September 1-5, 2014. BMVA Press
Cole F, Golovinskiy A, Limpaecher A, Barros HS, Finkelstein A, Funkhouser T, Rusinkiewicz S (2008) Where do people draw lines? ACM Trans Graph 27(3)
Dai G, Xie J, Fang Y (2018) Deep correlated holistic metric learning for sketch-based 3D shape retrieval. IEEE Trans Image Process 27 (7):3374–3386
Delanoy J, Aubry M, Isola P, Efros AA, Bousseau A (2018) 3D sketching using multi-view deep volumetric prediction. Proc ACM Comput Graph Interact Tech 1(1):21,1–21,22
Ding C, Liu L (2016) A survey of sketch based modeling systems. Front Comput Sci 10(6):985–999
Dupont S, Seddati O, Mahmoudi S (2016) Deepsketch 2: deep convolutional neural networks for partial sketch recognition. In: CBMI 2016, Bucharest, Romania, June 15-17, 2016, pp 1–6
Eitz M, Hays J, Alexa M (2012) How do humans sketch objects? ACM Trans Graph 31(4):44,1–44,10
Eitz M, Hildebrand K, Boubekeur T, Alexa M (2011) Sketch-based image retrieval: benchmark and bag-of-features descriptors. IEEE Trans Vis Comput Graph 17(11):1624–1636
Eitz M, Richter R, Boubekeur T, Hildebrand K, Alexa M (2012) Sketch-based shape retrieval. ACM Trans Graph 31(4):31,1–31,10
Funkhouser T, Min P, Kazhdan M, Chen J, Halderman A, Dobkin D, Jacobs D (2003) A search engine for 3D models. ACM Trans Graph 22(1):83–105
Furuya T, Ohbuchi R (2013) Ranking on cross-domain manifold for sketch-based 3D model retrieval. In: 2013 international conference on Cyberworlds (CW), pp 274–281
Giunchi D, James S, Steed A (2018) Model Retrieval by 3D Sketching in Immersive Virtual Reality. In: Kiyoshi K, Frank S, Bruce HT, Greg Welch (eds) 2018 IEEE Conference on Virtual Reality and 3D User Interfaces, VR 2018, Tuebingen/Reutlingen. IEEE Computer Society, Germany, pp 559–560
Giunchi D, James S, Steed A (2018) 3D sketching for interactive model retrieval in virtual reality. In: Proceedings of the joint symposium on computational aesthetics and sketch-based interfaces and modeling and non-photorealistic animation and rendering, Expressive’18. ACM, New York, pp 1:1–1:12
Ha D, Eck D (2017) A neural representation of sketch drawings. arXiv:1704.03477
Henshilwood CS, D’errico F, van Niekerk KL, Dayet L, Queffelec A, Pollarolo L (2018) An abstract drawing from the 73,000-year-old levels at Blombos Cave South Africa. Nature 562:115–118
Herot CF (1976) Graphical input through machine recognition of sketches. SIGGRAPH Comput Graph 10(2):97–102
Hu R, Collomosse JP (2013) A performance evaluation of gradient field HOG descriptor for sketch based image retrieval. Comput Vis Image Underst 117(7):790–806
Huang H, Kalogerakis E, Yumer E, Mech R (2017) Shape synthesis from sketches via procedural models and convolutional networks. IEEE Trans Vis Comput Graph 23(8):2003–2013
Huang Z, Fu H, Lau RWH (2014) Data-driven segmentation and labeling of freehand sketches. ACM Trans Graph 33(6):175,1–175,10
Igarashi T, Matsuoka S, Tanaka H (1999) Teddy: a sketching interface for 3D freeform design. In: SIGGRAPH 1999, Los Angeles, CA, USA, August 8-13, 1999, pp 409–416
Jackson B, Keefe DF (2016) Lift-off: using reference imagery and freehand sketching to create 3D models in VR. IEEE Trans Vis Comput Graph 22 (4):1442–1451
Johnson G, Gross MD, Hong J, Yi-Luen Do E (2009) Computational support for sketching in design: a review. Found Trends Hum-Comput Interact 2 (1):1–93
Jolliffe I (2002) Principal component analysis. Springer series in statistics. Springer, Berlin
Jung A, Hahmann S, Rohmer D, Bégault A, Boissieux L, Cani M (2015) Sketching folds: developable surfaces from non-planar silhouettes. ACM Trans Graph 34(5):155,1–155,12
Kalman RE (1960) A new approach to linear filtering and prediction problems. Transactions of the ASME–Journal of Basic Engineering 82:35–45
LeCun Y, Boser BE, Denker JS, Henderson D, Howard RE, Hubbard WE, Jackel LD (1989) Handwritten digit recognition with a back-propagation network. In: NIPS, Denver, Colorado, USA, 1989, pp 396–404
Li B, Lu Y (2016) http://cs.txstate.edu/yl12/SBR2016/
Li B, Lu Y, Duan F, Dong S, Fan Y, Qian L, Laga H, Li H, Li Y, Liu P, Ovsjanikov M, Tabia H, Ye Y, Yin H, Xue Z (2016) SHREC’16: 3D sketch-based 3D shape retrieval. In: 3DOR, pp 47–54. Eurographics Association
Li B, Lu Y, Ghumman A, Strylowski B, Gutierrez M, Sadiq S, Forster S, Feola N, Bugerin T (2015) 3D sketch-based 3D model retrieval. In: ICMR 2015, Shanghai, China, June 23-26, 2015, pp 555–558
Li B, Lu Y, Godil A, Schreck T, Aono M, Johan H, Saavedra JM, Tashiro S (2013) SHREC’13 track: large scale sketch-based 3D shape retrieval. In: Eurographics workshop on 3d object retrieval, Girona, Spain, 2013. Proceedings, pp 89–96
Li B, Lu Y, Godil A, Schreck T, Bustos B, Ferreira A, Furuya T, Fonseca MJ, Johan H, Matsuda T, Ohbuchi R, Pascoal PB, Saavedra JM (2014) A comparison of methods for sketch-based 3D shape retrieval. Comput Vis Image Underst 119:57–80
Li B, Lu Y, Li C, Godil A, Schreck T, Aono M, Burtscher M, Chen Q, Chowdhury NK, Fang B, Fu H, Furuya T, Li H, Liu J, Johan H, Kosaka R, Koyanagi H, Ohbuchi R, Tatsuma A, Wan Y, Zhang C, Zou C (2015) A comparison of 3D shape retrieval methods based on a large-scale benchmark supporting multimodal queries. Comput Vis Image Underst 131:1–27
Li B, Lu Y, Li C, Godil A, Schreck T, Aono M, Burtscher M, Fu H, Furuya T, Johan H, Liu J, Ohbuchi R, Tatsuma A, Zou C (2014) SHREC’14: extended Large scale sketch-based 3D shape retrieval. In: 3DOR, pp 121–130. Eurographics Association
Li K, Pang K, Song Y, Hospedales TM, Zhang H, Hu Y (2016) Fine-grained sketch-based image retrieval: the role of part-aware attributes. In: WACV 2016, Lake Placid, NY, USA, March 7-10, 2016, pp 1–9
Li Y, Hospedales TM, Song Y, Gong S (2015) Free-hand sketch recognition by multi-kernel feature learning. Comput Vis Image Underst 137:1–11
Li Y, Song Y, Hospedales TM, Gong S (2017) Free-hand sketch synthesis with deformable stroke models. Int J Comput Vis 122(1):169–190
Lu T, Tai C, Su F, Cai S (2005) A new recognition model for electronic architectural drawings. Comput Aided Des 37(10):1053–1069
Maybeck PS (1979) Stochastic models, estimation, and control: volume 1, volume 141 of Mathematics in Science and Engineering. Academic Press
Mohamad M, Shafry M, Rahim M, Othman N, Jupri Z (2009) A comparative study on extraction and recognition method of cad data from cad drawings. In: ICIME 2009, pp 709–713
Nealen A, Sorkine O, Alexa M, Cohen-Or D (2007) A sketch-based interface for detail-preserving mesh editing. In: SIGGRAPH San Diego, California, USA, August 5-9, 2007, Courses, p 42
Olsen L, Samavati FF, Sousa MC, Jorge JA (2009) Sketch-based modeling: a survey. Computers &, Graphics 33(1):85–103
Ouyang TY, Davis R (2011) Chemink: a natural real-time recognition system for chemical drawings. In: Proceedings of the 16th international conference on intelligent user interfaces, pp 267–276, New York, NY, USA. ACM
Paoli CD, Singh K (2015) Secondskin: sketch-based construction of layered 3D models. ACM Trans Graph 34(4):126,1–126,10
Radenovic F, Tolias G, Chum O (2018) Deep shape matching. In: The european conference on computer vision (ECCV)
Sahillioglu Y, Sezgin TM (2017) Sketch-based articulated 3D shape retrieval. IEEE Comput Graph Appl 37(6):88–101
Sangkloy P, Burnell N, Ham C, Hays J (2016) The Sketchy database: learning to retrieve badly drawn bunnies. ACM Trans Graph 35(4):119,1–119,12
Schneider RG, Tuytelaars T (2014) Sketch classification and classification-driven analysis using fisher vectors. ACM Trans Graph 33(6):1–9
Seddati O, Dupont S, Mahmoudi S (2015) Deepsketch: deep convolutional neural networks for sketch recognition and similarity search. In: CBMI 2015, Prague, Czech Republic, June 10-12, 2015, pp 1–6
Seddati O, Dupont S, Mahmoudi S (2017) Deepsketch 3. Multimedia Tools and Applications
Sedmidubský J, Zezula P (2019) Similarity search in 3D human motion data. In: El-Saddik A, Bimbo AD, Zhang Z, Hauptmann AG, Candan KS, Bertini M, Xie L, Wei X (eds) Proceedings of the 2019 on international conference on multimedia retrieval, ICMR 2019, Ottawa, ON, Canada, June 10-13, 2019, pp 5–6. ACM
Shao C, Bousseau A, Sheffer A, Singh K (2012) Crossshade: shading concept sketches using cross-section curves. ACM Trans Graph 31(4):45,1–45,11
Shilane P, Min P, Kazhdan MM, Funkhouser TA (2004) The princeton shape benchmark. In: (SMI 2004, 7-9 June 2004, Genova, Italy, pp 167–178
Sipiran I, Lokoc J, Bustos B, Skopal T (2017) Scalable 3D shape retrieval using local features and the signature quadratic form distance. Vis Compus 33(12):1571–1585
Snodgrass JG, Vanderwart M (1980) A standardized set of 260 pictures: norms for name agreement, image agreement, familiarity, and visual complexity. Journal of Experimental Pyschology: Human Learning and Memory 6 (2):174–215
Sousa P, Fonseca MJ (2009) Geometric matching for clip-art drawing retrieval. J Vis Comun Image Represent 20(2):71–83
Su H, Maji S, Kalogerakis E, Learned-Miller MJ (2015) Multi-view convolutional neural networks for 3D shape recognition. In: ICCV 2015, Santiago, Chile December 7-13, 2015, pp 945–953
Sun Z, Wang C, Zhang L, Zhang L (2012) Query-adaptive shape topic mining for hand-drawn sketch recognition. In: ACM MM’12, Nara, Japan, October 29 - November 02, 2012, pp 519–528
Sutherland IE (1964) Sketchpad: a man-machine graphical communication system. In: Proceedings of the SHARE design automation workshop, DAC’64, pages 329–346, New York, NY, USA. ACM
Tangelder JWH, Veltkamp RC (2008) A survey of content based 3d shape retrieval methods. Multimed Tools Appl 39(3):441–471
Vedaldi A, Lenc K (2015) Matconvnet: convolutional neural networks for MATLAB. In: ACM MM ’15, Brisbane, Australia, October 26 - 30, 2015, pp 689–692
Veltkamp RC, ter Haar FB (2007) SHREC 2007 3D Retrieval Contest. Technical Report 950 UU-CS- 2007-015, Department of Information and Computing Sciences, Utrecht University
Wang F, Kang L, Li Y (2015) Sketch-based 3D shape retrieval using convolutional neural networks. In: CVPR 2015, Boston, MA, USA, June 7-12, 2015, pp 1875–1883
Wang P-S, Liu Y, Guo Y-X, Sun C-Y, Tong X (2017) O-CNN: octree-based convolutional neural networks for 3D shape analysis. ACM Transactions on Graphics (SIGGRAPH) 36(4)
Welch G, Bishop G (1995) An introduction to the Kalman filter. Technical report, University of North Carolina at Chapel Hill, Chapel Hill, NC, USA
Xie J, Dai G, Zhu F, Fang Y (2017) Learning barycentric representations of 3D shapes for sketch-based 3D shape retrieval. In: CVPR 2017, Honolulu, HI, USA July 21-26, 2017, pp 3615–3623
Ye Y, Li B, Lu Y (2016) 3D sketch-based 3D model retrieval with convolutional neural network. In: ICPR 2016, Cancún, Mexico, December 4-8, 2016, pp 2936–2941
Yoon SM, Scherer M, Schreck T, Kuijper A (2010) Sketch-based 3D model retrieval using diffusion tensor fields of suggestive contours. In: ACM multimedia, pp 193–200
Yu Q, Liu F, Song Y, Xiang T, Hospedales TM, Loy CC (2016) Sketch me that shoe. In: CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016, pp 799–807
Yu Q, Yang Y, Liu F, Song Y, Xiang T, Hospedales TM (2017) Sketch-a-net: a deep neural network that beats humans. International Journal of Computer Vision, in press
Yu Q, Yang Y, Song Y, Xiang T, Hospedales TM (2015) Sketch-a-net that beats humans. In: BMVC 2015, Swansea, UK, September 7-10, 2015, pp 7.1–7.12
Zeleznik RC, Herndon KP, Hughes JF (2007) SKETCH: an interface for sketching 3D scenes. In: SIGGRAPH 2007, San Diego, California, USA, August 5-9, 2007, Courses, pp 19
Zhu F, Xie J, Fang Y (2016) Learning cross-domain neural networks for sketch-based 3D shape retrieval. In: Proceedings of the thirtieth AAAI conference on artificial intelligence, February 12-17, 2016, Phoenix, Arizona, USA, pp 3683–3689
Acknowledgments
This work is supported by Army Research Office grant W911NF-12-1-0057 to Dr. Yijuan Lu and Dr. Qi Tian, by NSF CRI-1305302, NSF CNS-1358939 and NSF OCI-1062439 to Dr. Yijuan Lu, and by the University of Southern Mississippi Faculty Startup Funds Award to Dr. Bo Li. We gratefully acknowledge the support from NVIDIA Corporation for the donation of the Titan X/Xp GPUs used in this research.
Author information
Authors and Affiliations
Corresponding author
Additional information
Publisher’s note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
About this article
Cite this article
Li, B., Yuan, J., Ye, Y. et al. 3D sketching for 3D object retrieval. Multimed Tools Appl 80, 9569–9595 (2021). https://doi.org/10.1007/s11042-020-10033-1
Received:
Revised:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11042-020-10033-1