Skip to main content
SpringerLink
Log in

3D sketching for 3D object retrieval

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

Sketching provides the most natural way to provide a visual search query for visual object search. However, how to draw 3D sketches in a three-dimensional space and how to use a hand-drawn 3D sketch to search similar 3D models are not only interesting and novel, but also challenging research topics. In this paper, we try to answer them by initiating a novel study on 3D sketching and build a 3D sketching system which allows users to freely draw 3D sketches in the air and demonstrate its promising potentials in related applications such as collecting 3D sketch data and conducting 3D sketch-based 3D model retrieval. By utilizing the 3D sketching system, we collect a 3D sketch dataset, build a 3D sketch-based 3D model retrieval benchmark, and organize a Eurographics Shape Retrieval Contest (SHREC) track on 3D sketch-based shape retrieval based on the benchmark. We investigate 3D sketch and model matching problems and propose a novel 3D sketch-based model retrieval algorithm CNN-SBR based on Convolutional Neural Networks (CNNs) and achieve the best performance in the SHREC track. We wish that the 3D sketching system, the 3D sketch-based model retrieval benchmark, and the proposed 3D sketch-based model retrieval algorithm CNN-SBR will further promote sketch-based shape retrieval and its applications. We have made all of these publicly available on the project homepage: http://orca.st.usm.edu/~bli/SBR16/project.html.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10

Similar content being viewed by others

Notes

  1. http://orca.st.usm.edu/~bli/SBR16/project.html

References

  1. Ankerst M, Kastenmüller G, Kriegel H, Seidl T (1999) 3D shape histograms for similarity search and classification in spatial databases. In: Advances in spatial databases, 6th international symposium, SSD’99, Hong Kong, China, July 20-23, 1999, Proceedings, pp 207–226

  2. Araújo C, Cabiddu D, Attene M, Livesu M, Vining N, Sheffer A (2019) Surface2Volume: surface segmentation conforming assemblable volumetric partition. ACM Trans Graph 38(4):80,1–80,16

    Article  Google Scholar 

  3. Aurenhammer F (1991) Voronoi diagrams - a survey of a fundamental geometric data structure. ACM Comput Surv 23(3):345–405

    Article  Google Scholar 

  4. Bae S, Balakrishnan R, Singh K (2008) Ilovesketch: as-natural-as-possible sketching system for creating 3D curve model. In: UIST. ACM, pp 151–160

  5. Beecks C, Grass A (2018) Efficient point-based pattern search in 3D motion capture databases. In: Younas M, Disso JP (eds) 6th IEEE international conference on future internet of things and cloud, FiCloud 2018, Barcelona, Spain, August 6-8, 2018. IEEE Computer Society, pp 230–235

  6. Beecks C, Hassani M, Brenger B, Hinnell J, Schüller D, Mittelberg I, Seidl T (2016) Efficient query processing in 3D motion capture gesture databases. Int J Semantic Comput 10(1):5–26

    Article  Google Scholar 

  7. Berger I, Shamir A, Mahler M, Carter EJ, Hodgins JK (2013) Style and abstraction in portrait sketching. ACM Trans Graph 32(4):55,1–55,12

    Article  Google Scholar 

  8. Chatfield K, Simonyan K, Vedaldi A, Zisserman A (2014) Return of the devil in the details: delving deep into convolutional nets. In: Valstar MF, French AP, Pridmore TP (eds) British machine vision conference, BMVC 2014, Nottingham, UK, September 1-5, 2014. BMVA Press

  9. Cole F, Golovinskiy A, Limpaecher A, Barros HS, Finkelstein A, Funkhouser T, Rusinkiewicz S (2008) Where do people draw lines? ACM Trans Graph 27(3)

  10. Dai G, Xie J, Fang Y (2018) Deep correlated holistic metric learning for sketch-based 3D shape retrieval. IEEE Trans Image Process 27 (7):3374–3386

    Article  MathSciNet  Google Scholar 

  11. Delanoy J, Aubry M, Isola P, Efros AA, Bousseau A (2018) 3D sketching using multi-view deep volumetric prediction. Proc ACM Comput Graph Interact Tech 1(1):21,1–21,22

    Article  Google Scholar 

  12. Ding C, Liu L (2016) A survey of sketch based modeling systems. Front Comput Sci 10(6):985–999

    Article  Google Scholar 

  13. Dupont S, Seddati O, Mahmoudi S (2016) Deepsketch 2: deep convolutional neural networks for partial sketch recognition. In: CBMI 2016, Bucharest, Romania, June 15-17, 2016, pp 1–6

  14. Eitz M, Hays J, Alexa M (2012) How do humans sketch objects? ACM Trans Graph 31(4):44,1–44,10

    Google Scholar 

  15. Eitz M, Hildebrand K, Boubekeur T, Alexa M (2011) Sketch-based image retrieval: benchmark and bag-of-features descriptors. IEEE Trans Vis Comput Graph 17(11):1624–1636

    Article  Google Scholar 

  16. Eitz M, Richter R, Boubekeur T, Hildebrand K, Alexa M (2012) Sketch-based shape retrieval. ACM Trans Graph 31(4):31,1–31,10

    Google Scholar 

  17. Funkhouser T, Min P, Kazhdan M, Chen J, Halderman A, Dobkin D, Jacobs D (2003) A search engine for 3D models. ACM Trans Graph 22(1):83–105

    Article  Google Scholar 

  18. Furuya T, Ohbuchi R (2013) Ranking on cross-domain manifold for sketch-based 3D model retrieval. In: 2013 international conference on Cyberworlds (CW), pp 274–281

  19. Giunchi D, James S, Steed A (2018) Model Retrieval by 3D Sketching in Immersive Virtual Reality. In: Kiyoshi K, Frank S, Bruce HT, Greg Welch (eds) 2018 IEEE Conference on Virtual Reality and 3D User Interfaces, VR 2018, Tuebingen/Reutlingen. IEEE Computer Society, Germany, pp 559–560

  20. Giunchi D, James S, Steed A (2018) 3D sketching for interactive model retrieval in virtual reality. In: Proceedings of the joint symposium on computational aesthetics and sketch-based interfaces and modeling and non-photorealistic animation and rendering, Expressive’18. ACM, New York, pp 1:1–1:12

  21. Ha D, Eck D (2017) A neural representation of sketch drawings. arXiv:1704.03477

  22. Henshilwood CS, D’errico F, van Niekerk KL, Dayet L, Queffelec A, Pollarolo L (2018) An abstract drawing from the 73,000-year-old levels at Blombos Cave South Africa. Nature 562:115–118

    Article  Google Scholar 

  23. Herot CF (1976) Graphical input through machine recognition of sketches. SIGGRAPH Comput Graph 10(2):97–102

    Article  Google Scholar 

  24. Hu R, Collomosse JP (2013) A performance evaluation of gradient field HOG descriptor for sketch based image retrieval. Comput Vis Image Underst 117(7):790–806

    Article  Google Scholar 

  25. Huang H, Kalogerakis E, Yumer E, Mech R (2017) Shape synthesis from sketches via procedural models and convolutional networks. IEEE Trans Vis Comput Graph 23(8):2003–2013

    Article  Google Scholar 

  26. Huang Z, Fu H, Lau RWH (2014) Data-driven segmentation and labeling of freehand sketches. ACM Trans Graph 33(6):175,1–175,10

    Article  Google Scholar 

  27. Igarashi T, Matsuoka S, Tanaka H (1999) Teddy: a sketching interface for 3D freeform design. In: SIGGRAPH 1999, Los Angeles, CA, USA, August 8-13, 1999, pp 409–416

  28. Jackson B, Keefe DF (2016) Lift-off: using reference imagery and freehand sketching to create 3D models in VR. IEEE Trans Vis Comput Graph 22 (4):1442–1451

    Article  Google Scholar 

  29. Johnson G, Gross MD, Hong J, Yi-Luen Do E (2009) Computational support for sketching in design: a review. Found Trends Hum-Comput Interact 2 (1):1–93

    Article  Google Scholar 

  30. Jolliffe I (2002) Principal component analysis. Springer series in statistics. Springer, Berlin

    Google Scholar 

  31. Jung A, Hahmann S, Rohmer D, Bégault A, Boissieux L, Cani M (2015) Sketching folds: developable surfaces from non-planar silhouettes. ACM Trans Graph 34(5):155,1–155,12

    Article  Google Scholar 

  32. Kalman RE (1960) A new approach to linear filtering and prediction problems. Transactions of the ASME–Journal of Basic Engineering 82:35–45

    Article  MathSciNet  Google Scholar 

  33. LeCun Y, Boser BE, Denker JS, Henderson D, Howard RE, Hubbard WE, Jackel LD (1989) Handwritten digit recognition with a back-propagation network. In: NIPS, Denver, Colorado, USA, 1989, pp 396–404

  34. Li B, Lu Y (2016) http://cs.txstate.edu/yl12/SBR2016/

  35. Li B, Lu Y, Duan F, Dong S, Fan Y, Qian L, Laga H, Li H, Li Y, Liu P, Ovsjanikov M, Tabia H, Ye Y, Yin H, Xue Z (2016) SHREC’16: 3D sketch-based 3D shape retrieval. In: 3DOR, pp 47–54. Eurographics Association

  36. Li B, Lu Y, Ghumman A, Strylowski B, Gutierrez M, Sadiq S, Forster S, Feola N, Bugerin T (2015) 3D sketch-based 3D model retrieval. In: ICMR 2015, Shanghai, China, June 23-26, 2015, pp 555–558

  37. Li B, Lu Y, Godil A, Schreck T, Aono M, Johan H, Saavedra JM, Tashiro S (2013) SHREC’13 track: large scale sketch-based 3D shape retrieval. In: Eurographics workshop on 3d object retrieval, Girona, Spain, 2013. Proceedings, pp 89–96

  38. Li B, Lu Y, Godil A, Schreck T, Bustos B, Ferreira A, Furuya T, Fonseca MJ, Johan H, Matsuda T, Ohbuchi R, Pascoal PB, Saavedra JM (2014) A comparison of methods for sketch-based 3D shape retrieval. Comput Vis Image Underst 119:57–80

    Article  Google Scholar 

  39. Li B, Lu Y, Li C, Godil A, Schreck T, Aono M, Burtscher M, Chen Q, Chowdhury NK, Fang B, Fu H, Furuya T, Li H, Liu J, Johan H, Kosaka R, Koyanagi H, Ohbuchi R, Tatsuma A, Wan Y, Zhang C, Zou C (2015) A comparison of 3D shape retrieval methods based on a large-scale benchmark supporting multimodal queries. Comput Vis Image Underst 131:1–27

    Article  Google Scholar 

  40. Li B, Lu Y, Li C, Godil A, Schreck T, Aono M, Burtscher M, Fu H, Furuya T, Johan H, Liu J, Ohbuchi R, Tatsuma A, Zou C (2014) SHREC’14: extended Large scale sketch-based 3D shape retrieval. In: 3DOR, pp 121–130. Eurographics Association

  41. Li K, Pang K, Song Y, Hospedales TM, Zhang H, Hu Y (2016) Fine-grained sketch-based image retrieval: the role of part-aware attributes. In: WACV 2016, Lake Placid, NY, USA, March 7-10, 2016, pp 1–9

  42. Li Y, Hospedales TM, Song Y, Gong S (2015) Free-hand sketch recognition by multi-kernel feature learning. Comput Vis Image Underst 137:1–11

    Article  Google Scholar 

  43. Li Y, Song Y, Hospedales TM, Gong S (2017) Free-hand sketch synthesis with deformable stroke models. Int J Comput Vis 122(1):169–190

    Article  MathSciNet  Google Scholar 

  44. Lu T, Tai C, Su F, Cai S (2005) A new recognition model for electronic architectural drawings. Comput Aided Des 37(10):1053–1069

    Article  Google Scholar 

  45. Maybeck PS (1979) Stochastic models, estimation, and control: volume 1, volume 141 of Mathematics in Science and Engineering. Academic Press

  46. Mohamad M, Shafry M, Rahim M, Othman N, Jupri Z (2009) A comparative study on extraction and recognition method of cad data from cad drawings. In: ICIME 2009, pp 709–713

  47. Nealen A, Sorkine O, Alexa M, Cohen-Or D (2007) A sketch-based interface for detail-preserving mesh editing. In: SIGGRAPH San Diego, California, USA, August 5-9, 2007, Courses, p 42

  48. Olsen L, Samavati FF, Sousa MC, Jorge JA (2009) Sketch-based modeling: a survey. Computers &, Graphics 33(1):85–103

    Article  Google Scholar 

  49. Ouyang TY, Davis R (2011) Chemink: a natural real-time recognition system for chemical drawings. In: Proceedings of the 16th international conference on intelligent user interfaces, pp 267–276, New York, NY, USA. ACM

  50. Paoli CD, Singh K (2015) Secondskin: sketch-based construction of layered 3D models. ACM Trans Graph 34(4):126,1–126,10

    Article  Google Scholar 

  51. Radenovic F, Tolias G, Chum O (2018) Deep shape matching. In: The european conference on computer vision (ECCV)

  52. Sahillioglu Y, Sezgin TM (2017) Sketch-based articulated 3D shape retrieval. IEEE Comput Graph Appl 37(6):88–101

    Article  Google Scholar 

  53. Sangkloy P, Burnell N, Ham C, Hays J (2016) The Sketchy database: learning to retrieve badly drawn bunnies. ACM Trans Graph 35(4):119,1–119,12

    Article  Google Scholar 

  54. Schneider RG, Tuytelaars T (2014) Sketch classification and classification-driven analysis using fisher vectors. ACM Trans Graph 33(6):1–9

    Article  Google Scholar 

  55. Seddati O, Dupont S, Mahmoudi S (2015) Deepsketch: deep convolutional neural networks for sketch recognition and similarity search. In: CBMI 2015, Prague, Czech Republic, June 10-12, 2015, pp 1–6

  56. Seddati O, Dupont S, Mahmoudi S (2017) Deepsketch 3. Multimedia Tools and Applications

  57. Sedmidubský J, Zezula P (2019) Similarity search in 3D human motion data. In: El-Saddik A, Bimbo AD, Zhang Z, Hauptmann AG, Candan KS, Bertini M, Xie L, Wei X (eds) Proceedings of the 2019 on international conference on multimedia retrieval, ICMR 2019, Ottawa, ON, Canada, June 10-13, 2019, pp 5–6. ACM

  58. Shao C, Bousseau A, Sheffer A, Singh K (2012) Crossshade: shading concept sketches using cross-section curves. ACM Trans Graph 31(4):45,1–45,11

    Article  Google Scholar 

  59. Shilane P, Min P, Kazhdan MM, Funkhouser TA (2004) The princeton shape benchmark. In: (SMI 2004, 7-9 June 2004, Genova, Italy, pp 167–178

  60. Sipiran I, Lokoc J, Bustos B, Skopal T (2017) Scalable 3D shape retrieval using local features and the signature quadratic form distance. Vis Compus 33(12):1571–1585

    Article  Google Scholar 

  61. Snodgrass JG, Vanderwart M (1980) A standardized set of 260 pictures: norms for name agreement, image agreement, familiarity, and visual complexity. Journal of Experimental Pyschology: Human Learning and Memory 6 (2):174–215

    Google Scholar 

  62. Sousa P, Fonseca MJ (2009) Geometric matching for clip-art drawing retrieval. J Vis Comun Image Represent 20(2):71–83

    Article  Google Scholar 

  63. Su H, Maji S, Kalogerakis E, Learned-Miller MJ (2015) Multi-view convolutional neural networks for 3D shape recognition. In: ICCV 2015, Santiago, Chile December 7-13, 2015, pp 945–953

  64. Sun Z, Wang C, Zhang L, Zhang L (2012) Query-adaptive shape topic mining for hand-drawn sketch recognition. In: ACM MM’12, Nara, Japan, October 29 - November 02, 2012, pp 519–528

  65. Sutherland IE (1964) Sketchpad: a man-machine graphical communication system. In: Proceedings of the SHARE design automation workshop, DAC’64, pages 329–346, New York, NY, USA. ACM

  66. Tangelder JWH, Veltkamp RC (2008) A survey of content based 3d shape retrieval methods. Multimed Tools Appl 39(3):441–471

    Article  Google Scholar 

  67. Vedaldi A, Lenc K (2015) Matconvnet: convolutional neural networks for MATLAB. In: ACM MM ’15, Brisbane, Australia, October 26 - 30, 2015, pp 689–692

  68. Veltkamp RC, ter Haar FB (2007) SHREC 2007 3D Retrieval Contest. Technical Report 950 UU-CS- 2007-015, Department of Information and Computing Sciences, Utrecht University

  69. Wang F, Kang L, Li Y (2015) Sketch-based 3D shape retrieval using convolutional neural networks. In: CVPR 2015, Boston, MA, USA, June 7-12, 2015, pp 1875–1883

  70. Wang P-S, Liu Y, Guo Y-X, Sun C-Y, Tong X (2017) O-CNN: octree-based convolutional neural networks for 3D shape analysis. ACM Transactions on Graphics (SIGGRAPH) 36(4)

  71. Welch G, Bishop G (1995) An introduction to the Kalman filter. Technical report, University of North Carolina at Chapel Hill, Chapel Hill, NC, USA

  72. Xie J, Dai G, Zhu F, Fang Y (2017) Learning barycentric representations of 3D shapes for sketch-based 3D shape retrieval. In: CVPR 2017, Honolulu, HI, USA July 21-26, 2017, pp 3615–3623

  73. Ye Y, Li B, Lu Y (2016) 3D sketch-based 3D model retrieval with convolutional neural network. In: ICPR 2016, Cancún, Mexico, December 4-8, 2016, pp 2936–2941

  74. Yoon SM, Scherer M, Schreck T, Kuijper A (2010) Sketch-based 3D model retrieval using diffusion tensor fields of suggestive contours. In: ACM multimedia, pp 193–200

  75. Yu Q, Liu F, Song Y, Xiang T, Hospedales TM, Loy CC (2016) Sketch me that shoe. In: CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016, pp 799–807

  76. Yu Q, Yang Y, Liu F, Song Y, Xiang T, Hospedales TM (2017) Sketch-a-net: a deep neural network that beats humans. International Journal of Computer Vision, in press

  77. Yu Q, Yang Y, Song Y, Xiang T, Hospedales TM (2015) Sketch-a-net that beats humans. In: BMVC 2015, Swansea, UK, September 7-10, 2015, pp 7.1–7.12

  78. Zeleznik RC, Herndon KP, Hughes JF (2007) SKETCH: an interface for sketching 3D scenes. In: SIGGRAPH 2007, San Diego, California, USA, August 5-9, 2007, Courses, pp 19

  79. Zhu F, Xie J, Fang Y (2016) Learning cross-domain neural networks for sketch-based 3D shape retrieval. In: Proceedings of the thirtieth AAAI conference on artificial intelligence, February 12-17, 2016, Phoenix, Arizona, USA, pp 3683–3689

Download references

Acknowledgments

This work is supported by Army Research Office grant W911NF-12-1-0057 to Dr. Yijuan Lu and Dr. Qi Tian, by NSF CRI-1305302, NSF CNS-1358939 and NSF OCI-1062439 to Dr. Yijuan Lu, and by the University of Southern Mississippi Faculty Startup Funds Award to Dr. Bo Li. We gratefully acknowledge the support from NVIDIA Corporation for the donation of the Titan X/Xp GPUs used in this research.

Author information

Authors and Affiliations

  1. School of Computing Sciences and Computer Engineering, University of Southern Mississippi, Long Beach, MS, USA

    Bo Li & Juefei Yuan

  2. Department of Computer Science, Texas State University, San Marcos, TX, USA

    Yuxiang Ye & Yijuan Lu

  3. School of Computing Sciences and Computer Engineering, University of Southern Mississippi, Hattiesburg, MS, USA

    Chaoyang Zhang

  4. Department of Computer Science, University of Texas at San Antonio, San Antonio, TX, USA

    Qi Tian

Authors
  1. Bo Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Juefei Yuan
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Yuxiang Ye
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Yijuan Lu
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Chaoyang Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Qi Tian
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Yijuan Lu.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Li, B., Yuan, J., Ye, Y. et al. 3D sketching for 3D object retrieval. Multimed Tools Appl 80, 9569–9595 (2021). https://doi.org/10.1007/s11042-020-10033-1

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-020-10033-1

Keywords

Search

Navigation