Skip to main content
SpringerLink
Log in

Sketch-based 3D model retrieval utilizing adaptive view clustering and semantic information

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

Searching for relevant 3D models based on hand-drawn sketches is both intuitive and important for many applications, such as sketch-based 3D modeling and recognition, human computer interaction, 3D animation, game design, and etc. In this paper, our target is to significantly improve the current sketch-based 3D retrieval performance in terms of both accuracy and efficiency. We propose a new sketch-based 3D model retrieval framework by utilizing adaptive view clustering and semantic information. It first utilizes a proposed viewpoint entropy-based 3D information complexity measurement to guide adaptive view clustering of a 3D model to shortlist a set of representative sample views for 2D-3D comparison. To bridge the gap between the query sketches and the target models, we then incorporate a novel semantic sketch-based search approach to further improve the retrieval performance. Experimental results on several latest benchmarks have evidently demonstrated our significant improvement in retrieval performance.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10

Similar content being viewed by others

References

  1. AIM@SHAPE: SHREC Contest. Home Page (2010). http://www.aimatshape.net/event/SHREC/

  2. Ansary TF, Daoudi M, Vandeborre JP (2007) A Bayesian 3-D search engine using adaptive views clustering. IEEE Trans Multimedia 9(1):78–88

    Article  Google Scholar 

  3. Belongie S, Malik J, Puzicha J (2002) Shape matching and object recognition using shape contexts. IEEE Trans Pattern Anal Mach Intell 24(4):509–522

    Article  Google Scholar 

  4. Bezdek JC (1981) Pattern Recognition with Fuzzy Objective Function Algorithms. Kluwer Academic Publishers, Norwell, MA, USA

  5. Eitz M, Hays J, Alexa M (2012) How do humans sketch objects?. ACM Trans Graph 31(4):44:1–44:10

    Google Scholar 

  6. Eitz M, Hildebrand K, Boubekeur T, Alexa M (2011) Sketch-based image retrieval: Benchmark and bag-of-features descriptors. IEEE Trans Vis Comput Graph 17(11):1624–1636

    Article  Google Scholar 

  7. Eitz M, Richter R, Boubekeur T, Hildebrand K, Alexa M (2012) Sketch-based shape retrieval. ACM Trans Graph 31(4):31:1–31:10

    Google Scholar 

  8. Fang Y, Xie J, Dai G, Wang M, Zhu F, Xu T, Wong EK (2015) 3D deep shape descriptor. In: IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2015, Boston, MA, USA, June 7-12, 2015, pp 2319–2328. IEEE Computer Society. doi:10.1109/CVPR.2015.7298845

  9. Gong B, Liu J, Wang X, Tang X (2013) Learning semantic signatures for 3D object retrieval. IEEE Trans Multimedia 15(2):369–377

    Article  Google Scholar 

  10. Hou S, Lou K, Ramani K (2005) SVM-based Semantic Clustering and Retrieval of A 3D Model Database. Comput-Aided Des Applic 2:155–164

    Article  Google Scholar 

  11. Jayanti S, Kalyanaraman Y, Iyer N, Ramani K (2006) Developing an engineering shape benchmark for CAD models. Comput Aided Des 38(9):939–953

    Article  Google Scholar 

  12. Jonker R, Volgenant A (1987) A shortest augmenting path algorithm for dense and sparse linear assignment problems. Computing 38(4):325–340

    Article  MathSciNet  MATH  Google Scholar 

  13. Li B, Johan H (2012) Sketch-based 3D model retrieval by incorporating 2D-3D alignment. Multimedia Tools Appl:1–23

  14. Li B (2013) Johan, H.: 3D model retrieval using hybrid features and class information. Multimedia Tools Appl 62(3):821–846

    Article  Google Scholar 

  15. Li B, Lu Y, Fares R (2013) Semantic sketch-based 3D model retrieval. In: ICME, pp 1–4

  16. Li B, Lu Y, Godil A, Schreck T, Aono M, Johan H, Saavedra JM, Tashiro S (2013) SHREC’13 track: Large scale sketch-based 3D shape retrieval. In: 3DOR, pp 89–96

  17. Li B, Lu Y, Godil A, Schreck T, Bustos B, Ferreira A, Furuya T, Fonseca MJ, Johan H, Matsuda T, Ohbuchi R, Pascoal PB, Saavedra JM (2014) A comparison of methods for sketch-based 3D shape retrieval. Comput Vis Image Underst 119:57–80

    Article  Google Scholar 

  18. Li B, Lu Y, Johan H (2013) Sketch-based 3D model retrieval by viewpoint entropy-based adaptive view clustering. In: Eurographics Workshop on 3D Object Retrieval, Girona, Spain, 2013. Proceedings, pp 49–56

  19. Li B, Lu Y, Li C, Godil A, Schreck T, Aono M, Burtscher M, Chen Q, Chowdhury NK, Fang B, Fu H, Furuya T, Li H, Liu J, Johan H, Kosaka R, Koyanagi H, Ohbuchi R, Tatsuma A, Wan Y, Zhang C, Zou C (2015) A comparison of 3D shape retrieval methods based on a large-scale benchmark supporting multimodal queries. Comput Vis Image Underst 131:1–27. doi:10.1016/j.cviu.2014.10.006

  20. Li B, Lu Y, Li C, Godil A, Schreck T, Aono M, Burtscher M, Fu H, Furuya T, Johan H, Liu J, Ohbuchi R, Tatsuma A, Zou C (2014) SHREC’14: Extended Large Scale Sketch-Based 3D Shape Retrieval. In: Eurographics Workshop on 3D Object Retrieval, Strasbourg, France, 2014. Proceedings, pp 121–130

  21. Li B, Schreck T, Godil A, Alexa M, Boubekeur T, Bustos B, Chen J, Eitz M, Furuya T, Hildebrand K, Huang S, Johan H, Kuijper A, Ohbuchi R, Richter R, Saavedra JM, Scherer M, Yanagimachi T, Yoon G, Yoon SM (2012) SHREC’12 track: Sketch-based 3D shape retrieval. In: Eurographics Workshop on 3D Object Retrieval 2012, Cagliari, Italy, May 13, 2012. Proceedings, pp 109– 118

  22. Mokhtarian F, Abbasi S (2005) Robust automatic selection of optimal views in multi-view free-form object recognition. Pattern Recogn 38(7):1021–1031

  23. Nie L, Yan S, Wang M, Hong R, Chua T (2012) Harvesting visual concepts for image search with complex queries. In: Babaguchi N, Aizawa K, Smith J.R., Satoh S., Plagemann T., Hua X., Yan R. (eds) Proceedings of the 20th ACM Multimedia Conference, MM ’12, Nara, Japan, October 29 - November 02, 2012, pp 59–68. ACM. doi:10.1145/2393347.2393363

  24. Ohbuchi R, Yamamoto A, Kobayashi J (2007) Learning semantic categories for 3D model retrieval. In: Multimedia Information Retrieval, pp 31–40

  25. Page DL, Koschan A, Sukumar SR, Roui-Abidi B, Abidi MA (2003) Shape analysis algorithm based on information theory. In: ICIP (1), pp 229–232

  26. Powers DMW (2007) Evaluation: From Precision, Recall and F-measure to ROC, Informedness, Markedness & Correlation. J Mach Learn Technol 2(1):37–63

    Google Scholar 

  27. Rossignac J (2005) Shape complexity. Vis Comput 21(12):985–996

    Article  Google Scholar 

  28. Saleem W, Belyaev AG, Wang D, Seidel HP (2011) On visual complexity of 3D shapes. Comput Graph 35(3):580–585

    Article  Google Scholar 

  29. (2016) ShapeNet: http://shapenet.cs.stanford.edu/

  30. Shilane P, Min P, Kazhdan M, Funkhouser T (2004) The Princeton shape benchmark. In: SMA, pp 167–178

  31. Takahashi S, Fujishiro I, Takeshima Y, Nishita T (2005) A feature-driven approach to locating optimal viewpoints for volume visualization. In: IEEE Visualization, pp 495–502

  32. Vázquez P.P., Feixas M, Sbert M, Heidrich W (2003) Automatic view selection using viewpoint entropy and its applications to image-based modelling. Comput Graph Forum 22(4):689–700

    Article  Google Scholar 

  33. Vranic D (2004) 3D Model Retrieval. University of Leipzig, PhD thesis

    MATH  Google Scholar 

  34. Wang F, Kang L, Li Y (2015) Sketch-based 3D shape retrieval using convolutional neural networks. In: IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2015, Boston, MA, USA, June 7-12, 2015, pp 1875–1883

  35. Wessel R, Blümel I., Klein R (2009) A 3D shape benchmark for retrieval and automatic classification of architectural data. In: Eurographics Workshop on 3D Object Retrieval, 3DOR 2009, pp 53–56

  36. Yoon SM, Scherer M, Schreck T, Kuijper A (2010) Sketch-based 3D model retrieval using diffusion tensor fields of suggestive contours. In: ACM Multimedia, pp 193–200

  37. Zhu F, Xie J, Fang Y (2016) Learning cross-domain neural networks for sketch-based 3D shape retrieval. In: Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence, February 12-17, 2016, Phoenix, Arizona, USA., pp 3683–3689

  38. Zhu J, Zhu F, Wong EK, Fang Y (2015) Learning pairwise neural network encoder for depth image-based 3d model retrieval. In: Zhou X, Smeaton AF, Tian Q, Bulterman DCA, Shen HT, Mayer-Patel K, Yan S (eds) Proceedings of the 23rd Annual ACM Conference on Multimedia Conference, MM ’15, Brisbane, Australia, October 26 - 30, 2015, pp 1227–1230. ACM. doi:10.1145/2733373.2806323

Download references

Acknowledgments

The work of Bo Li and Yijuan Lu is supported by the Texas State University Research Enhancement Program (REP), Army Research Office grant W911NF-12-1-0057, and NSF CNS 1305302 to Dr. Yijuan Lu.

The research done by Henry Johan in Fraunhofer IDM@NTU is supported by the National Research Foundation, Prime Minister’s Office, Singapore under its International Research Centres in Singapore Funding Initiative.

Author information

Authors and Affiliations

  1. School of Computing, University of Southern Mississippi, 730 E Beach Blvd, Long Beach, MS, 39560, USA

    Bo Li

  2. Department of Computer Science, Texas State University, 601 University Dr, San Marcos, TX, 78666, USA

    Yijuan Lu & Ribel Fares

  3. Fraunhofer IDM@NTU, Nanyang Technological University, 50 Nanyang Ave, Singapore, 639798, Singapore

    Henry Johan

Authors
  1. Bo Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Yijuan Lu
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Henry Johan
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Ribel Fares
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Bo Li.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Li, B., Lu, Y., Johan, H. et al. Sketch-based 3D model retrieval utilizing adaptive view clustering and semantic information. Multimed Tools Appl 76, 26603–26631 (2017). https://doi.org/10.1007/s11042-016-4187-3

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-016-4187-3

Keywords

Search

Navigation