Abstract
Virtual Environments (VE) relying on Virtual Reality (VR) can facilitate the co-design by enabling the users to create 3D mockups directly in the VE. Databases of 3D objects are helpful to populate the mockup and necessitate retrieving methods for the users. In early stages of the design process, the mockups are made up of common objects rather than variations of objects. Retrieving a 3D object in a large database can be fastidious even more in VR. Taking into account the necessity of a natural user’s interaction and the necessity to populate the mockup with common 3D objects, we propose, in this paper, a retrieval method based on 2D sketching in VR and machine learning. Our system is able to recognize 90 categories of objects related to VR interior design with an accuracy up to 86%. A preliminary study confirms the performance of the proposed solution.
R. Terrier and N. Martin—Contributed equally to this research.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Notes
- 1.
Top-1 accuracy: the answer with the highest probability must match the expected answer.
- 2.
Top-3 accuracy: any of the three answers with the highest probabilities must match the expected answer.
- 3.
Due to the COVID-19 pandemic and related lockdown, the evaluation could only be carried out on two participants.
References
Bell, J.: Machine Learning: Hands-On for Developers and Technical Professionals. Wiley, Indianapolis (2014)
Chollet, F., et al.: Keras (2015). https://keras.io
Wang, F., Kang, L., Li, Y.: Sketch-based 3D shape retrieval using convolutional neural networks. In: 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1875–1883. IEEE (2015)
Giunchi, D., James, S., Steed, A.: 3D sketching for interactive model retrieval in virtual reality. In: Proceedings of the Joint Symposium on Computational Aesthetics and Sketch-Based Interfaces and Modeling and Non-Photorealistic Animation and Rendering - Expressive 2018, Victoria, British Columbia, Canada, pp. 1–12. ACM Press (2018)
Google: The Quick, Draw! Dataset.: googlecreativelab/quickdraw-dataset (2017). https://github.com/googlecreativelab/quickdraw-dataset. original-date: 2017-05-09T18:28:32Z
Kingma, D.P., Ba, J.: Adam: A Method for Stochastic Optimization. arXiv:1412.6980 [cs], December 2014. http://arxiv.org/abs/1412.6980
LeCun, Y., Bengio, Y., Hinton, G.: Deep learning. Nature 521(7553), 436–444 (2015)
Lee, J.H., Shin, J., Realff, M.J.: Machine learning: overview of the recent progresses and implications for the process systems engineering field. Comput. Chem. Eng. 114, 111–121 (2018). https://doi.org/10.1016/j.compchemeng.2017.10.008. https://linkinghub.elsevier.com/retrieve/pii/S0098135417303538
Lei, H., Luo, G., Li, Y., Lin, S.: 3D model retrieval based on hand drawn sketches using LDA model. In: 2016 6th International Conference on Digital Home (ICDH), pp. 261–266. IEEE (2016)
Abadi, M., et al.: TensorFlow: Large-Scale Machine Learning on Heterogeneous Systems (2015). http://tensorflow.org/
Wang, P., et al.: A comprehensive survey of AR/MR-based co-design in manufacturing. Eng. Comput. 36, 1715–1738 (2020). https://doi.org/10.1007/s00366-019-00792-3
Acknowledgments
This study was carried out within b<>com, an institute of research and technology dedicated to digital technologies. It received support from the Future Investments program of the French National Research Agency (grant no. ANR-07-A0-AIRT).
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2020 Springer Nature Switzerland AG
About this paper
Cite this paper
Terrier, R., Martin, N. (2020). A Machine Learning Tool to Match 2D Drawings and 3D Objects’ Category for Populating Mockups in VR. In: Bourdot, P., Interrante, V., Kopper, R., Olivier, AH., Saito, H., Zachmann, G. (eds) Virtual Reality and Augmented Reality. EuroVR 2020. Lecture Notes in Computer Science(), vol 12499. Springer, Cham. https://doi.org/10.1007/978-3-030-62655-6_17
Download citation
DOI: https://doi.org/10.1007/978-3-030-62655-6_17
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-62654-9
Online ISBN: 978-3-030-62655-6
eBook Packages: Computer ScienceComputer Science (R0)