Material Identification System with Sound Simulation Assisted Method in VR/AR Scenarios
Pages 575 - 579
Abstract
We propose a new model using transfer learning method in motivation of specifying and identifying material of an object by knocking in VR/AR applications. Different from the traditional contact knocking material/object identification method, we apply the sound simulation method to enlarge the training dataset containing various models and materials in real-world scenarios. Our approach is based on Domain-Adversarial Training of Neural Networks that learns from the pre-collected simulated and corresponding real knocking sound to extract their common features determined by different materials. Given the scanned 3D model and the real knocking sound from users, we present an incremental learning model using the features extracted by pre-trained transfer learning model to generate the final material classifier. We perform an overall evaluation showing that our system achieves around 93.3% accuracy of identifying materials, which is much higher than the accuracy mentioned in previous work.
References
[1]
Arsen Abdulali, Ibragim Atadjanov, Seungkyu Lee, and Seokhee Jeon. 2019. Measurement-Based Hyper-Elastic Material Identification and Real-Time FEM Simulation for Haptic Rendering. In 25th ACM Symposium on Virtual Reality Software and Technology (Parramatta, NSW, Australia) (VRST ’19). Association for Computing Machinery, New York, NY, USA, Article 15, 10 pages. https://doi.org/10.1145/3359996.3364275
[2]
Vishal Agarwal, Shubham Kapse, Kishori Shimple, and Javed Shaikh. 2020. A Review On Various Techniques Used For Material Identification Using Acoustic Signal Processing. International Journal of Scientific & Technology Research 9 (03 2020), 5408.
[3]
Shijie Dai, Zhibin Gao, Zhiyuan Shi, and Lianfen Huang. 2015. Material Intelligent Identification Based on Hyperspectral Imaging and SVM. In 2015 First International Conference on Computational Intelligence Theory, Systems and Applications (CCITSA). 69–72. https://doi.org/10.1109/CCITSA.2015.16
[4]
Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, Mario Marchand, and Victor Lempitsky. 2016. Domain-adversarial training of neural networks. The journal of machine learning research 17, 1 (2016), 2096–2030.
[5]
Bruno L Giordano and Stephen McAdams. 2006. Material identification of real impact sounds: Effects of size variation in steel, glass, wood, and plexiglass plates. The Journal of the Acoustical Society of America 119, 2 (2006), 1171–1181.
[6]
Juan Jose Gonzalez Espana, Jovani Alberto Jimenez Builes, and Andres Felipe Jejen Tabares. 2015. Ultrasonic sensor for industrial inspection based on the acoustic impedance. In 2015 20th Symposium on Signal Processing, Images and Computer Vision (STSIVA). 1–6. https://doi.org/10.1109/STSIVA.2015.7330435
[7]
Watcharada Hamontree, Chowarit Mitsantisuk, and Jantanee Rungrangpitayagon. 2015. Object identification using knocking sound processing and reaction force from disturbance observer. In 2015 7th International Conference on Information Technology and Electrical Engineering (ICITEE). 370–375. https://doi.org/10.1109/ICITEED.2015.7408974
[8]
Ward Heylen, Stefan Lammens, Paul Sas, 1997. Modal analysis theory and testing. Vol. 200. Katholieke Universiteit Leuven Leuven, Belgium.
[9]
Xutong Jin, Sheng Li, Tianshu Qu, Dinesh Manocha, and Guoping Wang. 2020. Deep-Modal: Real-Time Impact Sound Synthesis for Arbitrary Shapes. In Proceedings of the 28th ACM International Conference on Multimedia (Seattle, WA, USA) (MM ’20). Association for Computing Machinery, New York, NY, USA, 1171–1179. https://doi.org/10.1145/3394171.3413572
[10]
N. Keshava. 2004. Distance metrics and band selection in hyperspectral processing with applications to material identification and spectral libraries. IEEE Transactions on Geoscience and Remote Sensing 42, 7 (2004), 1552–1565. https://doi.org/10.1109/TGRS.2004.830549
[11]
Munna Khan, Md Qaiser Reza, Ashok Kumar Salhan, and Shaila PSMA Sirdeshmukh. 2019. Acoustic resonance spectroscopy based simple system for spectral characterization and classification of materials. Journal of Intelligent & Fuzzy Systems 36, 5 (2019), 4389–4397.
[12]
Roberta L. Klatzky, Dinesh K. Pai, and Eric P. Krotkov. 2000. Perception of Material from Contact Sounds. Presence 9, 4 (2000), 399–410. https://doi.org/10.1162/105474600566907
[13]
Eric Krotkov, Roberta Klatzky, and Nina Zumel. 1997. Robotic perception of material: Experiments with shape-invariant acoustic measures of material type. In Experimental Robotics IV, Oussama Khatib and J. Kenneth Salisbury (Eds.). Springer Berlin Heidelberg, Berlin, Heidelberg, 204–211.
[14]
Man Liu and D.G. Gorman. 1995. Formulation of Rayleigh damping and its extensions. Computers & Structures 57, 2 (1995), 277–285. https://doi.org/10.1016/0045-7949(94)00611-6
[15]
Shan Luo, Leqi Zhu, Kaspar Althoefer, and Hongbin Liu. 2017. Knock-Knock: Acoustic object recognition by using stacked denoising autoencoders. Neurocomputing 267(2017), 18–24. https://doi.org/10.1016/j.neucom.2017.03.014
[16]
Christoph Mertz, Sanjeev J. Koppal, Solomon Sia, and Srinivasa Narasimhan. 2012. A low-power structured light sensor for outdoor scene reconstruction and dominant material identification. In 2012 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops. 15–22. https://doi.org/10.1109/CVPRW.2012.6239194
[17]
Mark Pauly, Niloy J Mitra, Joachim Giesen, Markus H Gross, and Leonidas J Guibas. 2005. Example-based 3d scan completion. In Symposium on Geometry Processing. 23–32.
[18]
Zhimin Ren, Hengchin Yeh, and Ming C. Lin. 2013. Example-Guided Physically Based Modal Sound Synthesis. ACM Trans. Graph. 32, 1, Article 1 (Feb. 2013), 16 pages. https://doi.org/10.1145/2421636.2421637
[19]
Auston Sterling and Ming C. Lin. 2016. Interactive Modal Sound Synthesis Using Generalized Proportional Damping. In Proceedings of the 20th ACM SIGGRAPH Symposium on Interactive 3D Graphics and Games (Redmond, Washington) (I3D ’16). Association for Computing Machinery, New York, NY, USA, 79–86. https://doi.org/10.1145/2856400.2856419
[20]
Zhenyu Tang, Nicholas J. Bryan, Dingzeyu Li, Timothy R. Langlois, and Dinesh Manocha. 2020. Scene-Aware Audio Rendering via Deep Acoustic Analysis. IEEE Transactions on Visualization and Computer Graphics 26, 5(2020), 1991–2001. https://doi.org/10.1109/TVCG.2020.2973058
[21]
Ju Wang, Jie Xiong, Xiaojiang Chen, Hongbo Jiang, Rajesh Krishna Balan, and Dingyi Fang. 2017. TagScan: Simultaneous Target Imaging and Material Identification with Commodity RFID Devices. In Proceedings of the 23rd Annual International Conference on Mobile Computing and Networking (Snowbird, Utah, USA) (MobiCom ’17). Association for Computing Machinery, New York, NY, USA, 288–300. https://doi.org/10.1145/3117811.3117830
[22]
Jui-Hsien Wang. 2019. Physics-based Sound Synthesis Using Time-domain Methods. 132 pages. https://www.proquest.com/dissertations-theses/physics-based-sound-synthesis-using-time-domain/docview/2466041968/se-2?accountid=13818- Database copyright ProQuest LLC; ProQuest does not claim copyright in the individual underlying works;.
[23]
Jui-Hsien Wang and Doug L James. 2019. KleinPAT: optimal mode conflation for time-domain precomputation of acoustic transfer.ACM Trans. Graph. 38, 4 (2019), 122–1.
[24]
Jui-Hsien Wang and Doug L. James. 2019. KleinPAT: Optimal Mode Conflation for Time-domain Precomputation of Acoustic Transfer. ACM Trans. Graph. 38, 4, Article 122 (July 2019), 12 pages. https://doi.org/10.1145/3306346.3322976
[25]
Hongyi Xu and Jernej Barbič. 2017. Example-Based Damping Design. ACM Trans. Graph. 36, 4, Article 53 (July 2017), 14 pages. https://doi.org/10.1145/3072959.3073631
[26]
Zhoutong Zhang, Jiajun Wu, Qiujia Li, Zhengjia Huang, James Traer, Josh H McDermott, Joshua B Tenenbaum, and William T Freeman. 2017. Generative modeling of audible shapes for object perception. In Proceedings of the IEEE International Conference on Computer Vision. 1251–1260.
Recommendations
Immersive auditory display system 'sound cask': three-dimensional sound field reproduction system based on the boundary surface control principle
VRST '18: Proceedings of the 24th ACM Symposium on Virtual Reality Software and TechnologySound cask was developed to realize the perfect 3D auditory display that creates 3D sound waves around the listener's head just the same as the primary sound field, based on the boundary surface control (BoSC) principle.
If we consider the sound ...
A Method for Virtual Acoustic Auralisation in VR
AM '18: Proceedings of the Audio Mostly 2018 on Sound in Immersion and EmotionIn today's industry, the use of prediction software in architectural acoustics is universal. Programs such as Odeon, CATT and CadnaA have become an integral part of the design process. These programs combine general acoustic theory with CAD modelling ...
Bird Sound Identification System using Deep Learning
AbstractThis paper identifies the model for classifying the bird sounds using the Convolutional Neural Network (CNN) algorithm. CNN are considered as one of the powerful toolkits developed in the machine learning and that have shown to be more efficient ...
Comments
Information & Contributors
Information
Published In
September 2021
711 pages
ISBN:9781450384612
DOI:10.1145/3460418
Copyright © 2021 ACM.
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]
Sponsors
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 24 September 2021
Check for updates
Author Tags
Qualifiers
- Research-article
- Research
- Refereed limited
Funding Sources
- NSFC
- Startup Fund for Youngman Research at SJTU, and Program of Shanghai Academic Research Leader
Conference
UbiComp '21
Sponsor:
UbiComp '21: The 2021 ACM International Joint Conference on Pervasive and Ubiquitous Computing
September 21 - 26, 2021
Virtual, USA
Acceptance Rates
Overall Acceptance Rate 764 of 2,912 submissions, 26%
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 203Total Downloads
- Downloads (Last 12 months)25
- Downloads (Last 6 weeks)3
Reflects downloads up to 08 Mar 2025
Other Metrics
Citations
View Options
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign inFull Access
View options
View or Download as a PDF file.
PDFeReader
View online with eReader.
eReaderHTML Format
View this article in HTML Format.
HTML Format