skip to main content
10.1145/3290605.3300906acmconferencesArticle/Chapter ViewAbstractPublication PageschiConference Proceedingsconference-collections
research-article

VibEye: Vibration-Mediated Object Recognition for Tangible Interactive Applications

Published: 02 May 2019 Publication History

Abstract

We present VibEye: a vibration-mediated recognition system of objects for tangible interaction. A user holds an object between two fingers wearing VibEye. VibEye triggers a vibration from one finger, and the vibration that has propagated through the object is sensed at the other finger. This vibration includes information about the object's identity, and we represent it using a spectrogram. Collecting the spectrograms of many objects, we formulate the object recognition problem to a classical classification problem among the images. This simple method, when tested with 20 users, shows 92.5% accuracy for 16 objects of the same shape with various materials. This material-based classifier is also extended to the recognition of everyday objects. Lastly, we demonstrate several tangible applications where VibEye provides the needed functionality while enhancing user experiences. VibEye is particularly effective for recognizing objects made of different materials, which is difficult to distinguish by other means such as light and sound.

Supplementary Material

ZIP File (paper676pvc.zip)
Preview video captions
MP4 File (paper676.mp4)
Supplemental video
MP4 File (paper676p.mp4)
Preview video

References

[1]
Chih-Chung Chang and Chih-Jen Lin. 2011. LIBSVM: A Library for Support Vector Machines. ACM Trans. Intell. Syst. Technol. 2, 3, Article 27 (May 2011), 27 pages.
[2]
Xiang 'Anthony' Chen, Stelian Coros, and Scott E. Hudson. 2018. Medley: A Library of Embeddables to Explore Rich Material Properties for 3D Printed Objects. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems (CHI '18). ACM, New York, NY, USA, Article 162, 12 pages.
[3]
Lung-Pan Cheng, Eyal Ofek, Christian Holz, Hrvoje Benko, and Andrew D. Wilson. 2017. Sparse Haptic Proxy: Touch Feedback in Virtual Environments Using a General Passive Prop. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems (CHI '17). ACM, New York, NY, USA, 3718--3728.
[4]
Jungchan Cho, Inhwan Hwang, and Songhwai Oh. 2016. VibePhone: efficient surface recognition for smartphones using vibration. Pattern Analysis and Applications 19, 1 (feb 2016), 251--265.
[5]
Youngjun Cho, Nadia Bianchi-Berthouze, Nicolai Marquardt, and Simon J. Julier. 2018. Deep Thermal Imaging: Proximate Material Type Recognition in the Wild Through Deep Learning of Spatial Surface Temperature Patterns. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems (CHI '18). ACM, New York, NY, USA, Article 2, 13 pages.
[6]
Seungmoon Choi and Katherine J. Kuchenbecker. 2013. Vibrotactile Display: Perception, Technology, and Applications. Proc. IEEE 101, 9 (2013), 2093--2104.
[7]
Waltenegus Dargie. 2009. Analysis of Time and Frequency Domain Features of Accelerometer Measurements. In Proceedings of 18th International Conference on Computer Communications and Networks (ICCCN'09). IEEE, 1--6.
[8]
Mingming Fan and Khai N. Truong. 2015. SoQr: Sonically Quantifying the Content Level Inside Containers. In Proceedings of the 2015 ACM International Joint Conference on Pervasive and Ubiquitous Computing (UbiComp '15). ACM, New York, NY, USA, 3--14.
[9]
Gene F. Franklin, J. David Powell, and Michael L. Workman. 1990. Digital Control of Dynamics Systems (second ed.). Addison-Wesley Publishing Company, Reading, MA, U.S.A.
[10]
Nobuhiro Funato and Kentaro Takemura. 2017. Grip Force Estimation by Emitting Vibration. Adjunct Publication of the 30th Annual ACM Symposium on User Interface Software and Technology - UIST '17 October, 141--142.
[11]
Chris Harrison and Scott E. Hudson. 2008. Lightweight Material Detection for Placement-aware Mobile Computing. In Proceedings of the 21st Annual ACM Symposium on User Interface Software and Technology (UIST '08). ACM, New York, NY, USA, 279--282.
[12]
Hiroyuki Kato and Kentaro Takemura. 2016. Hand Pose Estimation Based on Active Bone-conducted Sound Sensing. In Proceedings of the 2016 ACM International Joint Conference on Pervasive and Ubiquitous Computing: Adjunct (UbiComp '16). ACM, New York, NY, USA, 109--112.
[13]
Kai Kunze and Paul Lukowicz. 2007. Symbolic Object Localization Through Active Sampling of Acceleration and Sound Signatures. In Proceedings of the 9th International Conference on Ubiquitous Computing (UbiComp '07). Springer-Verlag, Berlin, Heidelberg, 163--180. http: //dl.acm.org/citation.cfm?id=1771592.1771602
[14]
Ben Lafreniere and Tovi Grossman. 2018. Blocks-to-CAD : A CrossApplication Bridge from Minecraft to 3D Modeling. In Proceedings of the 31th Annual ACM Symposium on User Interface Software and Technology (UIST '18). ACM, New York, NY, USA.
[15]
Nils Landin, Joseph M. Romano, William McMahan, and Katherine J. Kuchenbecker. 2010. Dimensional reduction of high-frequency accelerations for haptic rendering. In Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics). Vol. 6192 LNCS. Springer, Berlin, Heidelberg, 79--86.
[16]
Gierad Laput, Eric Brockmeyer, Scott E. Hudson, and Chris Harrison. 2015. Acoustruments: Passive, Acoustically-Driven, Interactive Controls for Handheld Devices. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems (CHI '15). ACM, New York, NY, USA, 2161--2170.
[17]
Gierad Laput, Xiang 'Anthony' Chen, and Chris Harrison. 2016. SweepSense: Ad Hoc Configuration Sensing Using Reflected SweptFrequency Ultrasonics. In Proceedings of the 21st International Conference on Intelligent User Interfaces (IUI '16). ACM, New York, NY, USA, 332--335.
[18]
Gierad Laput, Robert Xiao, and Chris Harrison. 2016. ViBand: HighFidelity Bio-Acoustic Sensing Using Commodity Smartwatch Accelerometers. In Proceedings of the 29th Annual Symposium on User Interface Software and Technology (UIST '16). ACM, New York, NY, USA, 321--333.
[19]
Gierad Laput, Chouchang Yang, Robert Xiao, Alanson Sample, and Chris Harrison. 2015. EM-Sense: Touch Recognition of Uninstrumented, Electrical and Electromechanical Objects. In Proceedings of the 28th Annual ACM Symposium on User Interface Software & Technology (UIST '15). ACM, New York, NY, USA, 157--166.
[20]
Lennart Ljung. 1999. System Identification: Theory for the User (second ed.). Prentice Hall, Upper Saddle River, NJ, USA.
[21]
Karen E. MacLean. 2008. Foundations of Transparency in Tactile Information Design. IEEE Transactions on Haptics 1, 2 (2008), 84--95.
[22]
Takuya Maekawa, Yasue Kishino, Yasushi Sakurai, and Takayuki Suyama. 2013. Activity Recognition with Hand-worn Magnetic Sensors. Personal Ubiquitous Comput. 17, 6 (Aug. 2013), 1085--1094.
[23]
Kenneth A Mann, Frederick W Wernere, and Andrew K Palmer. 1989. Frequency spectrum analysis of wrist motion for activities of daily living. Journal of Orthopaedic research 7, 2 (1989), 304--306.
[24]
EPS News. 2015. mCube Intros Tiny MEMS Accelerometer. (Nov. 2015). Retrieved January 7, 2019 from https://epsnews.com/2015/11/ 04/mcube-intros-tiny-mems-accelerometer/
[25]
Katsunori Ohnishi, Atsushi Kanehira, Asako Kanezaki, and Tatsuya Harada. 2016. Recognizing Activities of Daily Living with a WristMounted Camera. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 3103--3111.
[26]
Makoto Ono, Buntarou Shizuki, and Jiro Tanaka. 2013. Touch & Activate: Adding Interactivity to Existing Objects Using Active Acoustic Sensing. In Proceedings of the 26th Annual ACM Symposium on User Interface Software and Technology (UIST '13). ACM, New York, NY, USA, 31--40.
[27]
Makoto Ono, Buntarou Shizuki, and Jiro Tanaka. 2015. Sensing Touch Force Using Active Acoustic Sensing. In Proceedings of the Ninth International Conference on Tangible, Embedded, and Embodied Interaction (TEI '15). ACM, New York, NY, USA, 355--358.
[28]
Won-Hyeong Park, Eun-Jae Shin, Sungryul Yun, and Sang-Youn Kim. 2018. An Enhanced Soft Vibrotactile Actuator Based on ePVC Gel with Silicon Dioxide Nanoparticles. 11, 1 (2018), 22--29.
[29]
Won-Hyeong Park, Tae-Heon Yang, Yongjae Yoo, Seungmoon Choi, and Sang-Youn Kim. 2015. Flexible and Bendable Vibrotactile Actuator Using Electro-conductive Polyurethane. In Proceedings of the IEEE World Haptics (WHC '15). IEEE, 165--170.
[30]
Michal Piovarci, David I. W. Levin, Jason Rebello, Desai Chen, Roman Durikovic, Hanspeter Pfister, Wojciech Matusik, and Piotr Didyk. 2016. An Interaction-aware, Perceptual Model for Non-linear Elastic Objects. ACM Trans. Graph. 35, 4, Article 55 (July 2016), 13 pages.
[31]
Munehiko Sato, Shigeo Yoshida, Alex Olwal, Boxin Shi, Atsushi Hiyama, Tomohiro Tanikawa, Michitaka Hirose, and Ramesh Raskar. 2015. SpecTrans: Versatile Material Classification for Interaction with Textureless, Specular and Transparent Surfaces. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems (CHI '15). ACM, New York, NY, USA, 2191--2200.
[32]
Oliver Schneider, Karon MacLean, Colin Swindell, and Kellogg Booth. 2017. Haptic Experience Design: What Hapticians Do and Where They Need Help. International Journal of Human-Computer Studies 107 (2017), 5--21.
[33]
Yu-Chih Tung and Kang G. Shin. 2015. EchoTag: Accurate Infrastructure-Free Indoor Location Tagging with Smartphones. In Proceedings of the 21st Annual International Conference on Mobile Computing and Networking (MobiCom '15). ACM, New York, NY, USA, 525--536.
[34]
M.A. Turk and A.P. Pentland. 1991. Face recognition using eigenfaces. In Proceedings of the 1991 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR '91). IEEE Comput. Sco. Press, 586--591.
[35]
Edward J. Wang, Tien-Jui Lee, Alex Mariakakis, Mayank Goel, Sidhant Gupta, and Shwetak N. Patel. 2015. MagnifiSense: Inferring Device Interaction Using Wrist-worn Passive Magneto-inductive Sensors. In Proceedings of the 2015 ACM International Joint Conference on Pervasive and Ubiquitous Computing (UbiComp '15). ACM, New York, NY, USA, 15--26.
[36]
Xing-Dong Yang, Tovi Grossman, Daniel Wigdor, and George Fitzmaurice. 2012. Magic Finger: Always-available Input Through Finger Instrumentation. In Proceedings of the 25th Annual ACM Symposium on User Interface Software and Technology (UIST '12). ACM, New York, NY, USA, 147--156.
[37]
Hui-Shyong Yeo, Gergely Flamich, Patrick Schrempf, David HarrisBirtill, and Aaron Quigley. 2016. RadarCat: Radar Categorization for Input & Interaction. In Proceedings of the 29th Annual Symposium on User Interface Software and Technology (UIST '16). ACM, New York, NY, USA, 833--841.
[38]
Hui-Shyong Yeo, Juyoung Lee, Andrea Bianchi, David Harris-Birtill, and Aaron Quigley. 2017. SpeCam: Sensing Surface Color and Material with the Front-facing Camera of a Mobile Device. In Proceedings of the 19th International Conference on Human-Computer Interaction with Mobile Devices and Services (MobileHCI '17). ACM, New York, NY, USA, Article 25, 9 pages.
[39]
Jonas Zehnder, Espen Knoop, Moritz Bächer, and Bernhard Thomaszewski. 2017. Metasilicone: Design and Fabrication of Composite Silicone with Desired Mechanical Properties. ACM Trans. Graph. 36, 6, Article 240 (Nov. 2017), 13 pages.
[40]
Cheng Zhang, Qiuyue Xue, Anandghan Waghmare, Ruichen Meng, Sumeet Jain, Yizeng Han, Xinyu Li, Kenneth Cunefare, Thomas Ploetz, Thad Starner, Omer Inan, and Gregory D. Abowd. 2018. FingerPing: Recognizing Fine-grained Hand Poses Using Active Acoustic On-body Sensing. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems (CHI '18). ACM, New York, NY, USA, Article 437, 10 pages.
[41]
Yang Zhang, Gierad Laput, and Chris Harrison. 2018. Vibrosight: LongRange Vibrometry for Smart Environment Sensing. In Proceedings of the 31st Annual ACM Symposium on User Interface Software and Technology (UIST '18). ACM, New York, NY, USA, 225--236.
[42]
Yiwei Zhao, Lawrence H. Kim, Ye Wang, Mathieu Le Goc, and Sean Follmer. 2017. Robotic Assembly of Haptic Proxy Objects for Tangible Interaction and Virtual Reality. In Proceedings of the 2017 ACM International Conference on Interactive Surfaces and Spaces (ISS '17). ACM, New York, NY, USA, 82--91.
[43]
Yiran Zhao, Shuochao Yao, Shen Li, Shaohan Hu, Huajie Shao, and Tarek F. Abdelzaher. 2017. VibeBin: A Vibration-Based Waste Bin Level Detection System. Proc. ACM Interact. Mob. Wearable Ubiquitous Technol. 1, 3, Article 122, 22 pages.

Cited By

View all
  • (2024)ViObjectProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/36435478:1(1-26)Online publication date: 6-Mar-2024
  • (2024)TextureSightProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/36314137:4(1-27)Online publication date: 12-Jan-2024
  • (2024)EchoWrist: Continuous Hand Pose Tracking and Hand-Object Interaction Recognition Using Low-Power Active Acoustic Sensing On a WristbandProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642910(1-21)Online publication date: 11-May-2024
  • Show More Cited By

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences
CHI '19: Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems
May 2019
9077 pages
ISBN:9781450359702
DOI:10.1145/3290605
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 02 May 2019

Permissions

Request permissions for this article.

Check for updates

Author Tags

  1. augmented reality
  2. object recognition
  3. passive haptics
  4. tangible interaction
  5. vibration-based sensing
  6. virtual reality

Qualifiers

  • Research-article

Funding Sources

Conference

CHI '19
Sponsor:

Acceptance Rates

CHI '19 Paper Acceptance Rate 703 of 2,958 submissions, 24%;
Overall Acceptance Rate 6,199 of 26,314 submissions, 24%

Upcoming Conference

CHI 2025
ACM CHI Conference on Human Factors in Computing Systems
April 26 - May 1, 2025
Yokohama , Japan

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)133
  • Downloads (Last 6 weeks)7
Reflects downloads up to 17 Jan 2025

Other Metrics

Citations

Cited By

View all
  • (2024)ViObjectProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/36435478:1(1-26)Online publication date: 6-Mar-2024
  • (2024)TextureSightProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/36314137:4(1-27)Online publication date: 12-Jan-2024
  • (2024)EchoWrist: Continuous Hand Pose Tracking and Hand-Object Interaction Recognition Using Low-Power Active Acoustic Sensing On a WristbandProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642910(1-21)Online publication date: 11-May-2024
  • (2023)Z-Ring: Single-Point Bio-Impedance Sensing for Gesture, Touch, Object and User RecognitionProceedings of the 2023 CHI Conference on Human Factors in Computing Systems10.1145/3544548.3581422(1-18)Online publication date: 19-Apr-2023
  • (2021)Identifying Contact Fingers on Touch Sensitive Surfaces by Ring-Based Vibratory CommunicationThe 34th Annual ACM Symposium on User Interface Software and Technology10.1145/3472749.3474745(208-222)Online publication date: 10-Oct-2021
  • (2021)SpectroPhone: Enabling Material Surface Sensing with Rear Camera and Flashlight LEDsExtended Abstracts of the 2021 CHI Conference on Human Factors in Computing Systems10.1145/3411763.3451753(1-5)Online publication date: 8-May-2021
  • (2020)Haptics with Input: Back-EMF in Linear Resonant Actuators to Enable Touch, Pressure and Environmental AwarenessProceedings of the 33rd Annual ACM Symposium on User Interface Software and Technology10.1145/3379337.3415823(420-429)Online publication date: 20-Oct-2020

View Options

Login options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

HTML Format

View this article in HTML Format.

HTML Format

Media

Figures

Other

Tables

Share

Share

Share this Publication link

Share on social media