Seungjae Oh
Seungmoon Choi
ABSTRACT
We present VibEye: a vibration-mediated recognition system of objects for tangible interaction. A user holds an object between two fingers wearing VibEye. VibEye triggers a vibration from one finger, and the vibration that has propagated through the object is sensed at the other finger. This vibration includes information about the object's identity, and we represent it using a spectrogram. Collecting the spectrograms of many objects, we formulate the object recognition problem to a classical classification problem among the images. This simple method, when tested with 20 users, shows 92.5% accuracy for 16 objects of the same shape with various materials. This material-based classifier is also extended to the recognition of everyday objects. Lastly, we demonstrate several tangible applications where VibEye provides the needed functionality while enhancing user experiences. VibEye is particularly effective for recognizing objects made of different materials, which is difficult to distinguish by other means such as light and sound.
Supplemental Material
- Chih-Chung Chang and Chih-Jen Lin. 2011. LIBSVM: A Library for Support Vector Machines. ACM Trans. Intell. Syst. Technol. 2, 3, Article 27 (May 2011), 27 pages. Google Scholar
Digital Library
- Xiang 'Anthony' Chen, Stelian Coros, and Scott E. Hudson. 2018. Medley: A Library of Embeddables to Explore Rich Material Properties for 3D Printed Objects. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems (CHI '18). ACM, New York, NY, USA, Article 162, 12 pages. Google ScholarDigital Library
- Lung-Pan Cheng, Eyal Ofek, Christian Holz, Hrvoje Benko, and Andrew D. Wilson. 2017. Sparse Haptic Proxy: Touch Feedback in Virtual Environments Using a General Passive Prop. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems (CHI '17). ACM, New York, NY, USA, 3718--3728. Google ScholarDigital Library
- Jungchan Cho, Inhwan Hwang, and Songhwai Oh. 2016. VibePhone: efficient surface recognition for smartphones using vibration. Pattern Analysis and Applications 19, 1 (feb 2016), 251--265. Google ScholarDigital Library
- Youngjun Cho, Nadia Bianchi-Berthouze, Nicolai Marquardt, and Simon J. Julier. 2018. Deep Thermal Imaging: Proximate Material Type Recognition in the Wild Through Deep Learning of Spatial Surface Temperature Patterns. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems (CHI '18). ACM, New York, NY, USA, Article 2, 13 pages. Google ScholarDigital Library
- Seungmoon Choi and Katherine J. Kuchenbecker. 2013. Vibrotactile Display: Perception, Technology, and Applications. Proc. IEEE 101, 9 (2013), 2093--2104.Google ScholarCross Ref
- Waltenegus Dargie. 2009. Analysis of Time and Frequency Domain Features of Accelerometer Measurements. In Proceedings of 18th International Conference on Computer Communications and Networks (ICCCN'09). IEEE, 1--6. Google ScholarDigital Library
- Mingming Fan and Khai N. Truong. 2015. SoQr: Sonically Quantifying the Content Level Inside Containers. In Proceedings of the 2015 ACM International Joint Conference on Pervasive and Ubiquitous Computing (UbiComp '15). ACM, New York, NY, USA, 3--14. Google ScholarDigital Library
- Gene F. Franklin, J. David Powell, and Michael L. Workman. 1990. Digital Control of Dynamics Systems (second ed.). Addison-Wesley Publishing Company, Reading, MA, U.S.A.Google Scholar
- Nobuhiro Funato and Kentaro Takemura. 2017. Grip Force Estimation by Emitting Vibration. Adjunct Publication of the 30th Annual ACM Symposium on User Interface Software and Technology - UIST '17 October, 141--142. Google ScholarDigital Library
- Chris Harrison and Scott E. Hudson. 2008. Lightweight Material Detection for Placement-aware Mobile Computing. In Proceedings of the 21st Annual ACM Symposium on User Interface Software and Technology (UIST '08). ACM, New York, NY, USA, 279--282. Google ScholarDigital Library
- Hiroyuki Kato and Kentaro Takemura. 2016. Hand Pose Estimation Based on Active Bone-conducted Sound Sensing. In Proceedings of the 2016 ACM International Joint Conference on Pervasive and Ubiquitous Computing: Adjunct (UbiComp '16). ACM, New York, NY, USA, 109--112. Google ScholarDigital Library
- Kai Kunze and Paul Lukowicz. 2007. Symbolic Object Localization Through Active Sampling of Acceleration and Sound Signatures. In Proceedings of the 9th International Conference on Ubiquitous Computing (UbiComp '07). Springer-Verlag, Berlin, Heidelberg, 163--180. http: //dl.acm.org/citation.cfm?id=1771592.1771602 Google ScholarDigital Library
- Ben Lafreniere and Tovi Grossman. 2018. Blocks-to-CAD : A CrossApplication Bridge from Minecraft to 3D Modeling. In Proceedings of the 31th Annual ACM Symposium on User Interface Software and Technology (UIST '18). ACM, New York, NY, USA. Google ScholarDigital Library
- Nils Landin, Joseph M. Romano, William McMahan, and Katherine J. Kuchenbecker. 2010. Dimensional reduction of high-frequency accelerations for haptic rendering. In Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics). Vol. 6192 LNCS. Springer, Berlin, Heidelberg, 79--86. Google ScholarDigital Library
- Gierad Laput, Eric Brockmeyer, Scott E. Hudson, and Chris Harrison. 2015. Acoustruments: Passive, Acoustically-Driven, Interactive Controls for Handheld Devices. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems (CHI '15). ACM, New York, NY, USA, 2161--2170. Google ScholarDigital Library
- Gierad Laput, Xiang 'Anthony' Chen, and Chris Harrison. 2016. SweepSense: Ad Hoc Configuration Sensing Using Reflected SweptFrequency Ultrasonics. In Proceedings of the 21st International Conference on Intelligent User Interfaces (IUI '16). ACM, New York, NY, USA, 332--335. Google ScholarDigital Library
- Gierad Laput, Robert Xiao, and Chris Harrison. 2016. ViBand: HighFidelity Bio-Acoustic Sensing Using Commodity Smartwatch Accelerometers. In Proceedings of the 29th Annual Symposium on User Interface Software and Technology (UIST '16). ACM, New York, NY, USA, 321--333. Google ScholarDigital Library
- Gierad Laput, Chouchang Yang, Robert Xiao, Alanson Sample, and Chris Harrison. 2015. EM-Sense: Touch Recognition of Uninstrumented, Electrical and Electromechanical Objects. In Proceedings of the 28th Annual ACM Symposium on User Interface Software & Technology (UIST '15). ACM, New York, NY, USA, 157--166. Google ScholarDigital Library
- Lennart Ljung. 1999. System Identification: Theory for the User (second ed.). Prentice Hall, Upper Saddle River, NJ, USA. Google ScholarDigital Library
- Karen E. MacLean. 2008. Foundations of Transparency in Tactile Information Design. IEEE Transactions on Haptics 1, 2 (2008), 84--95. Google ScholarDigital Library
- Takuya Maekawa, Yasue Kishino, Yasushi Sakurai, and Takayuki Suyama. 2013. Activity Recognition with Hand-worn Magnetic Sensors. Personal Ubiquitous Comput. 17, 6 (Aug. 2013), 1085--1094. Google ScholarDigital Library
- Kenneth A Mann, Frederick W Wernere, and Andrew K Palmer. 1989. Frequency spectrum analysis of wrist motion for activities of daily living. Journal of Orthopaedic research 7, 2 (1989), 304--306.Google ScholarCross Ref
- EPS News. 2015. mCube Intros Tiny MEMS Accelerometer. (Nov. 2015). Retrieved January 7, 2019 from https://epsnews.com/2015/11/ 04/mcube-intros-tiny-mems-accelerometer/Google Scholar
- Katsunori Ohnishi, Atsushi Kanehira, Asako Kanezaki, and Tatsuya Harada. 2016. Recognizing Activities of Daily Living with a WristMounted Camera. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 3103--3111.Google ScholarCross Ref
- Makoto Ono, Buntarou Shizuki, and Jiro Tanaka. 2013. Touch & Activate: Adding Interactivity to Existing Objects Using Active Acoustic Sensing. In Proceedings of the 26th Annual ACM Symposium on User Interface Software and Technology (UIST '13). ACM, New York, NY, USA, 31--40. Google ScholarDigital Library
- Makoto Ono, Buntarou Shizuki, and Jiro Tanaka. 2015. Sensing Touch Force Using Active Acoustic Sensing. In Proceedings of the Ninth International Conference on Tangible, Embedded, and Embodied Interaction (TEI '15). ACM, New York, NY, USA, 355--358. Google ScholarDigital Library
- Won-Hyeong Park, Eun-Jae Shin, Sungryul Yun, and Sang-Youn Kim. 2018. An Enhanced Soft Vibrotactile Actuator Based on ePVC Gel with Silicon Dioxide Nanoparticles. 11, 1 (2018), 22--29.Google Scholar
- Won-Hyeong Park, Tae-Heon Yang, Yongjae Yoo, Seungmoon Choi, and Sang-Youn Kim. 2015. Flexible and Bendable Vibrotactile Actuator Using Electro-conductive Polyurethane. In Proceedings of the IEEE World Haptics (WHC '15). IEEE, 165--170.Google Scholar
- Michal Piovarci, David I. W. Levin, Jason Rebello, Desai Chen, Roman Durikovic, Hanspeter Pfister, Wojciech Matusik, and Piotr Didyk. 2016. An Interaction-aware, Perceptual Model for Non-linear Elastic Objects. ACM Trans. Graph. 35, 4, Article 55 (July 2016), 13 pages. Google ScholarDigital Library
- Munehiko Sato, Shigeo Yoshida, Alex Olwal, Boxin Shi, Atsushi Hiyama, Tomohiro Tanikawa, Michitaka Hirose, and Ramesh Raskar. 2015. SpecTrans: Versatile Material Classification for Interaction with Textureless, Specular and Transparent Surfaces. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems (CHI '15). ACM, New York, NY, USA, 2191--2200. Google ScholarDigital Library
- Oliver Schneider, Karon MacLean, Colin Swindell, and Kellogg Booth. 2017. Haptic Experience Design: What Hapticians Do and Where They Need Help. International Journal of Human-Computer Studies 107 (2017), 5--21. Google ScholarDigital Library
- Yu-Chih Tung and Kang G. Shin. 2015. EchoTag: Accurate Infrastructure-Free Indoor Location Tagging with Smartphones. In Proceedings of the 21st Annual International Conference on Mobile Computing and Networking (MobiCom '15). ACM, New York, NY, USA, 525--536. Google ScholarDigital Library
- M.A. Turk and A.P. Pentland. 1991. Face recognition using eigenfaces. In Proceedings of the 1991 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR '91). IEEE Comput. Sco. Press, 586--591.Google Scholar
- Edward J. Wang, Tien-Jui Lee, Alex Mariakakis, Mayank Goel, Sidhant Gupta, and Shwetak N. Patel. 2015. MagnifiSense: Inferring Device Interaction Using Wrist-worn Passive Magneto-inductive Sensors. In Proceedings of the 2015 ACM International Joint Conference on Pervasive and Ubiquitous Computing (UbiComp '15). ACM, New York, NY, USA, 15--26. Google ScholarDigital Library
- Xing-Dong Yang, Tovi Grossman, Daniel Wigdor, and George Fitzmaurice. 2012. Magic Finger: Always-available Input Through Finger Instrumentation. In Proceedings of the 25th Annual ACM Symposium on User Interface Software and Technology (UIST '12). ACM, New York, NY, USA, 147--156. Google ScholarDigital Library
- Hui-Shyong Yeo, Gergely Flamich, Patrick Schrempf, David HarrisBirtill, and Aaron Quigley. 2016. RadarCat: Radar Categorization for Input & Interaction. In Proceedings of the 29th Annual Symposium on User Interface Software and Technology (UIST '16). ACM, New York, NY, USA, 833--841. Google ScholarDigital Library
- Hui-Shyong Yeo, Juyoung Lee, Andrea Bianchi, David Harris-Birtill, and Aaron Quigley. 2017. SpeCam: Sensing Surface Color and Material with the Front-facing Camera of a Mobile Device. In Proceedings of the 19th International Conference on Human-Computer Interaction with Mobile Devices and Services (MobileHCI '17). ACM, New York, NY, USA, Article 25, 9 pages. Google ScholarDigital Library
- Jonas Zehnder, Espen Knoop, Moritz Bächer, and Bernhard Thomaszewski. 2017. Metasilicone: Design and Fabrication of Composite Silicone with Desired Mechanical Properties. ACM Trans. Graph. 36, 6, Article 240 (Nov. 2017), 13 pages. Google ScholarDigital Library
- Cheng Zhang, Qiuyue Xue, Anandghan Waghmare, Ruichen Meng, Sumeet Jain, Yizeng Han, Xinyu Li, Kenneth Cunefare, Thomas Ploetz, Thad Starner, Omer Inan, and Gregory D. Abowd. 2018. FingerPing: Recognizing Fine-grained Hand Poses Using Active Acoustic On-body Sensing. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems (CHI '18). ACM, New York, NY, USA, Article 437, 10 pages. Google ScholarDigital Library
- Yang Zhang, Gierad Laput, and Chris Harrison. 2018. Vibrosight: LongRange Vibrometry for Smart Environment Sensing. In Proceedings of the 31st Annual ACM Symposium on User Interface Software and Technology (UIST '18). ACM, New York, NY, USA, 225--236. Google ScholarDigital Library
- Yiwei Zhao, Lawrence H. Kim, Ye Wang, Mathieu Le Goc, and Sean Follmer. 2017. Robotic Assembly of Haptic Proxy Objects for Tangible Interaction and Virtual Reality. In Proceedings of the 2017 ACM International Conference on Interactive Surfaces and Spaces (ISS '17). ACM, New York, NY, USA, 82--91. Google ScholarDigital Library
- Yiran Zhao, Shuochao Yao, Shen Li, Shaohan Hu, Huajie Shao, and Tarek F. Abdelzaher. 2017. VibeBin: A Vibration-Based Waste Bin Level Detection System. Proc. ACM Interact. Mob. Wearable Ubiquitous Technol. 1, 3, Article 122, 22 pages. Google ScholarDigital Library
Index Terms
- VibEye: Vibration-Mediated Object Recognition for Tangible Interactive Applications
Recommendations
Physical hand interaction for controlling multiple virtual objects in virtual reality
IWISC '18: Proceedings of the 3rd International Workshop on Interactive and Spatial ComputingImmersive virtual reality systems often strive to support realistic physical interaction through direct hand control, but it is not often possible to configure the physical environment to exactly match the virtual environment. We study methods that ...
VR Grabbers: Ungrounded Haptic Retargeting for Precision Grabbing Tools
UIST '18: Proceedings of the 31st Annual ACM Symposium on User Interface Software and TechnologyHaptic feedback in VR is important for realistic simulation in virtual reality. However, recreating the haptic experience for hand tools in VR traditionally requires hardware with precise actuators, adding complexity to the system. We propose Ungrounded ...
Developing an augmented reality racing game
INTETAIN '08: Proceedings of the 2nd international conference on INtelligent TEchnologies for interactive enterTAINmentAugmented reality (AR) makes it possible to create games in which virtual objects are overlaid on the real world, and real objects are tracked and used to control virtual ones. We describe the development of an AR racing game created by modifying an ...
Comments