skip to main content
10.1145/3229434.3229453acmconferencesArticle/Chapter ViewAbstractPublication PagesmobilehciConference Proceedingsconference-collections
research-article

Knock knock, what's there: converting passive objects into customizable smart controllers

Published: 03 September 2018 Publication History

Abstract

Knocking is a way of interacting with everyday objects. We introduce BeatIt, a novel technique that allows users to use passive, everyday objects to control a smart environment by recognizing the sounds generated from knocking on the objects. BeatIt uses a BeatSet, a series of percussive sound samples, to represent the sound signature of knocking on an object. A user associates a BeatSet with an event. For example, a user can associate the BeatSet of knocking on a door with the event of turning on the lights. Decoder, a signal-processing module, classifies the sound signals into one of the recorded BeatSets, and then triggers the associated event. Unlike prior work, BeatIt can be implemented on microphone-enabled commodity devices. Our user studies with 12 participants showed that our proof-of-concept implementation based on a smartwatch could accurately classify eight BeatSets using a user-independent classifier.

References

[1]
Amazon. Alexa. Retrieved April 2, 2017 from https://developer.amazon.com/alexa
[2]
Amazon. Amazon Dash Button. Retrieved April 2, 2017 from https://www.amazon.com/Dash-Buttons/b?ie=UTF8&node=10667898011
[3]
Brian Amento, Will Hill, and Loren Terveen. 2002. The sound of one hand. In CHI '02 Extended Abstracts on Human Factors in Computing Systems (CHI EA '02), 724--725.
[4]
Apple. Siri. Retrieved April 2, 2017 from http://www.apple.com/ios/siri/
[5]
Dario Bonino, Emiliano Castellina, Fulvio Corno, and Luigi De Russis. 2011. DOGeye: Controlling your home with eye interaction. Interacting with Computers 23, 5: 484--498.
[6]
John Brooke. 1996. SUS-A Quick and Dirty Usability Scale. Usability evaluation in industry.
[7]
Aj J Bernheim Brush, Bongshin Lee, Ratul Mahajan, Sharad Agarwal, Stefan Saroiu, and Colin Dixon. 2011. Home Automation in the Wild: Challenges and Opportunities. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '11), 2115--2124.
[8]
Michael Buettner, Richa Prasad, Matthai Philipose, and David Wetherall. 2009. Recognizing Daily Activities with RFID-Based Sensors. In Proceedings of the 11th international conference on Ubiquitous computing (UbiComp '09), 51--60.
[9]
Canary. Canary - A complete security system in a single device. Retrieved April 2, 2017 from https://canary.is/
[10]
Roberta Carabalona, Ferdinando Grossi, Adam Tessadri, Paolo Castiglioni, Antonio Caracciolo, and Ilaria de Munari. 2012. Light on! Real world evaluation of a P300-based brain-computer interface (BCI) for environment control in a smart home. Ergonomics 55, 5: 552--563.
[11]
Eunjung Choi, Sunghyuk Kwon, Donghun Lee, Hogin Lee, and Min K Chung. 2014. Towards successful user interaction with systems: Focusing on user-derived gestures for smart home systems. Applied Ergonomics 45: 1196--1207.
[12]
Brian Clarkson, Nitin Sawhney, and Alex Pentland. 1998. Auditory Context Awareness via Wearable Computing. In Proceedings Of The 1998 Workshop On Perceptual User Interfaces (PUI 98), 600.
[13]
Gabe Cohn, Daniel Morris, Shwetak N Patel, and Desney S Tan. 2011. Your Noise is My Command: Sensing Gestures Using the Body as an Antenna. In Proceedings of the 2011 annual conference on Human factors in computing systems (CHI '11), 791--800.
[14]
A.K. Dey, Raffay Hamid, Chris Beckmann, Ian Li, and Daniel Hsu. 2004. a CAPpella: programming by demonstration of context-aware applications. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '04), 33--40.
[15]
Anind K. Dey, Timothy Sohn, Sara Streng, and Justin Kodama. 2006. iCAP: Interactive prototyping of context-aware applications. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics) 3968 LNCS: 254--271.
[16]
Alain Dufaux. 2001. Detection and Recognition of Impulsive Sound Signals. January.
[17]
Günter Edlinger, Clemens Holzner, and Christoph Guger. 2011. A hybrid brain-computer interface for smart home control. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics) 6762 LNCS, PART 2: 417--426.
[18]
W.K. Edwards and R.E. Grinter. 2001. At Home with Ubiquitous Computing: Seven Challenges. In International Conference on Ubiquitous Computing, 256--272.
[19]
James Fogarty, Carolyn Au, and Scott E Hudson. 2006. Sensing from the Basement: A Feasibility Study of Unobtrusive and Low-Cost Home Activity Recognition. In 19th annual ACM symposium on User interface software and technology (UIST'06), 91--100.
[20]
Sarthak Ghosh, Gilles Bailly, Robin Despouys, Eric Lecolinet, and Rémi Sharrock. 2015. SuperVision: Spatial Control of Connected Objects in a Smart Home. In Proceedings of the 33rd Annual ACM Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA '15), 2079--2084.
[21]
Tobias Grosse-Puppendahl, Sebastian Herber, Raphael Wimmer, Frank Englert, Sebastian Beck, Julian von Wilmsdorff, Reiner Wichert, and Arjan Kuijper. 2014. Capacitive Near-Field Communication for Ubiquitous Interaction and Perception. In Proceedings of the 2014 ACM International Joint Conference on Pervasive and Ubiquitous Computing (UbiComp '14), 231--242.
[22]
Chris Harrison and Scott E Hudson. 2008. Scratch input: creating large, inexpensive, unpowered and mobile finger input surfaces. In Proceedings of the 21st annual ACM symposium on User interface software and technology (UIST '08), 205--208.
[23]
Chris Harrison, Julia Schwarz, and Scott E. Hudson. 2011. TapSense: Enhancing Finger Interaction on Touch Surfaces. In Proceedings of the 24th annual ACM symposium on User interface software and technology (UIST '11), 627--634.
[24]
Chris Harrison, Desney Tan, and Dan Morris. 2010. Skinput: Appropriating the Body as an Input Surface. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '10), 453--462.
[25]
Steve Hodges, Alan Thorne, Hugo Mallinson, and Christian Floerkemeier. 2007. Assessing and optimizing the range of uhf rfid to enable real-world pervasive computing applications. Pervasive Computing 4480: 280--297.
[26]
iFixit. Motorola Moto 360 Teardown. Retrieved April 2, 2017 from https://www.ifixit.com/Teardown/Motorola+Moto+360+Teardown/28891
[27]
Stephen S. Intille. 2002. Designing a home of the future. IEEE Pervasive Computing 1, 76--82.
[28]
Julie a. Kientz, Shwetak N. Patel, Brian Jones, Ed Price, Elizabeth D. Mynatt, and Gregory D. Abowd. 2008. The Georgia Tech Aware home. In CHI '08 Extended Abstracts on Human Factors in Computing Systems (CHI EA '08), 3675--3680.
[29]
Hark-Joon Kim, Kyung-Ho Jeong, Seon-Kyo Kim, and Tack-Don Han. 2011. Ambient Wall: Smart Wall Display Interface Which Can Be Controlled By Simple Gesture for Smart Home. In SIGGRAPH Asia 2011 Sketches (SA '11), Article 1.
[30]
Knocki. Turn any surface into a remote controller. Retrieved April 2, 2017 from https://knocki.com/
[31]
Christine Kühnel, Tilo Westermann, Fabian Hemmert, Sven Kratz, Alexander Müller, and Sebastian Oller. 2011. I'm home: Defining and evaluating a gesture set for smart-home control. Journal of Human Computer Studies 69: 693--704.
[32]
Gierad Laput, Walter S Lasecki, Jason Wiese, Robert Xiao, Jeffrey P Bigham, and Chris Harrison. 2015. Zensors: Adaptive, rapidly deployable, human-intelligent sensor feeds. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems (CHI '15), 1935--1944.
[33]
Gierad Laput, Robert Xiao, and Chris Harrison. 2016. ViBand: High-Fidelity Bio-Acoustic Sensing Using Commodity Smartwatch Accelerometers. In Proceedings of the 29th Annual Symposium on User Interface Software and Technology - UIST '16, 321--333.
[34]
Gierad Laput, Chouchang Yang, Robert Xiao, and Alanson Sample. 2015. EM-Sense: Touch Recognition of Uninstrumented, Electrical and Electromechanical Objects. Proceedings of the 28th Annual ACM Symposium on User Interface Software & Technology (UIST '15): 157--166.
[35]
Jisoo Lee, Luis Garduño, Erin Walker, and Winslow Burleson. 2013. A tangible programming tool for creation of context-aware applications. In Proceedings of the 2013 ACM international joint conference on Pervasive and ubiquitous computing (UbiComp '13), 391--400.
[36]
Hanchuan Li, Can Ye, and Alanson P Sample. 2015. IDSense: A Human Object Interaction Detection System Based on Passive UHF RFID. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems (CHI '15), 2555--2564.
[37]
Pedro Lopes, Ricardo Jota, and Joaquim a. Jorge. 2011. Augmenting touch interaction through acoustic sensing. Proceedings of the ACM International Conference on Interactive Tabletops and Surfaces (ITS '11): 53.
[38]
Hong Lu, Wei Pan, Nicholas Lane, Tanzeem Choudhury, and Andrew Campbell. 2009. SoundSense: scalable sound sensing for people-centric applications on mobile phones. In Proceedings of the 7th international conference on Mobile systems, applications, and services (MobiSys '09), 165--178.
[39]
Takuya Maekawa, Yasue Kishino, Yasushi Sakurai, and Takayuki Suyama. Recognizing Use of Electrical Devices using Magnetic Sensors Attached to Hands. Pervasive Computing.
[40]
Takuya Maekawa, Yasue Kishino, Yutaka Yanagisawa, and Yasushi Sakurai. 2012. Recognizing handheld electrical device usage with hand-worn coil of wire. In Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), 234--252.
[41]
Sarah Mennicken, Jo Vermeulen, and Elaine M Huang. From Today's Augmented Houses to Tomorrow's Smart Homes: New Directions for Home Automation Research. In Proceedings of the 2014 ACM International Joint Conference on Pervasive and Ubiquitous Computing (UbiComp '14).
[42]
Sarah Mennicken, Oliver Zihler, Frida Juldaschewa, Veronika Molnar, David Aggeler, and Elaine May Huang. 2016. "It's like living with a friendly stranger" In Proceedings of the 2016 ACM International Joint Conference on Pervasive and Ubiquitous Computing - UbiComp '16, 120--131.
[43]
Microsoft. Cortana. Retrieved April 2, 2017 from https://www.microsoft.com/en-us/mobile/experiences/cortana/
[44]
Roderick Murray-Smith, John Williamson, Stephen Hughes, and Torben Quaade. 2008. Stane: Synthesized Surfaces for Tactile Input. Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '08): 1299--1302.
[45]
Nest. Meet the Nest Learning Thermostat | Nest. Retrieved April 2, 2017 from https://nest.com/thermostat/meet-nest-thermostat/
[46]
Makoto Ono, Buntarou Shizuki, and Jiro Tanaka. 2013. Touch & activate: adding interactivity to existing objects using active acoustic sensing. In Proceedings of the 26th annual ACM symposium on User interface software and technology (UIST '13), 31--40.
[47]
Shwetak N Patel and Gregory D Abowd. 2003. A 2-Way Laser-Assisted Selection Scheme for Handhelds in a Physical Environment. Proceedings of The International Conference on Ubiquitous Computing (UbiComp '03) 2864: 200--207.
[48]
F Pedregosa, G Varoquaux, and A Gramfort. 2011. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research 12: 2825--2830.
[49]
Perseus. Meet Perseus, the next generation mirror. Retrieved April 2, 2017 from http://www.perseusmirrors.com/
[50]
Xiaofeng Ren and Matthai Philipose. Egocentric Recognition of Handled Objects: Benchmark and Analysis. In Computer Vision and Pattern Recognition Workshops, 2009. (CVPR Workshops 2009).
[51]
Gabriel Reyes, W Keith Edwards, Dingtian Zhang, Sarthak Ghosh, Pratik Shah, Jason Wu, Aman Parnami, Bailey Bercik, Thad Starner, and Gregory D Abowd. 2016. Whoosh. In Proceedings of the 2016 ACM International Symposium on Wearable Computers (ISWC '16), 120--127.
[52]
Luigi De Russis and Fulvio Corno. 2015. HomeRules: A Tangible End-User Programming Interface for Smart Homes. In Proceedings of the 33rd Annual ACM Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA '15), 2109--2114.
[53]
Samsung. Smart Home - SmartThings | Samsung US. Retrieved April 2, 2017 from http://www.samsung.com/us/smart-home/smartthings/
[54]
Dominik Schmidt, David Molyneaux, and Xiang Cao. 2012. PICOntrol: Using a Handheld Projector for Direct Control of Physical Devices through Visible Light. Proceedings of the 25th annual ACM symposium on User interface software and technology (UIST '12) 2: 379--388.
[55]
Thomas Seifried, Michael Haller, Stacey D Scott, Florian Perteneder, Christian Rendl, Daisuke Sakamoto, and Masahiko Inami. 2009. CRISTAL: A Collaborative Home Media and Device Controller Based on a Multi-touch Display. In Proceedings of the ACM International Conference on Interactive Tabletops and Surfaces (ITS '09), 33.
[56]
Lei Shi. 2015. Talkabel: A Labeling Method for 3D Printed Models. In Proceedings of the 17th International ACM SIGACCESS Conference on Computers & Accessibility (ASSETS '15), 361--362.
[57]
Lei Shi, Idan Zelzer, Catherine Feng, and Shiri Azenkot. 2016. Tickers and Talker: An Accessible Labeling Toolkit for 3D Printed Models. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems (CHI '16), 4896--4907.
[58]
smarter. Coffee | The connected kitchen - a new way to start your day - Smarter. Retrieved April 2, 2017 from http://smarter.am/coffee/
[59]
Joshua R. Smith, Kenneth P. Fishkin, Bing Jiang, Alexander Mamishev, Matthai Philipose, Adam D. Rea, Sumit Roy, and Kishore Sundara-Rajan. 2005. RFID-based techniques for human-activity detection. Communications of the ACM 48, 9: 39.
[60]
Colin Swindells, Kori M. Inkpen, John C. Dill, and Melanie Tory. 2002. That one there! Pointing to establish device identity. In Proceedings of the 15th annual ACM symposium on User interface software and technology (UIST '02), 151--160.
[61]
Kentaro Takemura, Akihiro Ito, Jun Takamatsu, and Tsukasa Ogasawara. 2011. Active bone-conducted sound sensing for wearable interfaces. In 24th annual ACM Symposium Sdjunct on User Interface Software and Technology (UIST '11 Adjunct), 53--54.
[62]
Masayuki Tani, Kimiya Yamaashi, and Koichiro Tanikoshi. 1992. Object-oriented video: interaction with real-world objects through live video. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '92), 593--598.
[63]
KhaiN. N Truong, ElaineM. M Huang, and GregoryD. D Abowd. 2004. CAMP: A Magnetic Poetry Interface for End-User Programming of Capture Applications for the Home. In UbiComp 2004. 143--160.
[64]
Blase Ur, Elyse McManus, Melwyn Pak Yong Ho, and Michael L Littman. 2014. Practical trigger-action programming in the smart home. In Proceedings of the 32nd annual ACM conference on Human factors in computing systems (CHI '14), 803--812.
[65]
Jamie A Ward, Student Member, Paul Lukowicz, and Gerhard Tro. 2006. Activity Recognition of Assembly Tasks Using Body-Worn Microphones and Accelerometers. IEEE Transactions on Pattern Analysis and Machine Intelligence 28, 10: 1553--1567.
[66]
M Weiser, R Gold, J S Brown, Bob Sprague, and Richard Bruce. 1999. The origins of ubiquitous computing research at PARC. IBM Systems Journal 38, 4: 693--696.
[67]
Jacob Otto Wobbrock. 2009. TapSongs: Tapping Rhythm-Based Passwords on a Single Binary Sensor. In Proceedings of the 22nd annual ACM symposium on User interface software and technology (UIST '09), 93--96.
[68]
Yang Zhang, Gierad Laput, and Chris Harrison. 2017. Electrick: Low - Cost Touch Sensing Using Electric Field Tomography. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems (CHI '17), 1--14.
[69]
Yi Zhao, Anthony LaMarca, and Joshua R. Smith. 2014. A battery-free object localization and motion sensing platform. Proceedings of the 2014 ACM International Joint Conference on Pervasive and Ubiquitous Computing (UbiComp '14 Adjunct): 255--259.
[70]
Zoltan Kolozsvari. Cooking up an alternative: making a "knocking gesture" controlled app - Code Voyagers. Retrieved from http://codevoyagers.com/2016/07/22/cooking-up-an-alternative-making-a-knocking-gesture-controlled-app/

Cited By

View all
  • (2024)Acoustic-based Recognition of Multiple Objects with Overlapped Impact SoundsProceedings of the 2024 2nd International Conference on Communications, Computing and Artificial Intelligence10.1145/3676581.3676586(28-35)Online publication date: 21-Jun-2024
  • (2024)Toolkit Design for Building Camera Sensor-Driven DIY Smart HomesCompanion of the 2024 on ACM International Joint Conference on Pervasive and Ubiquitous Computing10.1145/3675094.3678363(256-261)Online publication date: 5-Oct-2024
  • (2024)ViObjectProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/36435478:1(1-26)Online publication date: 6-Mar-2024
  • Show More Cited By

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences
MobileHCI '18: Proceedings of the 20th International Conference on Human-Computer Interaction with Mobile Devices and Services
September 2018
552 pages
ISBN:9781450358989
DOI:10.1145/3229434
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 03 September 2018

Permissions

Request permissions for this article.

Check for updates

Author Tags

  1. interacting with passive objects
  2. smart environments

Qualifiers

  • Research-article

Conference

MobileHCI '18
Sponsor:

Acceptance Rates

Overall Acceptance Rate 202 of 906 submissions, 22%

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)57
  • Downloads (Last 6 weeks)6
Reflects downloads up to 14 Feb 2025

Other Metrics

Citations

Cited By

View all
  • (2024)Acoustic-based Recognition of Multiple Objects with Overlapped Impact SoundsProceedings of the 2024 2nd International Conference on Communications, Computing and Artificial Intelligence10.1145/3676581.3676586(28-35)Online publication date: 21-Jun-2024
  • (2024)Toolkit Design for Building Camera Sensor-Driven DIY Smart HomesCompanion of the 2024 on ACM International Joint Conference on Pervasive and Ubiquitous Computing10.1145/3675094.3678363(256-261)Online publication date: 5-Oct-2024
  • (2024)ViObjectProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/36435478:1(1-26)Online publication date: 6-Mar-2024
  • (2024)Surtify: A Smart Surface Identification System Based on Multi-dimensional Acoustic Dispersion2024 International Conference on Artificial Intelligence of Things and Systems (AIoTSys)10.1109/AIoTSys63104.2024.10780602(1-8)Online publication date: 17-Oct-2024
  • (2024)On the Acoustic-Based Recognition of Multiple Objects Using Overlapped Impact SoundsIEEE Access10.1109/ACCESS.2024.345942312(135651-135666)Online publication date: 2024
  • (2024)EchoTap: Non-Verbal Sound Interaction with Knock and Tap GesturesInternational Journal of Human–Computer Interaction10.1080/10447318.2024.2348837(1-22)Online publication date: 3-Jun-2024
  • (2023)Potential and Challenges of DIY Smart Homes with an ML-intensive Camera SensorProceedings of the 2023 CHI Conference on Human Factors in Computing Systems10.1145/3544548.3581462(1-19)Online publication date: 19-Apr-2023
  • (2023)Knock the Reality: Virtual Interface Registration in Mixed Reality2023 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops (VRW)10.1109/VRW58643.2023.00150(615-616)Online publication date: Mar-2023
  • (2022)Connecting Textiles: Exploring Textile Interior Surfaces for Power Supply, Communication and User Interaction in the IoTProceedings of the 12th International Conference on the Internet of Things10.1145/3567445.3567455(88-95)Online publication date: 7-Nov-2022
  • (2021)Identifying Contact Fingers on Touch Sensitive Surfaces by Ring-Based Vibratory CommunicationThe 34th Annual ACM Symposium on User Interface Software and Technology10.1145/3472749.3474745(208-222)Online publication date: 10-Oct-2021
  • Show More Cited By

View Options

Login options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Figures

Tables

Media

Share

Share

Share this Publication link

Share on social media