research-article

SurfaceLink: using inertial and acoustic sensing to enable multi-device interaction on a surface

Authors:

Md. Tanvir Islam Aumi,

Gaetano Borriello,

Bo BegoleAuthors Info & Claims

CHI '14: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems

Pages 1387 - 1396

https://doi.org/10.1145/2556288.2557120

Published: 26 April 2014 Publication History

Abstract

We present SurfaceLink, a system where users can make natural surface gestures to control association and information transfer among a set of devices that are placed on a mutually shared surface (e.g., a table). SurfaceLink uses a combination of on-device accelerometers, vibration motors, speakers and microphones (and, optionally, an off-device contact microphone for greater sensitivity) to sense gestures performed on the shared surface. In a controlled evaluation with 10 participants, SurfaceLink detected the presence of devices on the same surface with 97.7% accuracy, their relative arrangement with 89.4% accuracy, and various single- and multi-touch surface gestures with an average accuracy of 90.3%. A usability analysis showed that SurfaceLink has advantages over current multi-device interaction techniques in a number of situations.

Supplementary Material

suppl.mov (pn0896-file3.mp4)

Supplemental video

Download
87.11 MB

MP4 File (p1387-sidebyside.mp4)

Download
189.37 MB

References

[1]

Arentz, W. A. and Bandara, U. Near ultrasonic directional data transfer for modern smartphones. Proc. UbiComp'11.

Digital Library

[2]

Balfanz, D., Smetters, D. K., Stewart, P., and Wong, H. C. Talking To Strangers: Authentication in Ad-Hoc Wireless Networks.

[3]

Butler, A., Izadi, S., and Hodges, S. SideSight: Multi-"touch" interaction around small devices. Proc. UIST'08.

Digital Library

[4]

Crevoisier, A. and Polotti, P. Tangible acoustic Interfaces and their applications for the design of new musical instruments. Proc. Nime'05.

Digital Library

[5]

Cuypers, T., Francken, Y., Vanaken, C., Reeth, F. V., and Bekaert, P. Smartphone localization on interactive surfaces using the built-in camera. Proc. Procams'09.

[6]

Dearman, D., Guy, R. T., and Truong, K. N. Determining the orientation of proximate mobile devices using their back facing camera. Proc. CHI'12.

Digital Library

[7]

Goel, M., Wobbrock, J. O., and Patel, S. N. GripSense: Using built-in sensors to detect hand posture and pressure on commodity mobile phones. Proc. UIST'12.

Digital Library

[8]

Harrison, C. and Hudson, S. E. Scratch Input: Creating Large, Inexpensive, Unpowered and Mobile Finger Input Surfaces. Proc. UIST'08.

Digital Library

[9]

Harrison, C. and Hudson, S. E. Lightweight Material Detection for Placement-Aware Mobile Computing. Proc. UIST'08.

Digital Library

[10]

Harrison, C., Schwarz, J., and Hudson, S. E. TapSense: Enhancing finger interaction on touch surfaces. Proc. UIST'11.

Digital Library

[11]

Harrison, C., Xiao, R., and Hudson, S. E. Acoustic Barcodes: Passive, Durable and Inexpensive Notched Identification Tags. Proc. UIST'12.

Digital Library

[12]

Hart, S. G. Nasa-Task Load Index (NASA-TLX); 20 Years Later. Proceedings of the Human Factors and Ergonomics Society Annual Meeting 50, 9 (2006).

[13]

Hinckley, K., Ramos, G., Guimbretiere, F., Baudisch, P., and Smith, M. Stitching: Pen Gestures that Span Multiple Displays. Proc. AVI'04.

Digital Library

[14]

Hinckley, K. and Ramos, G. Cooperative Stitching: Spontaneous Wireless Connections for Small Co-Located Groups (TECHNOTE).

[15]

Holmquist, L.E., Mattern, F., Schiele, B., Alahuhta, P., Beigl, M., and Gellersen, H.-w. Smart-Its Friends: A Technique for Users to Easily Establish Connections between Smart Artefacts. Proc. UbiComp'01.

Digital Library

[16]

Hutama, W., Song, P., Fu, C.-W., and Goh, W. B. Distinguishing multiple smart-phone interactions on a multi-touch wall display using tilt correlation. Proc. CHI'11.

Digital Library

[17]

Kane, S. K., Avrahami, D., Wobbrock, J. O., et al. Bonfire: A Nomadic System for Hybrid Laptop-Tabletop Interaction. Proc. UIST'09.

Digital Library

[18]

Kortuem, G., Kray, C., and Gellersen, H. Sensing and visualizing spatial relations of mobile devices. Proc. UIST'05.

Digital Library

[19]

Kunze, K. and Lukowicz, P. Symbolic Object Localization Through Active Sampling of Acceleration and Sound Signatures. Proc. UbiComp'07.

Digital Library

[20]

Lopes, P., Jota, R., and Jorge, J. A. Augmenting touch interaction through acoustic sensing. Proc. ITS'11.

Digital Library

[21]

Lucero, A., Holopainen, J., and Jokela, T. Pass-Them-Around: Collaborative Use of Mobile Phones for Photo Sharing. Proc. CHI'11.

Digital Library

[22]

Lucero, A., Jokela, T., Palin, A., Aaltonen, V., and Nikara, J. EasyGroups: Binding Mobile Devices for Collaborative Interactions. Proc. CHI'12 (WIP).

Digital Library

[23]

Matsushita, N. and Rekimoto, J. HoloWall: Designing a Finger, Hand, Body, and Object Sensitive Wall. Proc. UIST'97.

Digital Library

[24]

Mayrhofer, R. and Gellersen, H. Shake Well Before Use: Authentication Based on Accelerometer Data. Proc. Pervasive'07.

Digital Library

[25]

Murray-Smith, R., Williamson, J., Hughes, S., and Quaade, T. Stane: Synthesized Surfaces for Tactile Input. Proc. CHI'08.

Digital Library

[26]

Paradiso, J. A., Leo, C. K., Checka, N., and Hsiao, K. Passive acoustic sensing for tracking knocks atop large interactive displays. Proc. IEEE Sensors 2002.

[27]

Ramos, G., Hinckley, K., Wilson, A., and Sarin, R. Synchronous Gestures in Multi-Display Environments. Human-Computer Interaction, Special Issue: Ubiquitous Multi-Display Environments 24, 1-2 (2009).

[28]

Rekimoto, J. and Saitoh, M. Augmented Surfaces: A Spatially Continuous Work Space for Hybrid Computing Environments. Proc. CHI'99.

Digital Library

[29]

Schmidt, D., Chehimi, F., Rukzio, E., and Gellersen, H. PhoneTouch: A Technique for Direct Phone Interaction on Surfaces. Proc. UIST'10.

Digital Library

[30]

Seniuk, A. and Blostein, D. Pen Acoustic Emissions for Text and Gesture Recognition. 10th Int'l Conference on Document Analysis and Recognition (2009).

Digital Library

[31]

Thiel, B., Kloch, K., and Lukowicz, P. Sound-based proximity detection with mobile phones. Proc. PhoneSense'12.

Digital Library

[32]

Wilson, A. D. and Sarin, R. BlueTable: Connecting Wireless Mobile Devices on Interactive Surfaces Using Vision-Based Handshaking. Proc. GI'07.

Digital Library

Cited By

Liu YShan QYao ZLu Q(2024)KeyFlow: Acoustic Motion Sensing for Cursor Control on Any KeyboardAdjunct Proceedings of the 37th Annual ACM Symposium on User Interface Software and Technology10.1145/3672539.3686348(1-3)Online publication date: 13-Oct-2024
https://dl.acm.org/doi/10.1145/3672539.3686348
Yang YChen THuang YGuo XShangguan L(2024)MAF: Exploring Mobile Acoustic Field for Hand-to-Face Gesture InteractionsProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642437(1-20)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3642437
Iravantchi YZhao YKin KSample A(2023)SAWSense: Using Surface Acoustic Waves for Surface-bound Event RecognitionProceedings of the 2023 CHI Conference on Human Factors in Computing Systems10.1145/3544548.3580991(1-18)Online publication date: 19-Apr-2023
https://dl.acm.org/doi/10.1145/3544548.3580991
Show More Cited By

Index Terms

SurfaceLink: using inertial and acoustic sensing to enable multi-device interaction on a surface
1. Applied computing
  1. Computers in other domains
    1. Personal computers and PC applications
2. Hardware
  1. Communication hardware, interfaces and storage
    1. Signal processing systems

Recommendations

TapSkin: Recognizing On-Skin Input for Smartwatches
ISS '16: Proceedings of the 2016 ACM International Conference on Interactive Surfaces and Spaces

The touchscreen has been the dominant input surface for smartphones and smartwatches. However, its small size compared to a phone limits the richness of the input gestures that can be supported. We present TapSkin, an interaction technique that ...
Acoustic tracking of hand activities on surfaces
iWOAR '15: Proceedings of the 2nd international Workshop on Sensor-based Activity Recognition and Interaction

Many common forms of activities are tactile in their nature. We touch, grasp, and interact with a plethora of objects every day. Some of those objects are registering our activities, such as the millions of touch screens we are using every day. Adding ...
DisplayPointers: seamless cross-device interactions
ACE '15: Proceedings of the 12th International Conference on Advances in Computer Entertainment Technology

We present a system for cross-device interactions and interaction scenarios based on touch events between devices. DisplayPointers were designed to explore the affordances of manipulating physical display-objects in multi-device environments. The ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

CHI '14: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems

April 2014

4206 pages

ISBN:9781450324731

DOI:10.1145/2556288

General Chairs:
Matt Jones
Swansea University, Wales, UK
,
Philippe Palanque
Université Paul Sabatier, France
,
Program Chairs:
Albrecht Schmidt
University of Stuttgart, Germany
,
Tovi Grossman
Autodesk Research, Canada

Copyright © 2014 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGCHI: ACM Special Interest Group on Computer-Human Interaction

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 26 April 2014

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

CHI '14

Sponsor:

SIGCHI

CHI '14: CHI Conference on Human Factors in Computing Systems

April 26 - May 1, 2014

Ontario, Toronto, Canada

Acceptance Rates

CHI '14 Paper Acceptance Rate 465 of 2,043 submissions, 23%;

Overall Acceptance Rate 6,199 of 26,314 submissions, 24%

Upcoming Conference

CHI 2025

Sponsor:
sigchi

ACM CHI Conference on Human Factors in Computing Systems

April 26 - May 1, 2025

Yokohama , Japan

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

37
Total Citations
View Citations
1,156
Total Downloads

Downloads (Last 12 months)30
Downloads (Last 6 weeks)5

Reflects downloads up to 01 Mar 2025

Other Metrics

View Author Metrics

Citations

Cited By

Liu YShan QYao ZLu Q(2024)KeyFlow: Acoustic Motion Sensing for Cursor Control on Any KeyboardAdjunct Proceedings of the 37th Annual ACM Symposium on User Interface Software and Technology10.1145/3672539.3686348(1-3)Online publication date: 13-Oct-2024
https://dl.acm.org/doi/10.1145/3672539.3686348
Yang YChen THuang YGuo XShangguan L(2024)MAF: Exploring Mobile Acoustic Field for Hand-to-Face Gesture InteractionsProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642437(1-20)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3642437
Iravantchi YZhao YKin KSample A(2023)SAWSense: Using Surface Acoustic Waves for Surface-bound Event RecognitionProceedings of the 2023 CHI Conference on Human Factors in Computing Systems10.1145/3544548.3580991(1-18)Online publication date: 19-Apr-2023
https://dl.acm.org/doi/10.1145/3544548.3580991
Kim MLee DKim H(2023)Surface-Assisted In-Air Gesture Recognition2023 14th International Conference on Information and Communication Technology Convergence (ICTC)10.1109/ICTC58733.2023.10392389(666-668)Online publication date: 11-Oct-2023
https://doi.org/10.1109/ICTC58733.2023.10392389
Yoshida MMatsui TIshiyama TFujimoto MSuwa HYasumoto K(2023)Smatable: A Vibration-Based Sensing Method for Making Ordinary Tables Touch-InterfacesIEEE Access10.1109/ACCESS.2023.334350011(142611-142627)Online publication date: 2023
https://doi.org/10.1109/ACCESS.2023.3343500
Chatiwala SHajarnis RKorolev APark DShenkerman DChen YYang Y(2022)A Secure and Reliable Mobile Authentication Alternative Utilizing Hand Structure2022 IEEE MIT Undergraduate Research Technology Conference (URTC)10.1109/URTC56832.2022.10002184(1-5)Online publication date: 30-Sep-2022
https://doi.org/10.1109/URTC56832.2022.10002184
Ali KLiu A(2022)Fine-Grained Vibration Based Sensing Using a SmartphoneIEEE Transactions on Mobile Computing10.1109/TMC.2021.306767921:11(3971-3985)Online publication date: 1-Nov-2022
https://doi.org/10.1109/TMC.2021.3067679
Rathore AXu CZhu WDaiyan AWang KLin FRen KXu W(2022)Scanning the Voice of Your Fingerprint With Everyday SurfacesIEEE Transactions on Mobile Computing10.1109/TMC.2021.304921721:8(3024-3040)Online publication date: 1-Aug-2022
https://doi.org/10.1109/TMC.2021.3049217
Grønbæk JKnudsen MO'Hara KKrogh PVermeulen JPetersen MBernhaupt RMueller FVerweij DAndres JMcGrenere JCockburn AAvellino IGoguey ABjørn PZhao SSamson BKocielnik R(2020)Proxemics Beyond ProximityProceedings of the 2020 CHI Conference on Human Factors in Computing Systems10.1145/3313831.3376379(1-14)Online publication date: 21-Apr-2020
https://dl.acm.org/doi/10.1145/3313831.3376379
Kim SPark JChae MJung SChang H(2019)A device binding method based on content illumination pattern in public display environmentsPLOS ONE10.1371/journal.pone.021449314:5(e0214493)Online publication date: 10-May-2019
https://doi.org/10.1371/journal.pone.0214493
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten