research-article

SymmetriSense: Enabling Near-Surface Interactivity on Glossy Surfaces using a Single Commodity Smartphone

Authors:

Robert F. DickersonAuthors Info & Claims

CHI '16: Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems

Pages 5126 - 5137

https://doi.org/10.1145/2858036.2858286

Published: 07 May 2016 Publication History

Abstract

Driven to create intuitive computing interfaces throughout our everyday space, various state-of-the-art technologies have been proposed for near-surface localization of a user's finger input such as hover or touch. However, these works require specialized hardware not commonly available, limiting the adoption of such technologies. We present SymmetriSense, a technology enabling near-surface 3-dimensional fingertip localization above arbitrary glossy surfaces using a single commodity camera device such as a smartphone. SymmetriSense addresses the localization challenges in using a single regular camera by a novel technique utilizing the principle of reflection symmetry and the fingertip's natural reflection casted upon surfaces like mirrors, granite countertops, or televisions. SymmetriSense achieves typical accuracies at sub-centimeter levels in our localization tests with dozens of volunteers and remains accurate under various environmental conditions. We hope SymmetriSense provides a technical foundation on which various everyday near-surface interactivity can be designed.

Supplementary Material

suppl.mov (pn1294-file3.mp4)

Supplemental video

Download
67.45 MB

MP4 File (p5126-yoo.mp4)

Download
200.59 MB

References

[1]

Michelle Annett, Tovi Grossman, Daniel Wigdor, and George Fitzmaurice. Medusa: a proximity-aware multi-touch tabletop. In Proc. UIST 2011. ACM, 337-346.

Digital Library

[2]

Arduino. Retrieved September 24, 2015 from https://www.arduino.cc.

[3]

Aras Balali Moghaddam, Jeremy Svendsen, Melanie Tory, and Alexandra Branzan Albu. Integrating touch and near touch interactions for information visualizations. In Proc. CHI 2011 Extended Abstracts. ACM, 2347-2352.

Digital Library

[4]

Dominikus Baur, Sebastian Boring, and Steven Feiner. Virtual projection: exploring optical projection as a metaphor for multi-device interaction. In Proc. CHI 2012. ACM, 1693-1702.

Digital Library

[5]

SMART Board. Retrieved December 28, 2015 from http://education.smarttech.com/.

[6]

Camera calibration with OpenCV. Retrieved December 26, 2015 from http: //docs.opencv.org/2.4/doc/tutorials/calib3d/ camera_calibration/camera_calibration.html.

[7]

Ke-Yu Chen, Gabe A Cohn, Sidhant Gupta, and Shwetak N Patel. uTouch: sensing touch gestures on unmodified LCDs. In Proc. CHI 2013. ACM, 2581-2584.

Digital Library

[8]

Victor Cheung, Jens Heydekorn, Stacey Scott, and Raimund Dachselt. Revisiting hovering: Interaction guides for interactive surfaces. In Proc. ITS 2012. ACM, 355-358.

Digital Library

[9]

Rita Cucchiara, Costantino Grana, Massimo Piccardi, and Andrea Prati. Detecting moving objects, ghosts, and shadows in video streams. Pattern Analysis and Machine Intelligence, IEEE Transactions on 25, 10 (2003), 1337-1342.

Digital Library

[10]

Eduardo Cuervo, Aruna Balasubramanian, Dae-ki Cho, Alec Wolman, Stefan Saroiu, Ranveer Chandra, and Paramvir Bahl. MAUI: making smartphones last longer with code offload. In Proc. MobiSys 2010 (MobiSys '10). ACM, NY, NY, USA, 49-62.

Digital Library

[11]

Frederic Devernay and Olivier Faugeras. Straight lines have to be straight. Machine vision and applications 13, 1 (2001), 14-24.

Digital Library

[12]

Florian Echtler, Andreas Dippon, Marcus Tonnis, and Gudrun Klinker. Inverted FTIR: easy multitouch sensing for fiatscreens. In Proc. ITS 2009. ACM, 29-32.

Digital Library

[13]

Mayank Goel, Brendan Lee, Md Tanvir Islam Aumi, Shwetak Patel, Gaetano Borriello, Stacie Hibino, and Bo Begole. SurfaceLink: using inertial and acoustic sensing to enable multi-device interaction on a surface. In Proc. CHI 2014. ACM, 1387-1396.

Digital Library

[14]

Chris Harrison, Hrvoje Benko, and Andrew D Wilson. OmniTouch: wearable multitouch interaction everywhere. In Proc. UIST 2011. ACM, 441-450.

Digital Library

[15]

Chris Harrison and Scott E Hudson. Scratch input: creating large, inexpensive, unpowered and mobile finger input surfaces. In Proc. UIST 2008. ACM, 205-208.

Digital Library

[16]

Otmar Hilliges, Shahram Izadi, Andrew D Wilson, Steve Hodges, Armando Garcia-Mendoza, and Andreas Butz. Interactions in the air: adding further depth to interactive tabletops. In Proc. UIST 2009. ACM, 139-148.

Digital Library

[17]

Pan Hu, Guobin Shen, Liqun Li, and Donghuan Lu. ViRi: view it right. In Proc. MobiSys 2013. ACM, 277-290.

Digital Library

[18]

David Kim, Shahram Izadi, Jakub Dostal, Christoph Rhemann, Cem Keskin, Christopher Zach, Jamie Shotton, Timothy Large, Steven Bathiche, Matthias Nießner, and others. RetroDepth: 3D silhouette sensing for high-precision input on and above physical surfaces. In Proc. CHI 2014. ACM, 1377-1386.

Digital Library

[19]

Sven Kratz, Patrick Chiu, and Maribeth Back. Pointpose: finger pose estimation for touch input on mobile devices using a depth sensor. In Proc. ITS 2013. ACM, 223-230.

Digital Library

[20]

Shenwei Liu and François Guimbreti'ere. FlexAura: a flexible near-surface range sensor. In Proc. UIST 2012. ACM, 327-330.

Digital Library

[21]

Nicolai Marquardt, Ricardo Jota, Saul Greenberg, and Joaquim A Jorge. The continuous interaction space: interaction techniques unifying touch and gesture on and above a digital surface. In Proc. INTERACT 2011. Springer, 461-476.

Digital Library

[22]

Davide A Migliore, Matteo Matteucci, and Matteo Naccari. A revaluation of frame difference in fast and robust motion detection. In Proc. 4th ACM international workshop on Video surveillance and sensor networks, 2006. ACM, 215-218.

Digital Library

[23]

Evan Mills and Nils Borg. Trends in recommended illuminance levels: an international comparison. Journal of the Illuminating Engineering Society 28, 1 (1999), 155-163.

[24]

Leap Motion. Retrieved September 24, 2015 from https://www.leapmotion.com.

[25]

Anna Ostberg and Nada Matic. Hover cursor: Improving touchscreen acquisition of small targets with hover-enabled pre-selection. In Proc. CHI 2015 Extended Abstracts. ACM, 1723-1728.

Digital Library

[26]

Son Lam Phung, Abdesselam Bouzerdoum, and Douglas Chai. A novel skin color model in ycbcr color space and its application to human face detection. In Proc. International Conference on Image Processing, 2002, Vol. 1. IEEE, I-289.

[27]

Rasberry Pi. Retrieved September 24, 2015 from https://www.raspberrypi.org.

[28]

Jun Rekimoto. SmartSkin: an infrastructure for freehand manipulation on interactive surfaces. In Proc. CHI 2002. ACM, 113-120.

Digital Library

[29]

Munehiko Sato, Ivan Poupyrev, and Chris Harrison. Touche: enhancing touch interaction on humans, screens, liquids, and everyday objects. In Proc. CHI 2012. ACM, 483-492.

Digital Library

[30]

Toby Sharp, Cem Keskin, Duncan Robertson, Jonathan Taylor, Jamie Shotton, David Kim Christoph Rhemann Ido Leichter, Alon Vinnikov Yichen Wei, Daniel Freedman Pushmeet Kohli Eyal Krupka, Andrew Fitzgibbon, and Shahram Izadi. Accurate, robust, and flexible real-time hand tracking. In Proc. CHI 2015. ACM, 3633-3642.

Digital Library

[31]

Jeongho Shin, Sangjin Kim, Sangkyu Kang, Seong-Won Lee, Joonki Paik, Besma Abidi, and Mongi Abidi. Optical flow-based real-time object tracking using non-prior training active feature model. Real-Time Imaging 11, 3 (2005), 204-218.

Digital Library

[32]

Jie Song, Gabor Soros, Fabrizio Pece, Sean Ryan Fanello, Shahram Izadi, Cem Keskin, and Otmar Hilliges. In-air gestures around unmodified mobile devices. In Proc. UIST 2014. ACM, 319-329.

Digital Library

[33]

Martin Spindler, Marcel Martsch, and Raimund Dachselt. Going beyond the surface: studying multi-layer interaction above the tabletop. In Proc. CHI 2012. ACM, 1277-1286.

Digital Library

[34]

Yoshiki Takeoka, Takashi Miyaki, and Jun Rekimoto. Z-touch: an infrastructure for 3d gesture interaction in the proximity of tabletop surfaces. In Proc. ITS 2010. ACM, 91-94.

Digital Library

[35]

Robby T Tan and Katsushi Ikeuchi. Separating reflection components of textured surfaces using a single image. Pattern Analysis and Machine Intelligence, IEEE Transactions on 27, 2 (2005), 178-193.

Digital Library

[36]

Sony Floating Touch. Retrieved September 24, 2015 from http://developer.sonymobile.com/ knowledge-base/technologies/floating-touch.

[37]

Samsung Air View. Retrieved September 24, 2015 from http://www.samsung.com/us/support/answer/ ANS00044011/997407174/.

[38]

Chat Wacharamanotham, Kashyap Todi, Marty Pye, and Jan Borchers. Understanding finger input above desktop devices. In Proc. CHI 2014. ACM, 1083-1092.

Digital Library

[39]

Andrew D Wilson. PlayAnywhere: a compact interactive tabletop projection-vision system. In Proc. UIST 2005. ACM, 83-92.

Digital Library

[40]

Robert Xiao, Chris Harrison, and Scott E Hudson. WorldKit: rapid and easy creation of ad-hoc interactive applications on everyday surfaces. In Proc. CHI 2013. ACM, 879-888.

Digital Library

[41]

Robert Xiao, Greg Lew, James Marsanico, Divya Hariharan, Scott Hudson, and Chris Harrison. Toffee: enabling ad hoc, around-device interaction with acoustic time-of-arrival correlation. In Proc. MobileHCI 2014. ACM, 67-76.

Digital Library

[42]

Alper Yilmaz, Omar Javed, and Mubarak Shah. Object tracking: A survey. ACM Computing Surveys (CSUR) 38, 4 (2006), 13.

Digital Library

Cited By

Wu YVillalobos LYang ZCroisdale GKarataş ÇLiu J(2023)SmarCyPadProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/36109277:3(1-26)Online publication date: 27-Sep-2023
https://dl.acm.org/doi/10.1145/3610927
Liang KZhou AZhang ZZhou HMa HWu C(2023)mmStressProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/36109267:3(1-36)Online publication date: 27-Sep-2023
https://dl.acm.org/doi/10.1145/3610926
Stanke DDuente TDemir KRohs M(2023)Can You Ear Me?Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/36109257:3(1-23)Online publication date: 27-Sep-2023
https://dl.acm.org/doi/10.1145/3610925
Show More Cited By

Index Terms

SymmetriSense: Enabling Near-Surface Interactivity on Glossy Surfaces using a Single Commodity Smartphone
1. Human-centered computing
  1. Human computer interaction (HCI)

Recommendations

Tactile Presentation to the Back of a Smartphone with Simultaneous Screen Operation
CHI EA '16: Proceedings of the 2016 CHI Conference Extended Abstracts on Human Factors in Computing Systems

In most common methods of tactile presentation on touch screen, the tactile display was directly attached or contacted onto the screens. Therefore, the tactile display must be transparent so that it does not obstruct the view of the screen. On the other ...
Investigating Screen Shifting Techniques to Improve One-Handed Smartphone Usage
NordiCHI '16: Proceedings of the 9th Nordic Conference on Human-Computer Interaction

With increasingly large smartphones, it becomes more difficult to use these devices one-handed. Due to a large touchscreen, users can not reach across the whole screen using their thumb. In this paper, we investigate approaches to move the screen ...
Combination of tactile devices to interact with a spreadsheet on tablet
IHM '18: Proceedings of the 30th Conference on l'Interaction Homme-Machine

Multi-touch interaction with spreadsheets on tablets is tedious and raises multiple interaction issues. In this paper, we present the design and evaluation of novel interaction techniques relying on the usage of an always-available device, the ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

CHI '16: Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems

May 2016

6108 pages

ISBN:9781450333627

DOI:10.1145/2858036

General Chairs:
Jofish Kaye
Yahoo
,
Allison Druin
University of Maryland / National Park Service
,
Program Chairs:
Cliff Lampe
University of Michigan
,
Dan Morris
Microsoft
,
Juan Pablo Hourcade
University of Iowa

Copyright © 2016 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

SIGCHI: ACM Special Interest Group on Computer-Human Interaction

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 07 May 2016

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

CHI'16

Sponsor:

SIGCHI

CHI'16: CHI Conference on Human Factors in Computing Systems

May 7 - 12, 2016

California, San Jose, USA

Acceptance Rates

CHI '16 Paper Acceptance Rate 565 of 2,435 submissions, 23%;

Overall Acceptance Rate 6,199 of 26,314 submissions, 24%

Upcoming Conference

CHI 2025

Sponsor:
sigchi

ACM CHI Conference on Human Factors in Computing Systems

April 26 - May 1, 2025

Yokohama , Japan

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

22
Total Citations
View Citations
504
Total Downloads

Downloads (Last 12 months)18
Downloads (Last 6 weeks)2

Reflects downloads up to 03 Mar 2025

Other Metrics

View Author Metrics

Citations

Cited By

Wu YVillalobos LYang ZCroisdale GKarataş ÇLiu J(2023)SmarCyPadProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/36109277:3(1-26)Online publication date: 27-Sep-2023
https://dl.acm.org/doi/10.1145/3610927
Liang KZhou AZhang ZZhou HMa HWu C(2023)mmStressProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/36109267:3(1-36)Online publication date: 27-Sep-2023
https://dl.acm.org/doi/10.1145/3610926
Stanke DDuente TDemir KRohs M(2023)Can You Ear Me?Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/36109257:3(1-23)Online publication date: 27-Sep-2023
https://dl.acm.org/doi/10.1145/3610925
Zhang JDai RRjob AWang RHamauon RCandell JBailey TFraser VGuillamet MLu C(2023)Contact Tracing for Healthcare Workers in an Intensive Care UnitProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/36109247:3(1-23)Online publication date: 27-Sep-2023
https://dl.acm.org/doi/10.1145/3610924
Medeiros DDubus RWilliamson JWilson GPöhlmann KMcGill M(2023)Surveying the Social Comfort of Body, Device, and Environment-Based Augmented Reality Interactions in Confined Passenger Spaces Using Mixed Reality Composite VideosProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/36109237:3(1-25)Online publication date: 27-Sep-2023
https://dl.acm.org/doi/10.1145/3610923
Jiang WShi LNiu QLiu N(2023)Fast Radio Map Construction with Domain Disentangled Learning for Wireless LocalizationProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/36109227:3(1-27)Online publication date: 27-Sep-2023
https://dl.acm.org/doi/10.1145/3610922
Kim JNam YLee JSuh YHwang I(2023)ProxiFitProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/36109207:3(1-32)Online publication date: 27-Sep-2023
https://dl.acm.org/doi/10.1145/3610920
Bhalla SLiaqat SWu RGershon Ade Lara EMariakakis A(2023)PulmoListenerProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/36108897:3(1-24)Online publication date: 27-Sep-2023
https://dl.acm.org/doi/10.1145/3610889
Cho SJang JKim YHwang I(2023)Demonstrating AHA: Boosting Unmodified AI's Robustness by Proactively Inducing Favorable Human Sensing ConditionsAdjunct Proceedings of the 2023 ACM International Joint Conference on Pervasive and Ubiquitous Computing & the 2023 ACM International Symposium on Wearable Computing10.1145/3594739.3610718(187-190)Online publication date: 8-Oct-2023
https://dl.acm.org/doi/10.1145/3594739.3610718
Kim MHwang I(2023)TeleVIP: On-site Person Removal and Context Distillation Platform for Dedicated Telepresence ExperienceAdjunct Proceedings of the 2023 ACM International Joint Conference on Pervasive and Ubiquitous Computing & the 2023 ACM International Symposium on Wearable Computing10.1145/3594739.3610704(123-127)Online publication date: 8-Oct-2023
https://dl.acm.org/doi/10.1145/3594739.3610704
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten