research-article

RetroDepth: 3D silhouette sensing for high-precision input on and above physical surfaces

Authors:

Christoph Rhemann,

Christopher Zach,

Steven Bathiche,

Matthias Nießner,

D. Alex Butler,

Vivek PradeepAuthors Info & Claims

CHI '14: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems

Pages 1377 - 1386

https://doi.org/10.1145/2556288.2557336

Published: 26 April 2014 Publication History

Abstract

We present RetroDepth, a new vision-based system for accurately sensing the 3D silhouettes of hands, styluses, and other objects, as they interact on and above physical surfaces. Our setup is simple, cheap, and easily reproducible, comprising of two infrared cameras, diffuse infrared LEDs, and any off-the-shelf retro-reflective material. The retro-reflector aids image segmentation, creating a strong contrast between the surface and any object in proximity. A new highly efficient stereo matching algorithm precisely estimates the 3D contours of interacting objects and the retro-reflective surfaces. A novel pipeline enables 3D finger, hand and object tracking, as well as gesture recognition, purely using these 3D contours. We demonstrate high-precision sensing, allowing robust disambiguation between a finger or stylus touching, pressing or interacting above the surface. This allows many interactive scenarios that seamlessly mix together freehand 3D interactions with touch, pressure and stylus input. As shown, these rich modalities of input are enabled on and above any retro-reflective surface, including custom "physical widgets" fabricated by users. We compare our system with Kinect and Leap Motion, and conclude with limitations and future work.

Supplementary Material

suppl.mov (pn2022-file3.mp4)

Supplemental video

Download
81.18 MB

MP4 File (p1377-sidebyside.mp4)

Download
232.88 MB

References

[1]

3Gear Systems Inc. http://threegear.com/, 2013.

[2]

Agarwal, A., Izadi, S., Chandraker, M., and Blake, A. High precision multi-touch sensing on surfaces using cameras. In Tabletop'07, 197--200.

[3]

Akaoka, E., Ginn, T., and Vertegaal, R. Displayobjects: prototyping functional physical interfaces on 3d styrofoam, paper or cardboard models. In TEI'10, ACM (2010), 49--56.

Digital Library

[4]

Annett, M., Grossman, T., Wigdor, D., and Fitzmaurice, G. Medusa: a proximity-aware multi-touch tabletop. In UIST'11, ACM (2011), 337--346.

Digital Library

[5]

Benko, H., Jota, R., and Wilson, A. Miragetable: freehand interaction on a projected augmented reality tabletop. In CHI'12, ACM (2012), 199--208.

Digital Library

[6]

Bleyer, M., Rhemann, C., and Rother, C. Patchmatch stereo - stereo matching with slanted support windows. In British Machine Vision Conference (2011).

[7]

Breiman, L. Random Forests. Machine Learning 45, 1 (Oct. 2001), 5--32.

Digital Library

[8]

Brown, M. Z., Burschka, D., and Hager, G. D. Advances in computational stereo. PAMI 25, 8 (2003), 993--1008.

Digital Library

[9]

Davis, J. W., and Bobick, A. F. Sideshow: A silhouettebased interactive dual-screen environment. Tech. rep., MIT, 1998.

[10]

Dippon, A., and Klinker, G. Kinecttouch: accuracy test for a very low-cost 2.5 d multitouch tracking system. In ITS'11, ACM (2011), 49--52.

Digital Library

[11]

Haptix. http://www.haptixtouch.com/, 2013.

[12]

Haubner, N., Schwanecke, U., Dörner, R., Lehmann, S., and Luderschmidt, J. Detecting interaction above digital tabletops using a single depth camera. Machine Vision and Applications (2013), 1--13.

Digital Library

[13]

Hilliges, O., Izadi, S., Wilson, A. D., Hodges, S., Garcia-Mendoza, A., and Butz, A. Interactions in the air: adding further depth to interactive tabletops. In UIST'09, ACM (2009), 139--148.

Digital Library

[14]

Hilliges, O., Kim, D., Izadi, S., Weiss, M., and Wilson, A. Holodesk: direct 3d interactions with a situated seethrough display. In CHI'12, ACM (2012), 2421--2430.

Digital Library

[15]

Hirsch, M., Lanman, D., Holtzman, H., and Raskar, R. Bidi screen: a thin, depth-sensing lcd for 3d interaction using light fields. In TOG, vol. 28, ACM (2009), 159.

Digital Library

[16]

Izadi, S., Agarwal, A., Criminisi, A., Winn, J., Blake, A., and Fitzgibbon, A. C-slate: a multi-touch and object recognition system for remote collaboration using horizontal surfaces. In Tabletop'07, IEEE (2007), 3--10.

[17]

Izadi, S., Hodges, S., Butler, A., West, D., Rrustemi, A., Molloy, M., and Buxton,W. Thinsight: a thin form-factor interactive surface technology. CACM 52, 12, 90--98.

Digital Library

[18]

Izadi, S., Hodges, S., Taylor, S., Rosenfeld, D., Villar, N., Butler, A., and Westhues, J. Going beyond the display: a surface technology with an electronically switchable diffuser. In UIST'08, ACM (2008), 269--278.

Digital Library

[19]

Keskin, C., Kirac, F., Kara, Y. E., and Akarun, L. Hand Pose Estimation and Hand Shape Classification Using Multi-layered Randomized Decision Forests. In ECCV'12 (2012).

Digital Library

[20]

Koike, H., Sato, Y., and Kobayashi, Y. Integrating paper and digital information on enhanceddesk: a method for realtime finger tracking on an augmented desk system. TOCHI 8, 4 (2001), 307--322.

Digital Library

[21]

Krueger, M. W. Artificial reality II, vol. 10. Addison-Wesley Reading (Ma), 1991.

[22]

Lanman, D., and Taubin, G. Build your own 3d scanner: 3d photography for beginners. In ACM SIGGRAPH 2009 Courses, ACM (2009), 8.

Digital Library

[23]

Lee, J., Park, K. S., and Hahn, M. The 3d sensor table for bare hand tracking and posture recognition. In Advances in Multimedia Modeling. Springer, 2006, 138--146.

Digital Library

[24]

Liu, Y., Weibel, N., and Hollan, J. Interactive space: A framework for prototyping multi-touch interaction on and above the desktop. In CHI'13, vol. 13 (2013).

[25]

Malik, S., and Laszlo, J. Visual touchpad: a two-handed gestural input device. In ICMI'04, 289--296.

Digital Library

[26]

Marquardt, N., Jota, R., Greenberg, S., and Jorge, J. A. The continuous interaction space: interaction techniques unifying touch and gesture on and above a digital surface. In INTERACT'11. Springer, 2011, 461--476.

Digital Library

[27]

Marroquim, R., Kraus, M., and Cavalcanti, P. R. Efficient point-based rendering using image reconstruction. In SPBG (2007), 101--108.

[28]

Matsushita, N., and Rekimoto, J. Holowall: designing a finger, hand, body, and object sensitive wall. In UIST'97, ACM (1997), 209--210.

Digital Library

[29]

Moeller, J., Kerne, A., and Moeller, J. Zerotouch: An optical multi-touch and free-air interaction architecture. In CH'12, ACM (2012), 2165--2174.

Digital Library

[30]

Paradiso, J. A. Several sensor approaches that retrofit large surfaces for interactivity. In UbiComp 2002 (2002).

[31]

Remondino, F., and Stoppa, D. Tof range-imaging cameras. Springer, 2013.

Digital Library

[32]

Scharstein, D., and Szeliski, R. A taxonomy and evaluation of dense two-frame stereo correspondence algorithms. In IJCV (2002).

Digital Library

[33]

Shotton, J., Fitzgibbon, A., Cook, M., Sharp, T., Finocchio, M., Moore, R., Kipman, A., and Blake, A. Realtime Human Pose Recognition in Parts from Single Depth Images. In CVPR (2011).

Digital Library

[34]

Strecha, C., and Van Gool, L. PDE-based multi-view depth estimation. In Proc. 3DIMPVT (2002), 416--425.

[35]

Subramanian, S., Aliakseyeu, D., and Lucero, A. Multilayer interaction for digital tables. In UIST'06, ACM (2006), 269--272.

Digital Library

[36]

Takeoka, Y., Miyaki, T., and Rekimoto, J. Z-touch: an infrastructure for 3d gesture interaction in the proximity of tabletop surfaces. In ITS'10, ACM (2010), 91--94.

Digital Library

[37]

Ullmer, B., and Ishii, H. The metadesk: models and prototypes for tangible user interfaces. In UIST'97, ACM (1997), 223--232.

Digital Library

[38]

Wang, R., Paris, S., and Popović, J. 6d hands: markerless hand-tracking for computer aided design. In UIST'11, ACM (2011), 549--558.

Digital Library

[39]

Wellner, P. Interacting with paper on the digitaldesk. CACM 36, 7 (1993), 87--96.

Digital Library

[40]

Wesche, G., and Seidel, H.-P. Freedrawer: a free-form sketching system on the responsive workbench. In VRST, ACM (2001), 167--174.

Digital Library

[41]

Wilson, A. D. Playanywhere: a compact interactive tabletop projection-vision system. In UIST'05, ACM (2005), 83--92.

Digital Library

[42]

Wilson, A. D. Robust computer vision-based detection of pinching for one and two-handed gesture input. In UIST'06, ACM (2006), 255--258.

Digital Library

[43]

Wilson, A. D. Depth-sensing video cameras for 3d tangible tabletop interaction. In Tabletop'07, IEEE (2007), 201--204.

[44]

Wilson, A. D. Using a depth camera as a touch sensor. In ITS'10, ACM (2010), 69--72.

Digital Library

Cited By

Maierhöfer VSchmid AWimmer RCiolfi LHogan TDöring TJenkins Tvan Dijk JHuron SLi ZCoyle DSigner B(2024)TipTrack: Precise, Low-Latency, Robust Optical Pen Tracking on Arbitrary Surfaces Using an IR-Emitting Pen TipProceedings of the Eighteenth International Conference on Tangible, Embedded, and Embodied Interaction10.1145/3623509.3633366(1-13)Online publication date: 11-Feb-2024
https://dl.acm.org/doi/10.1145/3623509.3633366
Balaji AKimber CLi DWu SDu RKim D(2023)RetroSphereProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/35694796:4(1-36)Online publication date: 11-Jan-2023
https://dl.acm.org/doi/10.1145/3569479
Zhang YLi ZXu SLi CYang JTong XGuo B(2023)RemoteTouch: Enhancing Immersive 3D Video Communication with Hand Touch2023 IEEE Conference Virtual Reality and 3D User Interfaces (VR)10.1109/VR55154.2023.00016(1-10)Online publication date: Mar-2023
https://doi.org/10.1109/VR55154.2023.00016
Show More Cited By

Index Terms

RetroDepth: 3D silhouette sensing for high-precision input on and above physical surfaces
1. Human-centered computing
  1. Human computer interaction (HCI)

Recommendations

Occlusion-aware Hand Posture Based Interaction on Tabletop Projector
UIST '19 Adjunct: Adjunct Proceedings of the 32nd Annual ACM Symposium on User Interface Software and Technology

Conventional camera-based hand interaction technique suffered from self-occlusion among fingers, which lowers the detection accuracy of fingertip positions, leading to uncomfortable UI controls. Based on observations, self-occlusion depends on hand ...
Easy vs. Tricky: The Shape Effect in Tracing, Selecting, and Steering With Mouse, Stylus, and Touch
AcademicMindTrek '13: Proceedings of International Conference on Making Sense of Converging Media

This short paper is a work-in-progress report on an experimental comparison and evaluation of users' performance in four line-tracing tasks based on two shapes and performed with three input methods (mouse, stylus, and touch-input). The shapes' ...
Reflective haptics: enhancing stylus-based interactions on touch screens
EuroHaptics'10: Proceedings of the 2010 international conference on Haptics: generating and perceiving tangible sensations, Part I

In this paper, we introduce the prototype of a low cost haptically augmented stylus for pen computing on touch screens. The stylus supports human-computer interaction through a dynamic haptic feedback. This reflective feedback is generated by a ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

CHI '14: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems

April 2014

4206 pages

ISBN:9781450324731

DOI:10.1145/2556288

General Chairs:
Matt Jones
Swansea University, Wales, UK
,
Philippe Palanque
Université Paul Sabatier, France
,
Program Chairs:
Albrecht Schmidt
University of Stuttgart, Germany
,
Tovi Grossman
Autodesk Research, Canada

Copyright © 2014 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGCHI: ACM Special Interest Group on Computer-Human Interaction

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 26 April 2014

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

CHI '14

Sponsor:

SIGCHI

CHI '14: CHI Conference on Human Factors in Computing Systems

April 26 - May 1, 2014

Ontario, Toronto, Canada

Acceptance Rates

CHI '14 Paper Acceptance Rate 465 of 2,043 submissions, 23%;

Overall Acceptance Rate 6,199 of 26,314 submissions, 24%

Upcoming Conference

CHI 2025

Sponsor:
sigchi

ACM CHI Conference on Human Factors in Computing Systems

April 26 - May 1, 2025

Yokohama , Japan

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

26
Total Citations
View Citations
1,455
Total Downloads

Downloads (Last 12 months)15
Downloads (Last 6 weeks)1

Reflects downloads up to 25 Feb 2025

Other Metrics

View Author Metrics

Citations

Cited By

Maierhöfer VSchmid AWimmer RCiolfi LHogan TDöring TJenkins Tvan Dijk JHuron SLi ZCoyle DSigner B(2024)TipTrack: Precise, Low-Latency, Robust Optical Pen Tracking on Arbitrary Surfaces Using an IR-Emitting Pen TipProceedings of the Eighteenth International Conference on Tangible, Embedded, and Embodied Interaction10.1145/3623509.3633366(1-13)Online publication date: 11-Feb-2024
https://dl.acm.org/doi/10.1145/3623509.3633366
Balaji AKimber CLi DWu SDu RKim D(2023)RetroSphereProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/35694796:4(1-36)Online publication date: 11-Jan-2023
https://dl.acm.org/doi/10.1145/3569479
Zhang YLi ZXu SLi CYang JTong XGuo B(2023)RemoteTouch: Enhancing Immersive 3D Video Communication with Hand Touch2023 IEEE Conference Virtual Reality and 3D User Interfaces (VR)10.1109/VR55154.2023.00016(1-10)Online publication date: Mar-2023
https://doi.org/10.1109/VR55154.2023.00016
Grubert J(2023)Mixed Reality Interaction TechniquesSpringer Handbook of Augmented Reality10.1007/978-3-030-67822-7_5(109-129)Online publication date: 1-Jan-2023
https://doi.org/10.1007/978-3-030-67822-7_5
Komninos AKavvathas VSimou I(2022)LEDBoard: Using Visual Feedback to Support Text Entry with Physical KeyboardsProceedings of the International Conference on Ubiquitous Computing & Ambient Intelligence (UCAmI 2022)10.1007/978-3-031-21333-5_85(852-863)Online publication date: 21-Nov-2022
https://doi.org/10.1007/978-3-031-21333-5_85
Wei HLi PTang KWang WChen X(2019)Alternating Electric Field-Based Static Gesture-Recognition TechnologySensors10.3390/s1910237519:10(2375)Online publication date: 23-May-2019
https://doi.org/10.3390/s19102375
Becker VKalbermatter SMayer SSörös GLee BLee GScott STory MKim J(2019)Tailored ControlsProceedings of the 2019 ACM International Conference on Interactive Surfaces and Spaces10.1145/3343055.3359700(289-301)Online publication date: 10-Nov-2019
https://dl.acm.org/doi/10.1145/3343055.3359700
Otte ASchneider DMenzner TGesslein TGagel PGrubert J(2019)Evaluating Text Entry in Virtual Reality using a Touch-sensitive Physical Keyboard2019 IEEE International Symposium on Mixed and Augmented Reality Adjunct (ISMAR-Adjunct)10.1109/ISMAR-Adjunct.2019.000-4(387-392)Online publication date: Oct-2019
https://doi.org/10.1109/ISMAR-Adjunct.2019.000-4
Li TXiong XXie YHito GYang XZhou X(2017)Reconstructing Hand Poses Using Visible LightProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/31309371:3(1-20)Online publication date: 11-Sep-2017
https://dl.acm.org/doi/10.1145/3130937
Jang YJeon IKim TWoo W(2017)Metaphoric Hand Gestures for Orientation-Aware VR Object Manipulation With an Egocentric ViewpointIEEE Transactions on Human-Machine Systems10.1109/THMS.2016.2611824(1-15)Online publication date: 2017
https://doi.org/10.1109/THMS.2016.2611824
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten