skip to main content
10.1145/2556288.2557336acmconferencesArticle/Chapter ViewAbstractPublication PageschiConference Proceedingsconference-collections
research-article

RetroDepth: 3D silhouette sensing for high-precision input on and above physical surfaces

Published: 26 April 2014 Publication History

Abstract

We present RetroDepth, a new vision-based system for accurately sensing the 3D silhouettes of hands, styluses, and other objects, as they interact on and above physical surfaces. Our setup is simple, cheap, and easily reproducible, comprising of two infrared cameras, diffuse infrared LEDs, and any off-the-shelf retro-reflective material. The retro-reflector aids image segmentation, creating a strong contrast between the surface and any object in proximity. A new highly efficient stereo matching algorithm precisely estimates the 3D contours of interacting objects and the retro-reflective surfaces. A novel pipeline enables 3D finger, hand and object tracking, as well as gesture recognition, purely using these 3D contours. We demonstrate high-precision sensing, allowing robust disambiguation between a finger or stylus touching, pressing or interacting above the surface. This allows many interactive scenarios that seamlessly mix together freehand 3D interactions with touch, pressure and stylus input. As shown, these rich modalities of input are enabled on and above any retro-reflective surface, including custom "physical widgets" fabricated by users. We compare our system with Kinect and Leap Motion, and conclude with limitations and future work.

Supplementary Material

suppl.mov (pn2022-file3.mp4)
Supplemental video
MP4 File (p1377-sidebyside.mp4)

References

[1]
3Gear Systems Inc. http://threegear.com/, 2013.
[2]
Agarwal, A., Izadi, S., Chandraker, M., and Blake, A. High precision multi-touch sensing on surfaces using cameras. In Tabletop'07, 197--200.
[3]
Akaoka, E., Ginn, T., and Vertegaal, R. Displayobjects: prototyping functional physical interfaces on 3d styrofoam, paper or cardboard models. In TEI'10, ACM (2010), 49--56.
[4]
Annett, M., Grossman, T., Wigdor, D., and Fitzmaurice, G. Medusa: a proximity-aware multi-touch tabletop. In UIST'11, ACM (2011), 337--346.
[5]
Benko, H., Jota, R., and Wilson, A. Miragetable: freehand interaction on a projected augmented reality tabletop. In CHI'12, ACM (2012), 199--208.
[6]
Bleyer, M., Rhemann, C., and Rother, C. Patchmatch stereo - stereo matching with slanted support windows. In British Machine Vision Conference (2011).
[7]
Breiman, L. Random Forests. Machine Learning 45, 1 (Oct. 2001), 5--32.
[8]
Brown, M. Z., Burschka, D., and Hager, G. D. Advances in computational stereo. PAMI 25, 8 (2003), 993--1008.
[9]
Davis, J. W., and Bobick, A. F. Sideshow: A silhouettebased interactive dual-screen environment. Tech. rep., MIT, 1998.
[10]
Dippon, A., and Klinker, G. Kinecttouch: accuracy test for a very low-cost 2.5 d multitouch tracking system. In ITS'11, ACM (2011), 49--52.
[11]
Haptix. http://www.haptixtouch.com/, 2013.
[12]
Haubner, N., Schwanecke, U., Dörner, R., Lehmann, S., and Luderschmidt, J. Detecting interaction above digital tabletops using a single depth camera. Machine Vision and Applications (2013), 1--13.
[13]
Hilliges, O., Izadi, S., Wilson, A. D., Hodges, S., Garcia-Mendoza, A., and Butz, A. Interactions in the air: adding further depth to interactive tabletops. In UIST'09, ACM (2009), 139--148.
[14]
Hilliges, O., Kim, D., Izadi, S., Weiss, M., and Wilson, A. Holodesk: direct 3d interactions with a situated seethrough display. In CHI'12, ACM (2012), 2421--2430.
[15]
Hirsch, M., Lanman, D., Holtzman, H., and Raskar, R. Bidi screen: a thin, depth-sensing lcd for 3d interaction using light fields. In TOG, vol. 28, ACM (2009), 159.
[16]
Izadi, S., Agarwal, A., Criminisi, A., Winn, J., Blake, A., and Fitzgibbon, A. C-slate: a multi-touch and object recognition system for remote collaboration using horizontal surfaces. In Tabletop'07, IEEE (2007), 3--10.
[17]
Izadi, S., Hodges, S., Butler, A., West, D., Rrustemi, A., Molloy, M., and Buxton,W. Thinsight: a thin form-factor interactive surface technology. CACM 52, 12, 90--98.
[18]
Izadi, S., Hodges, S., Taylor, S., Rosenfeld, D., Villar, N., Butler, A., and Westhues, J. Going beyond the display: a surface technology with an electronically switchable diffuser. In UIST'08, ACM (2008), 269--278.
[19]
Keskin, C., Kirac, F., Kara, Y. E., and Akarun, L. Hand Pose Estimation and Hand Shape Classification Using Multi-layered Randomized Decision Forests. In ECCV'12 (2012).
[20]
Koike, H., Sato, Y., and Kobayashi, Y. Integrating paper and digital information on enhanceddesk: a method for realtime finger tracking on an augmented desk system. TOCHI 8, 4 (2001), 307--322.
[21]
Krueger, M. W. Artificial reality II, vol. 10. Addison-Wesley Reading (Ma), 1991.
[22]
Lanman, D., and Taubin, G. Build your own 3d scanner: 3d photography for beginners. In ACM SIGGRAPH 2009 Courses, ACM (2009), 8.
[23]
Lee, J., Park, K. S., and Hahn, M. The 3d sensor table for bare hand tracking and posture recognition. In Advances in Multimedia Modeling. Springer, 2006, 138--146.
[24]
Liu, Y., Weibel, N., and Hollan, J. Interactive space: A framework for prototyping multi-touch interaction on and above the desktop. In CHI'13, vol. 13 (2013).
[25]
Malik, S., and Laszlo, J. Visual touchpad: a two-handed gestural input device. In ICMI'04, 289--296.
[26]
Marquardt, N., Jota, R., Greenberg, S., and Jorge, J. A. The continuous interaction space: interaction techniques unifying touch and gesture on and above a digital surface. In INTERACT'11. Springer, 2011, 461--476.
[27]
Marroquim, R., Kraus, M., and Cavalcanti, P. R. Efficient point-based rendering using image reconstruction. In SPBG (2007), 101--108.
[28]
Matsushita, N., and Rekimoto, J. Holowall: designing a finger, hand, body, and object sensitive wall. In UIST'97, ACM (1997), 209--210.
[29]
Moeller, J., Kerne, A., and Moeller, J. Zerotouch: An optical multi-touch and free-air interaction architecture. In CH'12, ACM (2012), 2165--2174.
[30]
Paradiso, J. A. Several sensor approaches that retrofit large surfaces for interactivity. In UbiComp 2002 (2002).
[31]
Remondino, F., and Stoppa, D. Tof range-imaging cameras. Springer, 2013.
[32]
Scharstein, D., and Szeliski, R. A taxonomy and evaluation of dense two-frame stereo correspondence algorithms. In IJCV (2002).
[33]
Shotton, J., Fitzgibbon, A., Cook, M., Sharp, T., Finocchio, M., Moore, R., Kipman, A., and Blake, A. Realtime Human Pose Recognition in Parts from Single Depth Images. In CVPR (2011).
[34]
Strecha, C., and Van Gool, L. PDE-based multi-view depth estimation. In Proc. 3DIMPVT (2002), 416--425.
[35]
Subramanian, S., Aliakseyeu, D., and Lucero, A. Multilayer interaction for digital tables. In UIST'06, ACM (2006), 269--272.
[36]
Takeoka, Y., Miyaki, T., and Rekimoto, J. Z-touch: an infrastructure for 3d gesture interaction in the proximity of tabletop surfaces. In ITS'10, ACM (2010), 91--94.
[37]
Ullmer, B., and Ishii, H. The metadesk: models and prototypes for tangible user interfaces. In UIST'97, ACM (1997), 223--232.
[38]
Wang, R., Paris, S., and Popović, J. 6d hands: markerless hand-tracking for computer aided design. In UIST'11, ACM (2011), 549--558.
[39]
Wellner, P. Interacting with paper on the digitaldesk. CACM 36, 7 (1993), 87--96.
[40]
Wesche, G., and Seidel, H.-P. Freedrawer: a free-form sketching system on the responsive workbench. In VRST, ACM (2001), 167--174.
[41]
Wilson, A. D. Playanywhere: a compact interactive tabletop projection-vision system. In UIST'05, ACM (2005), 83--92.
[42]
Wilson, A. D. Robust computer vision-based detection of pinching for one and two-handed gesture input. In UIST'06, ACM (2006), 255--258.
[43]
Wilson, A. D. Depth-sensing video cameras for 3d tangible tabletop interaction. In Tabletop'07, IEEE (2007), 201--204.
[44]
Wilson, A. D. Using a depth camera as a touch sensor. In ITS'10, ACM (2010), 69--72.

Cited By

View all
  • (2024)TipTrack: Precise, Low-Latency, Robust Optical Pen Tracking on Arbitrary Surfaces Using an IR-Emitting Pen TipProceedings of the Eighteenth International Conference on Tangible, Embedded, and Embodied Interaction10.1145/3623509.3633366(1-13)Online publication date: 11-Feb-2024
  • (2023)RetroSphereProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/35694796:4(1-36)Online publication date: 11-Jan-2023
  • (2023)RemoteTouch: Enhancing Immersive 3D Video Communication with Hand Touch2023 IEEE Conference Virtual Reality and 3D User Interfaces (VR)10.1109/VR55154.2023.00016(1-10)Online publication date: Mar-2023
  • Show More Cited By

Index Terms

  1. RetroDepth: 3D silhouette sensing for high-precision input on and above physical surfaces

    Recommendations

    Comments

    Information & Contributors

    Information

    Published In

    cover image ACM Conferences
    CHI '14: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems
    April 2014
    4206 pages
    ISBN:9781450324731
    DOI:10.1145/2556288
    Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

    Sponsors

    Publisher

    Association for Computing Machinery

    New York, NY, United States

    Publication History

    Published: 26 April 2014

    Permissions

    Request permissions for this article.

    Check for updates

    Author Tags

    1. 3D contours
    2. 3D input
    3. contour classification
    4. depth sensing
    5. nui
    6. stereo matching
    7. stylus
    8. touch
    9. vision-based uis

    Qualifiers

    • Research-article

    Conference

    CHI '14
    Sponsor:
    CHI '14: CHI Conference on Human Factors in Computing Systems
    April 26 - May 1, 2014
    Ontario, Toronto, Canada

    Acceptance Rates

    CHI '14 Paper Acceptance Rate 465 of 2,043 submissions, 23%;
    Overall Acceptance Rate 6,199 of 26,314 submissions, 24%

    Upcoming Conference

    CHI 2025
    ACM CHI Conference on Human Factors in Computing Systems
    April 26 - May 1, 2025
    Yokohama , Japan

    Contributors

    Other Metrics

    Bibliometrics & Citations

    Bibliometrics

    Article Metrics

    • Downloads (Last 12 months)15
    • Downloads (Last 6 weeks)1
    Reflects downloads up to 25 Feb 2025

    Other Metrics

    Citations

    Cited By

    View all
    • (2024)TipTrack: Precise, Low-Latency, Robust Optical Pen Tracking on Arbitrary Surfaces Using an IR-Emitting Pen TipProceedings of the Eighteenth International Conference on Tangible, Embedded, and Embodied Interaction10.1145/3623509.3633366(1-13)Online publication date: 11-Feb-2024
    • (2023)RetroSphereProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/35694796:4(1-36)Online publication date: 11-Jan-2023
    • (2023)RemoteTouch: Enhancing Immersive 3D Video Communication with Hand Touch2023 IEEE Conference Virtual Reality and 3D User Interfaces (VR)10.1109/VR55154.2023.00016(1-10)Online publication date: Mar-2023
    • (2023)Mixed Reality Interaction TechniquesSpringer Handbook of Augmented Reality10.1007/978-3-030-67822-7_5(109-129)Online publication date: 1-Jan-2023
    • (2022)LEDBoard: Using Visual Feedback to Support Text Entry with Physical KeyboardsProceedings of the International Conference on Ubiquitous Computing & Ambient Intelligence (UCAmI 2022)10.1007/978-3-031-21333-5_85(852-863)Online publication date: 21-Nov-2022
    • (2019)Alternating Electric Field-Based Static Gesture-Recognition TechnologySensors10.3390/s1910237519:10(2375)Online publication date: 23-May-2019
    • (2019)Tailored ControlsProceedings of the 2019 ACM International Conference on Interactive Surfaces and Spaces10.1145/3343055.3359700(289-301)Online publication date: 10-Nov-2019
    • (2019)Evaluating Text Entry in Virtual Reality using a Touch-sensitive Physical Keyboard2019 IEEE International Symposium on Mixed and Augmented Reality Adjunct (ISMAR-Adjunct)10.1109/ISMAR-Adjunct.2019.000-4(387-392)Online publication date: Oct-2019
    • (2017)Reconstructing Hand Poses Using Visible LightProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/31309371:3(1-20)Online publication date: 11-Sep-2017
    • (2017)Metaphoric Hand Gestures for Orientation-Aware VR Object Manipulation With an Egocentric ViewpointIEEE Transactions on Human-Machine Systems10.1109/THMS.2016.2611824(1-15)Online publication date: 2017
    • Show More Cited By

    View Options

    Login options

    View options

    PDF

    View or Download as a PDF file.

    PDF

    eReader

    View online with eReader.

    eReader

    Figures

    Tables

    Media

    Share

    Share

    Share this Publication link

    Share on social media