Skip to main content

Advertisement

SpringerLink
Log in

Detecting interaction above digital tabletops using a single depth camera

  • Original Paper
  • Published:
Machine Vision and Applications Aims and scope Submit manuscript

Abstract

Digital tabletop environments offer a huge potential to realize application scenarios where multiple users interact simultaneously or aim to solve collaborative tasks. So far, research in this field focuses on touch and tangible interaction, which only takes place on the tabletop’s surface. First approaches aim at involving the space above the surface, e.g., by employing freehand gestures. However, these are either limited to specific scenarios or employ obtrusive tracking solutions. In this paper, we propose an approach to unobtrusively segment and detect interaction above a digital surface using a depth sensing camera. To achieve this, we adapt a previously presented approach that segments arms in depth data from a front-view to a top-view setup facilitating the detection of hand positions. Moreover, we propose a novel algorithm to merge segments and give a comparison to the original segmentation algorithm. Since the algorithm involves a large number of parameters, estimating the optimal configuration is necessary. To accomplish this, we describe a low effort approach to estimate the parameter configuration based on simulated annealing. An evaluation of our system to detect hands shows that a repositioning precision of approximately 1 cm is achieved. This accuracy is sufficient to reliably realize interaction metaphors above a surface.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11

Similar content being viewed by others

Notes

  1. http://www.xbox.com/kinect

  2. http://www.openni.org

  3. http://openkinect.org

References

  1. Annett, M., Grossman, T., Wigdor, D., Fitzmaurice, G.: Medusa: a proximity-aware multi-touch tabletop. In: Proceedings of the 24th Annual ACM Symposium on User Interface Software and Technology, UIST ’11, pp. 337–346. ACM, New York (2011)

  2. Benko, H.: Beyond flat surface computing: challenges of depth-aware and curved interfaces. In: Proceeding of the 17th ACM International Conference on Multimedia, pp. 935–944. ACM, new York (2009)

  3. Benko, H., Morris, M.R., Brush, A.J.B., Wilson, A.D.: Insights on interactive tabletops: a survey of researchers and developers. In: Technical Report MSR-TR-2009-22, Microsoft Research Technical, Report (2009)

  4. Hilliges, O., Izadi, S., Wilson, A.D., Hodges, S., Garcia-Mendoza, A., Butz, A.: Interactions in the air: adding further depth to interactive tabletops. In: UIST ’09: Proceeding of the 22nd Annual ACM Symposis on User Interface Software and Technology, pp. 139–148. ACM, New York (2009)

  5. Huppmann, D., Luderschmidt, J., Haubner, N., Lehmann, S., Dörner, R., Schwanecke, U.: Exploring and evaluating the combined multi-touch and in-the-air tabletop interaction space. In: Geiger, C., Herder, J., Vierjahn, T. (eds.) 9, pp. 37–48. Workshop “Virtuelle und Erweiterte Realität” der GI-Fachgruppe VR/ARShaker Verlag, Aachen (2012)

  6. Jain, H.P., Subramanian, A., Das, S., Mittal, A.: Real-time upper-body human pose estimation using a depth camera. In: Proceedings of the 5th International Conference on Computer Vision/Computer Graphics Collaboration Techniques, MIRAGE’11, pp. 227–238. Springer-Verlag, Berlin (2011)

  7. Kattinakere, R.S., Grossman, T., Subramanian, S.: Modeling steering within above-the-surface interaction layers. In: Proceeding of the SIGCHI Conference on Human Factors in Computer System, pp. 317–326. ACM (2007)

  8. Khoshelham, K., Elberink, S.O.: Accuracy and resolution of kinect depth data for indoor mapping applications. Sensors 12(2), 1437–1454 (2012)

    Article  Google Scholar 

  9. Kirkpatrick, S., Gelatt, C.D., Vecchi, M.P.: Optimization by simulated annealing. Science 220(4598), 671–680 (1983)

    Article  MathSciNet  MATH  Google Scholar 

  10. Knoop, S., Vacek, S., Dillmann, R.: Fusion of 2d and 3d sensor data for articulated body tracking. Robot. Auton. Syst. 57, 321–329 (March 2009)

    Google Scholar 

  11. Kolb, A., Barth, E., Koch, R., Larsen, R.: Time-of-flight sensors in computer graphics. in: Eurographics State of the Art Reports, pp. 119–134 (2009)

  12. Lucero, A., Aliakseyeu, D., Martens, J.-B.: Augmenting mood boards: flexible and intuitive interaction in the context of the design studio. In: Horizontal Interactive Human-Computer Systems, 2007. TABLETOP ’07. 2nd Annual IEEE International Workshop on, pp 147–154 (2007)

  13. Malassiotis, S., Strintzis, M.G.: Real-time hand posture recognition using range data. Image Vis. Comput. 26, 1027–1037 (2008)

    Article  Google Scholar 

  14. Marquardt, N., Jota, R., Greenberg, S., Jorge, J.A.: The continuous interaction space: interaction techniques unifying touch and gesture on and above a digital surface. In: Proceedings of the 13th IFIP TC 13 International Conference on Human-Computer Interaction, Vol. Part III, INTERACT’11, pp. 461–476. Springer-Verlag, Berlin (2011)

  15. Plagemann, C., Ganapathi, V., Koller, D., Thrun, S.: Real-time identification and localization of body parts from depth images. In: Robotics and Automation (ICRA), 2010 IEEE International Conference on, pp. 3108–3113 (2010)

  16. Poppe, R.: Vision-based human motion analysis: an overview. Comput. Vis. Image Underst. 108, 4–18 (October 2007)

  17. Rosenfeld, A.: Some uses of pyramids in image processing and segmentation. In: Proceeding of Image Understanding, Workshop, pp. 112–120 (1980)

  18. Saffer, D.: Designing Gestural Interfaces, 1st edn. O’Reilly Media, Inc., USA (2009)

  19. Schwarz, L., Mkhitaryan, A., Mateus, D., Navab, N.: Estimating human 3D pose from time-of-flight images based on geodesic distances and optical flow. In: Automatic Face Gesture Recognition and Workshops (FG 2011), 2011 IEEE International Conference on, pp. 700–706 (2011)

  20. Shotton, J., Fitzgibbon, A., Cook, M., Sharp, T., Finocchio, M., Moore, R., Kipman, A., Blake, A.: Real-time human pose recognition in parts from single depth images. In: Computer Vision and Pattern Recognition (CVPR), 2011 IEEE Conference on, pp 1297–1304 (2011)

  21. Takeoka, Y., Miyaki, T., Rekimoto, J.: Z-touch: an infrastructure for 3d gesture interaction in the proximity of tabletop surfaces. In: ACM International Conference on Interactive Tabletops and Surfaces, pp. 91–94. ACM (2010)

  22. Wilson, A.D., Benko, H.: Combining multiple depth cameras and projectors for interactions on, above and between surfaces. In: Proceeding of the 23rd Annual ACM Symposis on User Interface Software and Technology, pp. 273–282. ACM (2010)

Download references

Author information

Authors and Affiliations

  1. RheinMain University of Applied Sciences, Unter den Eichen 5, 65195 , Wiesbaden, Germany

    Nadia Haubner, Ulrich Schwanecke, Ralf Dörner, Simon Lehmann & Johannes Luderschmidt

Authors
  1. Nadia Haubner
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Ulrich Schwanecke
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Ralf Dörner
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Simon Lehmann
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Johannes Luderschmidt
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Nadia Haubner.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Haubner, N., Schwanecke, U., Dörner, R. et al. Detecting interaction above digital tabletops using a single depth camera. Machine Vision and Applications 24, 1575–1587 (2013). https://doi.org/10.1007/s00138-013-0538-5

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00138-013-0538-5

Keywords

Search

Navigation