research-article

Creepy Tracker Toolkit for Context-aware Interfaces

Authors:

Maurício Sousa,

Rafael Kuffner Dos Anjos,

Daniel Medeiros,

Alfredo Ferreira,

Alberto Raposo,

João Madeiras Pereira,

Joaquim JorgeAuthors Info & Claims

ISS '17: Proceedings of the 2017 ACM International Conference on Interactive Surfaces and Spaces

Pages 191 - 200

https://doi.org/10.1145/3132272.3134113

Published: 17 October 2017 Publication History

Abstract

Context-aware pervasive applications can improve user experiences by tracking people in their surroundings. Such systems use multiple sensors to gather information regarding people and devices. However, when developing novel user experiences, researchers are left to building foundation code to support multiple network-connected sensors, a major hurdle to rapidly developing and testing new ideas.

We introduce Creepy Tracker, an open-source toolkit to ease prototyping with multiple commodity depth cameras. It automatically selects the best sensor to follow each person, handling occlusions and maximizing interaction space, while providing full-body tracking in scalable and extensible manners. It also keeps position and orientation of stationary interactive surfaces while offering continuously updated point-cloud user representations combining both depth and color data. Our performance evaluation shows that, although slightly less precise than marker-based optical systems, Creepy Tracker provides reliable multi-joint tracking without any wearable markers or special devices. Furthermore, implemented representative scenarios show that Creepy Tracker is well suited for deploying spatial and context-aware interactive experiences.

References

[1]

Aggarwal, J. K., and Ryoo, M. S. Human activity analysis: A review. ACM Computing Surveys (CSUR) 43, 3 (2011), 16.

[2]

Annett, M., Grossman, T., Wigdor, D., and Fitzmaurice, G. Medusa: a proximity-aware multi-touch tabletop. In Proceedings of the 24th annual ACM symposium on User interface software and technology, ACM (2011), 337--346.

Digital Library

[3]

Antifakos, S., and Schiele, B. Beyond position awareness. Personal and Ubiquitous Computing 6, 5--6 (2002), 313--317.

Digital Library

[4]

Ark, W. S., and Selker, T. A look at human interaction with pervasive computers. IBM systems journal 38, 4 (1999), 504--507.

Digital Library

[5]

Augsten, T., Kaefer, K., Meusel, R., Fetzer, C., Kanitz, D., Stoff, T., Becker, T., Holz, C., and Baudisch, P. Multitoe: High-precision interaction with back-projected floors based on high-resolution multi-touch input. In Proceedings of the 23Nd Annual ACM Symposium on User Interface Software and Technology, UIST '10, ACM (New York, NY, USA, 2010), 209--218.

Digital Library

[6]

Azimi, M. Skeletal joint smoothing white paper. Tech. rep., 2012. http://msdn.microsoft.com/en-us/library/jj131429.aspx.

[7]

Ballendat, T., Marquardt, N., and Greenberg, S. Proxemic interaction: Designing for a proximity and orientation-aware environment. In ACM International Conference on Interactive Tabletops and Surfaces, ITS '10, ACM (New York, NY, USA, 2010), 121--130.

Digital Library

[8]

Beck, S., Kunert, A., Kulik, A., and Froehlich, B. Immersive group-to-group telepresence. IEEE Transactions on Visualization and Computer Graphics 19, 4 (2013), 616--625.

Digital Library

[9]

Bolt, R. A. Put-that-there: Voice and gesture at the graphics interface. In Proceedings of the 7th Annual Conference on Computer Graphics and Interactive Techniques, SIGGRAPH '80, ACM (New York, NY, USA, 1980), 262--270.

Digital Library

[10]

Dey, A. K., Abowd, G. D., and Salber, D. A conceptual framework and a toolkit for supporting the rapid prototyping of context-aware applications. Human-computer interaction 16, 2 (2001), 97--166.

Digital Library

[11]

Fails, J. A., and Olsen, J. D. Light widgets: interacting in every-day spaces. In Proceedings of the 7th international conference on Intelligent user interfaces, ACM (2002), 63--69.

Digital Library

[12]

Grønbæk, K., Iversen, O. S., Kortbek, K. J., Nielsen, K. R., and Aagaard, L. Igamefloor: A platform for co-located collaborative games. In Proceedings of the International Conference on Advances in Computer Entertainment Technology, ACE '07, ACM (New York, NY, USA, 2007), 64--71.

Digital Library

[13]

Gugenheimer, J., Stemasov, E., Frommel, J., and Rukzio, E. Sharevr: Enabling co-located experiences for virtual reality between hmd and non-hmd users. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems, CHI '17, ACM (New York, NY, USA, 2017), 4021--1033.

Digital Library

[14]

Hall, E. T. The hidden dimension. Doubleday & Co, 1966.

[15]

Harris, A., Rick, J., Bonnett, V., Yuill, N., Fleck, R., Marshall, P., and Rogers, Y. Around the table: Are multiple-touch surfaces better than single-touch for children's collaborative interactions? In Proceedings of the 9th international conference on Computer supported collaborative learning-Volume 1, International Society of the Learning Sciences (2009), 335--344.

[16]

Insafutdinov, E., Pishchulin, L., Andres, B., Andriluka, M., and Schiele, B. Deepercut: A deeper, stronger, and faster multi-person pose estimation model. arXiv preprint arXiv:1605.03170 (2016).

[17]

Jokela, T., Ojala, J., and Olsson, T. A diary study on combining multiple information devices in everyday activities and tasks. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems, CHI '15, ACM (New York, NY, USA, 2015), 3903--3912.

Digital Library

[18]

Jones, B., Sodhi, R., Murdock, M., Mehra, R., Benko, H., Wilson, A., Ofek, E., MacIntyre, B., Raghuvanshi, N., and Shapira, L. Roomalive: Magical experiences enabled by scalable, adaptive projector-camera units. In Proceedings of the 27th Annual ACM Symposium on User Interface Software and Technology, UIST '14, ACM (New York, NY, USA, 2014), 637--644.

Digital Library

[19]

Marquardt, N., Ballendat, T., Boring, S., Greenberg, S., and Hinckley, K. Gradual engagement: Facilitating information exchange between digital devices as a function of proximity. In Proceedings of the 2012 ACM International Conference on Interactive Tabletops and Surfaces, ITS '12, ACM (New York, NY, USA, 2012), 31--40.

Digital Library

[20]

Marquardt, N., Diaz-Marino, R., Boring, S., and Greenberg, S. The proximity toolkit: Prototyping proxemic interactions in ubiquitous computing ecologies. In Proceedings of the 24th Annual ACM Symposium on User Interface Software and Technology, UIST '11, ACM (New York, NY, USA, 2011), 315--326.

Digital Library

[21]

Marquardt, N., Hinckley, K., and Greenberg, S. Cross-device interaction via micro-mobility and f-formations. In Proceedings of the 25th Annual ACM Symposium on User Interface Software and Technology, UIST '12, ACM (New York, NY, USA, 2012), 13--22.

Digital Library

[22]

Marquardt, N., Jota, R., Greenberg, S., and Jorge, J. The continuous interaction space: interaction techniques unifying touch and gesture on and above a digital surface. Human-Computer Interaction--INTERACT 2011 (2011), 461--476.

[23]

Marshall, P., Hornecker, E., Morris, R., Dalton, N. S., and Rogers, Y. When the fingers do the talking: A study of group participation with varying constraints to a tabletop interface. In Horizontal Interactive Human Computer Systems, 2008. TABLETOP 2008. 3rd IEEE International Workshop on, IEEE (2008), 33--10.

[24]

Matthews, T., Dey, A. K., Mankoff, J., Carter, S., and Rattenbury, T. A toolkit for managing user attention in peripheral displays. In Proceedings of the 17th annual ACM symposium on User interface software and technology, ACM (2004), 247--256.

Digital Library

[25]

Mendes, D., Relvas, F., Ferreira, A., and Jorge, J. The benefits of dof separation in mid-air 3d object manipulation. In Proceedings of the 22Nd ACM Conference on Virtual Reality Software and Technology, VRST '16, ACM (New York, NY, USA, 2016), 261--268.

Digital Library

[26]

Orts-Escolano, S., Rhemann, C., Fanello, S., Chang, W., Kowdle, A., Degtyarev, Y., Kim, D., Davidson, P. L., Khamis, S., Dou, M., et al. Holoportation: Virtual 3d teleportation in real-time. In Proceedings of the 29th Annual Symposium on User Interface Software and Technology, ACM (2016), 741--754.

Digital Library

[27]

Pejsa, T., Kantor, J., Benko, H., Ofek, E., and Wilson, A. Room2room: Enabling life-size telepresence in a projected augmented reality environment. In Proceedings of the 19th ACM Conference on Computer-Supported Cooperative Work & Social Computing, ACM (2016), 1716--1725.

Digital Library

[28]

Pierce, J. S., Forsberg, A. S., Conway, M. J., Hong, S., Zeleznik, R. C., and Mine, M. R. Image plane interaction techniques in 3d immersive environments. In Proceedings of the 1997 symposium on Interactive 3D graphics, ACM (1997), 39-ff.

Digital Library

[29]

Raskar, R., Welch, G., Cutts, M., Lake, A., Stesin, L., and Fuchs, H. The office of the future: A unified approach to image-based modeling and spatially immersive displays. In Proceedings of the 25th Annual Conference on Computer Graphics and Interactive Techniques, SIGGRAPH '98, ACM (New York, NY, USA, 1998), 179--188.

Digital Library

[30]

Rekimoto, J., and Nagao, K. The world through the computer: Computer augmented interaction with real world environments. In Proceedings of the 8th annual ACM symposium on User interface and software technology, ACM (1995), 29--36.

Digital Library

[31]

Schilit, B., Adams, N., and Want, R. Context-aware computing applications. In Mobile Computing Systems and Applications, 1994. WMCSA 1994. First Workshop on, IEEE (1994), 85--90.

Digital Library

[32]

Seyed, T., Azazi, A., Chan, E., Wang, Y., and Maurer, F. Sod-toolkit: A toolkit for interactively prototyping and developing multi-sensor, multi-device environments. In Proceedings of the 2015 International Conference on Interactive Tabletops & Surfaces, ITS '15, ACM (New York, NY, USA, 2015), 171--180.

Digital Library

[33]

Shotton, J., Sharp, T., Kipman, A., Fitzgibbon, A., Finocchio, M., Blake, A., Cook, M., and Moore, R. Real-time human pose recognition in parts from single depth images. Communications of the ACM 56, 1 (2013), 116--124.

Digital Library

[34]

Slater, M., and Usoh, M. Body centred interaction in immersive virtual environments. Artificial life and virtual reality 1, 1994 (1994), 125--148.

[35]

Sousa, M., Mendes, D., Ferreira, A., Pereira, J. M., and Jorge, J. Eery space: facilitating virtual meetings through remote proxemics. In Human-Computer Interaction, Springer, Cham (2015), 622--629.

Digital Library

[36]

Tang, R., Alizadeh, H., Tang, A., Bateman, S., and Jorge, J. A. Physio@home: Design explorations to support movement guidance. In CHI '14 Extended Abstracts on Human Factors in Computing Systems, CHI EA '14, ACM (New York, NY, USA, 2014), 1651--1656.

Digital Library

[37]

Vermeulen, J., Luyten, K., Coninx, K., Marquardt, N., and Bird, J. Proxemic flow: Dynamic peripheral floor visualizations for revealing and mediating large surface interactions. In Human-Computer Interaction, Springer (2015), 264--281.

[38]

Vogel, D., and Balakrishnan, R. Interactive public ambient displays: Transitioning from implicit to explicit, public to personal, interaction with multiple users. In Proceedings of the 17th Annual ACM Symposium on User Interface Software and Technology, UIST '04, ACM (New York, NY, USA, 2004), 137--146.

Digital Library

[39]

Weiser, M. The computer for the 21st century. Scientific american 265, 3 (1991), 94--104.

[40]

Weiser, M. The computer for the 21st century. IEEE pervasive computing 1, 1 (2002), 19--25.

Digital Library

[41]

Wilson, A. D. Depth-sensing video cameras for 3d tangible tabletop interaction. In Horizontal Interactive Human-Computer Systems, 2007. TABLETOP'07. Second Annual IEEE International Workshop on, IEEE (2007), 201--204.

[42]

Wilson, A. D., and Benko, H. Combining multiple depth cameras and projectors for interactions on, above and between surfaces. In Proceedings of the 23nd annual ACM symposium on User interface software and technology, ACM (2010), 273--282.

Digital Library

[43]

Wilson, A. D., and Benko, H. Crossmotion: Fusing device and image motion for user identification, tracking and device association. In Proceedings of the 16th International Conference on Multimodal Interaction, ICMI '14, ACM (New York, NY, USA, 2014), 216--223.

Digital Library

[44]

Wu, C.-J., Houben, S., and Marquardt, N. Eaglesense: Tracking people and devices in interactive spaces using real-time top-view depth-sensing. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems, CHI '17, ACM (New York, NY, USA, 2017), 3929--3942.

Digital Library

Cited By

Bateman SGutwin CMansoor HNacenta Mvan der Kamp MBaliesnyi MGagnon KRollheiser J(2023)WAMS: A Flexible API for Visual Workspaces Across Multiple SurfacesProceedings of the ACM on Human-Computer Interaction10.1145/35932367:EICS(1-40)Online publication date: 19-Jun-2023
https://dl.acm.org/doi/10.1145/3593236
Irlitti ALatifoglu MZhou QReinoso MHoang TVelloso EVetere F(2023)Volumetric Mixed Reality Telepresence for Real-time Cross Modality CollaborationProceedings of the 2023 CHI Conference on Human Factors in Computing Systems10.1145/3544548.3581277(1-14)Online publication date: 19-Apr-2023
https://dl.acm.org/doi/10.1145/3544548.3581277
Li JSousa MMahadevan KWang BAoyagui PYu NYang ABalakrishnan RTang AGrossman T(2023)Stargazer: An Interactive Camera Robot for Capturing How-To Videos Based on Subtle Instructor CuesProceedings of the 2023 CHI Conference on Human Factors in Computing Systems10.1145/3544548.3580896(1-16)Online publication date: 19-Apr-2023
https://dl.acm.org/doi/10.1145/3544548.3580896
Show More Cited By

Index Terms

Creepy Tracker Toolkit for Context-aware Interfaces
1. Human-centered computing
  1. Human computer interaction (HCI)

Recommendations

Context tracker: Exploring supporters and distracters in unconstrained environments
CVPR '11: Proceedings of the 2011 IEEE Conference on Computer Vision and Pattern Recognition

Visual tracking in unconstrained environments is very challenging due to the existence of several sources of varieties such as changes in appearance, varying lighting conditions, cluttered background, and frame-cuts. A major factor causing tracking ...
Out of sight: a toolkit for tracking occluded human joint positions

Real-time identification and tracking of the joint positions of people can be achieved with off-the-shelf sensing technologies such as the Microsoft Kinect, or other camera-based systems with computer vision. However, tracking is constrained by the ...
A Multi-Camera 6-DOF Pose Tracker
ISMAR '04: Proceedings of the 3rd IEEE/ACM International Symposium on Mixed and Augmented Reality

Most of the work in head-pose tracking has concentrated on single-camera systems with a relatively small field of view which have limited accuracy because features are only observed in a single viewing direction. We present a multi-camera pose tracker ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

ISS '17: Proceedings of the 2017 ACM International Conference on Interactive Surfaces and Spaces

October 2017

504 pages

ISBN:9781450346917

DOI:10.1145/3132272

General Chairs:
Sriram Subramanian
University of Sussex, UK
,
Jürgen Steimle
Saarland University, Germany
,
Program Chairs:
Raimund Dachselt
Technische Universität Dresden, Germany
,
Diego Martinez Plasencia
University of Sussex, UK
,
Tovi Grossman
Autodesk Research

Copyright © 2017 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

SIGCHI: ACM Special Interest Group on Computer-Human Interaction

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 17 October 2017

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed limited

Funding Sources

Fundação para a Ciência e a Tecnologia
IT-Medex

Conference

ISS '17

Sponsor:

SIGCHI

ISS '17: Interactive Surfaces and Spaces

October 17 - 20, 2017

Brighton, United Kingdom

Acceptance Rates

ISS '17 Paper Acceptance Rate 32 of 119 submissions, 27%;

Overall Acceptance Rate 147 of 533 submissions, 28%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

19
Total Citations
View Citations
354
Total Downloads

Downloads (Last 12 months)15
Downloads (Last 6 weeks)4

Reflects downloads up to 05 Mar 2025

Other Metrics

View Author Metrics

Citations

Cited By

Bateman SGutwin CMansoor HNacenta Mvan der Kamp MBaliesnyi MGagnon KRollheiser J(2023)WAMS: A Flexible API for Visual Workspaces Across Multiple SurfacesProceedings of the ACM on Human-Computer Interaction10.1145/35932367:EICS(1-40)Online publication date: 19-Jun-2023
https://dl.acm.org/doi/10.1145/3593236
Irlitti ALatifoglu MZhou QReinoso MHoang TVelloso EVetere F(2023)Volumetric Mixed Reality Telepresence for Real-time Cross Modality CollaborationProceedings of the 2023 CHI Conference on Human Factors in Computing Systems10.1145/3544548.3581277(1-14)Online publication date: 19-Apr-2023
https://dl.acm.org/doi/10.1145/3544548.3581277
Li JSousa MMahadevan KWang BAoyagui PYu NYang ABalakrishnan RTang AGrossman T(2023)Stargazer: An Interactive Camera Robot for Capturing How-To Videos Based on Subtle Instructor CuesProceedings of the 2023 CHI Conference on Human Factors in Computing Systems10.1145/3544548.3580896(1-16)Online publication date: 19-Apr-2023
https://dl.acm.org/doi/10.1145/3544548.3580896
Wu SLi JSousa MGrossman T(2023)Investigating Guardian Awareness Techniques to Promote Safety in Virtual Reality2023 IEEE Conference Virtual Reality and 3D User Interfaces (VR)10.1109/VR55154.2023.00078(631-640)Online publication date: Mar-2023
https://doi.org/10.1109/VR55154.2023.00078
Herskovitz JCheng YGuo ASample ANebeling M(2022)XSpace: An Augmented Reality Toolkit for Enabling Spatially-Aware Distributed CollaborationProceedings of the ACM on Human-Computer Interaction10.1145/35677216:ISS(277-302)Online publication date: 14-Nov-2022
https://dl.acm.org/doi/10.1145/3567721
Raheb KStergiou MKatifori AIoannidis Y(2021)Moving in the CubeJournal on Computing and Cultural Heritage 10.1145/342737914:2(1-26)Online publication date: 29-May-2021
https://dl.acm.org/doi/10.1145/3427379
Mendes DReis SGuerreiro JNicolau H(2020)Collaborative Tabletops for Blind PeopleProceedings of the ACM on Human-Computer Interaction10.1145/34273254:ISS(1-19)Online publication date: 4-Nov-2020
https://dl.acm.org/doi/10.1145/3427325
Grønbæk JKnudsen MO'Hara KKrogh PVermeulen JPetersen MBernhaupt RMueller FVerweij DAndres JMcGrenere JCockburn AAvellino IGoguey ABjørn PZhao SSamson BKocielnik R(2020)Proxemics Beyond ProximityProceedings of the 2020 CHI Conference on Human Factors in Computing Systems10.1145/3313831.3376379(1-14)Online publication date: 21-Apr-2020
https://dl.acm.org/doi/10.1145/3313831.3376379
Medeiros DSousa MRaposo AJorge J(2020)Magic Carpet: Interaction Fidelity for Flying in VRIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2019.290520026:9(2793-2804)Online publication date: 1-Sep-2020
https://doi.org/10.1109/TVCG.2019.2905200
Sousa MMendes DJorge JJorge JKim JVan Eyken HGlencross MMitchell KNarumi T(2019)Safe Walking in VRProceedings of the 17th ACM SIGGRAPH International Conference on Virtual-Reality Continuum and its Applications in Industry10.1145/3359997.3365737(1-2)Online publication date: 14-Nov-2019
https://dl.acm.org/doi/10.1145/3359997.3365737
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten