TUIOFX: A JavaFX Toolkit for Shared Interactive Surfaces
Article No.: 10, Pages 1 - 18
Abstract
Building multi-touch multi-user applications for Shared Interactive Surfaces is a complex endeavour that requires fundamental knowledge in touch enabling hardware, gesture recognition, graphical representation of digital information and multi-user interaction. While several specialised toolkits help developers in this effort, we identified a variety of challenges with these toolkits, as for example the lack of cross-platform support, the limited number of touch-enabled multi-user widgets, missing documentation, and lacking community support -- all raising the barriers to entry. In this paper, we present TUIOFX, a toolkit for developing multi-touch, multi-user applications for Shared Interactive Surfaces in Java, which tackles all of the identified problems. The sophisticated implementation of TUIOFX adds support for TUIO-enabled hardware and multi-user interaction under the hood of JavaFX, and leaves the well-learned JavaFX API for the developers fully intact -- thus allowing particularly novices a very quick start. In this paper we provide the technical insights, in the concepts and their elegant implementation.
References
[1]
Hrvoje Benko, Shahram Izadi, Andrew D Wilson, Xiang Cao, Dan Rosenfeld and Ken Hinckley. Design and Evaluation of Interaction Models for Multi-touch Mice. In Proceedings of Graphics Interface 2010 - GI 2010 (May 31 - Jun 2, Ottawa, Canada). Canadian Information Processing Society, Toronto, Canada, 2010. pp. 253--260.
[2]
Christophe Bortolaso, T. Nicholas C. Graham, Stacey D. Scott, Doug Brown and Liam Porter. Design of a Multi-Touch Tabletop for Simulation-Based Training. In Proceedings of International Command and Control Research and Technology Symposium - ICCRTS 2014 (Jun 16--19, Alexandria, VA, USA). International Command and Control Research Institute, Washington, DC, USA, 2014.
[3]
Tom Brinck and Louis M. Gomez. A Collaborative Medium for the Support of Conversational Props. In Proceedings of the 1992 ACM Conference on Computer-Supported Cooperative Work - CSCW 1992 (Oct. 31 - Nov. 4, Toronto, Ontario, Canada). ACM Press, New York, NY, USA 1992. pp. 171--178.
[4]
Andrew Clayphan, Anthony Collins, Christopher Ackad, Bob Kummerfeld and Judy Kay. Firestorm: A Brainstorming Application for Collaborative Group Work at Tabletops. In Proceedings of the 2011 International Conference on Interactive Tabletops & Surfaces - ITS 2011 (Oct. 30 - Nov. 2, Kobe, Japan). ACM Press, New York, NY, USA, 2011. pp. 162--171.
[5]
Paul Dietz and Darren Leigh. Diamondtouch: A Multi-User Touch Technology. In Proceedings of the 14th Annual ACM Symposium on User Interface Software and Technology - UIST 2001 (Nov. 11--14, Orlando, FL, USA). ACM Press, New York, NY, USA, 2001. pp. 219--226.
[6]
Florian Echtler and Andreas Butz. GISpL: Gestures Made Easy. In Proceedings of the Sixth International Conference on Tangible Embedded Interaction - TEI 2012 (Feb.19--22, Kingston, Ontario, Canada). ACM Press, New York, NY, USA, 2012. pp. 233--240.
[7]
Florian Echtler and Gudrun Klinker. A Multitouch Software Architecture. In Proceedings of the 5th Nordic Conference on Human-Computer Interaction - NordiCHI 2008 (Lund, Sweden). ACM Press, New York, NY, USA, 2008. pp. 463--466.
[8]
Mirko Fetter and David Bimamisa. TUIOFX-Toolkit Support for the Development of JavaFX Applications for Interactive Tabletops. In Proceedings of the 15th IFIP TC.13 International Conference on Human-Computer Interaction - INTERACT 2015 (Sept. 14--18, Bamberg, Germany). Springer, Heidelberg, Germany, 2015. pp. 476--479.
[9]
Mirko Fetter, David Bimamisa and Tom Gross. Task-Based Focus and AdHoc-Focus-Territory-Novel Concepts for Shared Interactive Surfaces. In Extended Abstracts of the Conference on Human Factors in Computing Systems - CHI 2016 (May 7--12, San Jose, CA, USA). ACM Press, New York, NY, USA, 2016. pp. 1193--1200.
[10]
Mirko Fetter, Tom Gross and Maxi Hucke. Supporting Social Protocols in Tabletop Interaction through Visual Cues. In Proceedings of the Thirteenth IFIP TC.13 International Conference on Human-Computer Interaction - INTERACT 2011 (Sept. 5--9, Lisbon, Portugal). Springer, Heidelberg, 2011. pp. 435--442.
[11]
Mirko Fetter, Sascha Leicht, David Bimamisa and Tom Gross. Structuring Interaction in Group Decision Making on Tabletops. In Mensch & Computer - 13. Fachuebergreifende Konferenz fuer interaktive und kooperative Medien - M&C 2013 (Sept. 8--11, Bremen, Germany). Oldenbourg, Munich, Germany, 2013. pp. 277--280.
[12]
Thomas E. Hansen, Juan Pablo Hourcade, Mathieu Virbel, Sharath Patali and Tiago Serra. PyMT: A post-WIMP Multi-touch User Interface Toolkit. In Proceedings of the International Conference on Interactive Tabletops and Surfaces - ITS 2009 (Nov. 23--25, Banff, Alberta, Canada). ACM Press, New York, NY, USA, 2009. pp. 17--24.
[13]
Björn Hartmann, Meredith Ringel Morris, Hrvoje Benko and Andrew D. Wilson. Augmenting Interactive Tables with Mice & Keyboards. In Proceedings of the 22nd Annual ACM Symposium on User Interface Software and Technology - UIST 2009 (Oct. 4--7, Victoria, BC, Canada). ACM Press, New York, NY, USA, 2009. pp. 149--152.
[14]
Uta Hinrichs, Mark Hancock, Sheelagh Carpendale and Christopher Collins. Examination of Text-Entry Methods for Tabletop Displays. In Proceedings of the Second Annual IEEE International Workshop on Horizontal Interactive Human-Computer Systems - TABLETOP 2007 (Oct. 10--12, Newport, RI, USA). IEEE Computer Society, Los Alamitos, CA, USA, 2007. pp. 105--112.
[15]
IntuiLab SA. IntuiFace. https://http://www.intuilab.com/, 2017. (Last accessed: 29/03//2017).
[16]
Martin Kaltenbrunner, T Bovermann, Ross Bencina and E Costanza. TUIO - A Protocol for Table-Top Tangible User Interfaces. In Proceedings of the 6th International Workshop on Gesture in Human-Computer Interaction and Simulation - GW 2005 (May 18--20, Ile de Berder, France). 2005.
[17]
Kenrick Kin, Björn Hartmann, Tony DeRose and Maneesh Agrawala. Proton++: A Customizable Declarative Multitouch Framework. In Proceedings of the 25th Annual ACM Symposium on User Interface Software and Technology - UIST 2012 (Oct. 7--10, Cambridge, MA, USA). ACM Press, New York, NY, USA, 2012. pp. 477--486.
[18]
Ulrike Kister, Patrick Reipschlaeger, Fabrice Matulic and Raimund Dachselt. BodyLenses: Embodied Magic Lenses and Personal Territories for Wall Displays. In Proceedings of the 2015 International Conference on Interactive Tabletops & Surfaces - ITS 2015 (Nov. 15--18, Madeira, Portugal). ACM Press, New York, NY, USA, 2015. pp. 117--126.
[19]
Kivy. Kivy: Cross-Platform Python Framework for NUI Development. http://kivy.org/, 2017. (Last accessed: 29/03//2017).
[20]
Sungahn Ko, KyungTae Kim, Tejas Kulkarni and Niklas Elmqvist. Applying Mobile Device Soft Keyboards to Collaborative Multitouch Tabletop Displays. In Proceedings of the 2011 International Conference on Interactive Tabletops & Surfaces - ITS 2011 (Oct. 30 - Nov. 2, Kobe, Japan). ACM Press, New York, NY, USA, 2011. pp. 130--139.
[21]
Werner A Koenig, Roman Raedle and Harald Reiterer. Squidy: A Zoomable Design Environment for Natural User Interfaces. In Extended Abstracts of the Conference on Human Factors in Computing Systems - CHI 2009 (Apr. 4--9, Boston, MA, USA). ACM Press, New York, NY, USA, 2009. pp. 4561--4566.
[22]
Russell Kruger, Sheelagh Carpendale, Stacey D. Scott and Saul Greenberg. Roles of Orientation in Tabletop Collaboration: Comprehension, Coordination and Communication. Computer Supported Cooperative Work (CSCW) 13, 5--6 (2004). pp. 501--537.
[23]
Russell Kruger, Sheelagh Carpendale, Stacey D. Scott and Anthony Tang. Fluid Integration of Rotation and Translation. In Proceedings of the Conference on Human Factors in Computing Systems - CHI 2005 (Apr. 2--7, Portland, USA). ACM Press, New York, NY, USA, 2005. pp. 601--610.
[24]
Uwe Laufs, Christopher Ruff and Jan Zibuschka. MT4j -- A Cross-platform Multi-touch Development Framework. CoRR - http://arxiv.org/abs/1012.0467, 2010. (Last accessed: 12/01/2017).
[25]
I. Scott MacKenzie, Shawn X. Zhang and R. William Soukoreff. Text Entry Using Soft Keyboards. Behaviour & Information Technology 18, 4 (1999). pp. 235--244.
[26]
Nicolai Marquardt, Johannes Kiemer, David Ledo, Sebastian Boring and Saul Greenberg. Designing User-, Hand-, and Handpart-aware Tabletop Interactions with the TouchID Toolkit. In Proceedings of the 2011 International Conference on Interactive Tabletops & Surfaces - ITS 2011 (Oct. 30 - Nov. 2, Kobe, Japan). ACM Press, New York, NY, USA, 2011. pp. 21--30.
[27]
Microsoft Corporation. The Microsoft Surface 2.0 SDK. https://msdn.microsoft.com/en-us/library/ee692162(v=surface.10).aspx, 2014. (Last accessed: 29/03//2017).
[28]
Meredith Ringel Morris, Anthony Cassanego, Andreas Paepcke, Terry Winograd, Anne Marie Piper and Anqi Huang. Mediating Group Dynamics through Tabletop Interface Design. IEEE Computer Graphics and Applications 26, 5 (2006). pp. 65--73.
[29]
Meredith Ringel Morris, Jarrod Lombardo and Daniel Wigdor. WeSearch: Supporting Collaborative Search and Sensemaking on a Tabletop Display. In Proceedings of the ACM 2010 Conference on Computer-Supported Cooperative Work - CSCW 2010 (Feb. 6--10, Savannah, GA, USA). ACM Press, New York, NY, USA, 2010. pp. 401--410.
[30]
Meredith Ringel Morris, Kathy Ryall, Chia Shen, Clifton Forlines and Frederic Vernier. Beyond "Social Protocols": Multi-User Coordination Policies for Co-located Groupware. In Proceedings of the 2004 ACM Conference on Computer-Supported Cooperative Work - CSCW 2004 (Nov. 6--10, Chicago, IL, USA). ACM Press, New York, NY, USA, 2004. pp. 262--265.
[31]
Meredith Ringel Morris, Kathy Ryall, Chia Shen, Clifton Forlines and Frederic Vernier. Release, Relocate, Reorient, Resize. In Extended Abstracts of the Conference on Human Factors in Computing Systems - CHI 2004 (Apr. 24--29, Vienna, Austria). ACM Press, New York, NY, USA, 2004. pp. 1441--1444.
[32]
MultiTouch Ltd. Cornerstone SDK. https://cornerstone.multitouch.fi/developer_guide, 2015. (Last accessed: 29/03//2017).
[33]
Miguel A. Nacenta, David Pinelle, Dane Stuckel and Carl Gutwin. The Effects of Interaction Technique on Coordination in Tabletop Groupware. In Proceedings of Graphics Interface 2007 - GI 2007 (May 28 - 30, Montréal, Canada). Canadian Information Processing Society, Mississauga, Ontario, Canada, 2007. pp. 191--198.
[34]
Michael Nebeling and Moira Norrie. jQMultiTouch: Lightweight Toolkit and Development Framework for Multi-touch/Multi-device Web Interfaces. In Proceedings of the 4th ACM SIGCHI Symposium on Engineering Interactive Computing Systems - EICS 2012 (Jun. 25 - 28, Copenhagen, Denmark). ACM Press, New York, NY, USA, 2012. pp. 61--70.
[35]
David Pinelle, Miguel Nacenta, Carl Gutwin and Tadeusz Stach. The Effects of Co-Present Embodiments on Awareness and Collaboration in Tabletop Groupware. In Proceedings of Graphics Interface 2008 - GI 2008 (May 28--30, Windsor, Ontario, Canada). Canadian Information Processing Society, Mississauga, Ontario, Canada, 2008. pp. 1--8.
[36]
Mark Roseman and Saul Greenberg. Building Real Time Groupware with GroupKit, A Groupware Toolkit. ACM Transactions on Computer-Human Interaction (TOCHI) 3, 1 (March 1996). pp. 66--106.
[37]
Volker Roth, Philipp Schmidt and Benjamin Gueldenring. The IR Ring: Authenticating Users' Touches on a Multi-touch Display. In Proceedings of the 23rd Annual ACM Symposium on User Interface Software and Technology - UIST 2010 (Oct. 3--6, New York, NY, USA). ACM Press, New York, NY, USA, 2010. pp. 259--262.
[38]
Kathy Ryall, A. Esenther, C. Forlines, C. Shen, S. Shipman, Meredith Ringel Morris, K. Everitt and F. D. Vernier. Identity-Differentiating Widgets for Multiuser Interactive Surfaces. IEEE Computer Graphics and Applications 26, 5 (2006). pp. 56--64.
[39]
Dominik Schmidt, Ming Ki Chong and Hans W Gellersen. IdLenses: Dynamic Personal Areas on Shared Surfaces. In Proceedings of the ACM International Conference on Interactive Tabletops and Surfaces - ITS 2010 (Nov. 7--10, Saarbrücken, Germany). ACM Press, New York, NY, USA, 2010. pp. 131--134.
[40]
Stacey D. Scott, M. Sheelagh T Carpendale and Kori M Inkpen. Territoriality in Collaborative Tabletop Workspaces. In Proceedings of the 2004 ACM Conference on Computer-Supported Cooperative Work - CSCW 2004 (Nov. 6--10, Chicago, IL, USA). ACM Press, New York, NY, USA, 2004. pp. 294--303.
[41]
Stacey D. Scott, Karen D. Grant and Regan L. Mandryk. System Guidelines for Co-located, Collaborative Work on a Tabletop Display. In Proceedings of the 8th European Conference on Computer-Supported Cooperative Work - ECSCW 2003 (Sept. 14--18, Helsinki, Finland). Kluwer Academic Publishers, Netherlands, 2003. pp. 159--178.
[42]
Chia Shen, Frédéric D. Vernier, Clifton Forlines and Meredith Ringel. DiamondSpin: An Extensible Toolkit for Around-the-Table Interaction. In Proceedings of the Conference on Human Factors in Computing Systems - CHI 2004 (Apr. 24--29, Vienna, Austria). ACM Press, New York, NY, USA, 2004. pp. 167--174.
[43]
Jason Stewart, Benjamin B. Bederson and Allison Druin. Single Display Groupware: A Model for Co-present Collaboration. In Proceedings of the Conference on Human Factors in Computing Systems - CHI 1999 (May 15--20, Pittsburgh, PA, USA). ACM Press, New York, NY, USA, 1999. pp. 286--293.
[44]
John C. Tang. Findings from Observational Studies of Collaborative Work. International Journal of Man-Machine Studies 34, 2 (1991). pp. 143--160.
[45]
Edward Tse, Jonathan Histon, Stacey D. Scott and Saul Greenberg. Avoiding Interference: How People Use Spatial Separation and Partitioning in SDG Workspace. In Proceedings of the ACM 2004 Conference on Computer-Supported Cooperative Work - CSCW 2004 (Nov. 6--10, Chicago, IL, USA). ACM, New York, NY, USA, 2004. pp. 252--261.
[46]
Malte Weiss, Julie Wagner, Yvonne Jansen, Roger Jennings, Ramsin Khoshabeh, James D. Hollan and Jan Borchers. SLAP Widgets: Bridging the Gap Between Virtual and Physical Controls on Tabletops. In Proceedings of the Conference on Human Factors in Computing Systems - CHI 2006 (Apr. 22--27, Montreal, Canada). ACM Press, New York, NY, USA, 2006. pp. 481--490.
[47]
Daniel Wigdor and Ravin Balakrishnan. Empirical Investigation into the Effect of Orientation on Text Readability in Tabletop Displays. In Proceedings of the Ninth Conference on European Conference on Computer-Supported Cooperative Work - ECSCW 2005 (Paris, France). Springer-Verlag, Berlin/Heidelberg, Germany, 2005. pp. 205--224.
Index Terms
- TUIOFX: A JavaFX Toolkit for Shared Interactive Surfaces
Recommendations
TUIOFX—Toolkit Support for the Development of JavaFX Applications for Interactive Tabletops
Human-Computer Interaction – INTERACT 2015AbstractTUIOFX is a novel toolkit for developing multi-touch, multi-user applications for interactive tabletops and surfaces. By seamlessly integrating with JavaFX, TUIOFX provides a low entry barrier for developing state-of-the-art applications with ...
Multi-user chorded toolkit for multi-touch screens
EICS '11: Proceedings of the 3rd ACM SIGCHI symposium on Engineering interactive computing systemsIn this work, we present the design and implementation of a chorded menu for multiple users on a large multi-touch vertical display. Instead of selecting an item in a fixed menu by reaching for it, users make a selection by touching multiple fingers ...
Clip-on gadgets: expanding multi-touch interaction area with unpowered tactile controls
UIST '11: Proceedings of the 24th annual ACM symposium on User interface software and technologyVirtual keyboards and controls, commonly used on mobile multi-touch devices, occlude content of interest and do not provide tactile feedback. Clip-on Gadgets solve these issues by extending the interaction area of multi-touch devices with physical ...
Comments
Information & Contributors
Information
Published In
Copyright © 2017 ACM.
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 30 June 2017
Published in PACMHCI Volume 1, Issue EICS
Check for updates
Author Tags
Qualifiers
- Research-article
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 254Total Downloads
- Downloads (Last 12 months)4
- Downloads (Last 6 weeks)0
Reflects downloads up to 07 Mar 2025
Other Metrics
Citations
View Options
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in