skip to main content
10.1145/3322276.3322329acmconferencesArticle/Chapter ViewAbstractPublication PagesdisConference Proceedingsconference-collections
research-article

Astral: Prototyping Mobile and Smart Object Interactive Behaviours Using Familiar Applications

Published: 18 June 2019 Publication History

Abstract

Astral is a prototyping tool for authoring mobile and smart object interactive behaviours. It mirrors selected display contents of desktop applications onto mobile devices (smartphones and smartwatches), and streams/remaps mobile sensor data to desktop input events (mouse or keyboard) to manipulate selected desktop contents. This allows designers to use familiar desktop applications (e.g. PowerPoint, AfterEffects) to prototype rich interactive behaviours. Astral combines and integrates display mirroring, sensor streaming and input remapping, where designers can exploit familiar desktop applications to prototype, explore and fine-tune dynamic interactive behaviours. With Astral, designers can visually author rules to test real-time behaviours while interactions take place, as well as after the interaction has occurred. We demonstrate Astral's applicability, workflow and expressiveness within the interaction design process through both new examples and replication of prior approaches that illustrate how various familiar desktop applications are leveraged and repurposed.

Supplementary Material

MP4 File (disfp1279.mp4)
Supplemental video

References

[1]
Daniel Ashbrook and Thad Starner. 2010. MAGIC: a motion gesture design tool. In Proceedings of the SIGCHI Conference on Human Factors in Compu-ting Systems (CHI '10). ACM, New York, NY, USA, 2159--2168.
[2]
Olivier Bau and Wendy E. Mackay. 2008. OctoPo-cus: a dynamic guide for learning gesture-based command sets. In Proceedings of ACM Symposium on User Interface Software and Technology (UIST '08). ACM, New York, NY, USA, 37--46.
[3]
Dominikus Baur, Sebastian Boring, and Steven Feiner. 2012. Virtual projection: exploring optical projection as a metaphor for multi-device interaction. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '12). ACM, New York, NY, USA, 1693--1702.
[4]
Beginner's Mind Collective and David Shaw. 2012. Makey Makey: improvising tangible and nature-based user interfaces. In Proceedings of the ACM In-ternational Conference on Tangible, Embedded and Embodied Interaction (TEI '12). ACM, New York, NY, USA, 367--370.
[5]
Tracey Booth, Simone Stumpf, Jon Bird, and Sara Jones. 2016. Crossed Wires: Investigating the Prob-lems of End-User Developers in a Physical Compu-ting Task. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '16). ACM, New York, NY, USA, 3485--3497.
[6]
John Brosz, Miguel A. Nacenta, Richard Pusch, Sheelagh Carpendale, and Christophe Hurter. 2013. Transmogrification: causal manipulation of visuali-zations. In Proceedings of the ACM Symposium on User Interface Software and Technology (UIST '13). ACM, New York, NY, USA, 97--106.
[7]
Jesse Burstyn, Juan Pablo Carrascal, and Roel Verte-gaal. 2016. Fitts' Law and the Effects of Input Map-ping and Stiffness on Flexible Display Interactions. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '16). ACM, New York, NY, USA, 3649--3658.
[8]
Xiang 'Anthony' Chen and Yang Li. 2017. Improv: An Input Framework for Improvising Cross-Device Interaction by Demonstration. ACM Trans. Comput.-Hum. Interact. 24, 2, Article 15 (April 2017), 21 pages.
[9]
Morgan Dixon, Alexander Nied, and James Fogarty. 2014. Prefab layers and prefab annotations: extensi-ble pixel-based interpretation of graphical interfaces. In Proceedings of the ACM Symposium on User In-terface Software and Technology (UIST '14). ACM, New York, NY, USA, 221--230.
[10]
Pierre Dragicevic and Jean-Daniel Fekete. 2004. Support for input adaptability in the ICON toolkit. In Proceedings of the International Conference on Multimodal Interfaces (ICMI '04). ACM, New York, NY, USA, 212--219.
[11]
Pierre Dragicevic, Gonzalo Ramos, Jacobo Bib-liowitcz, Derek Nowrouzezahrai, Ravin Balakrish-nan, and Karan Singh. 2008. Video browsing by di-rect manipulation. In Proceedings of the SIGCHI Conference on Human Factors in Computing Sys-tems (CHI '08). ACM, New York, NY, USA, 237--246.
[12]
George W. Fitzmaurice. 1993. Situated information spaces and spatially aware palmtop comput-ers. Commun. ACM 36, 7 (July 1993), 39--49.
[13]
Saul Greenberg. 2007. Toolkits and interface creativ-ity. Multimedia Tools and Applications, 32(2), Springer, 139--159.
[14]
Saul Greenberg and Bill Buxton. 2008. Usability evaluation considered harmful (some of the time). In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '08). ACM, New York, NY, USA, 111--120.
[15]
Saul Greenberg and Chester Fitchett. 2001. Phidgets: easy development of physical interfaces through physical widgets. In Proceedings of the ACM Sym-posium on User Interface Software and Technolo-gy (UIST '01). ACM, New York, NY, USA, 209--218.
[16]
Saul Greenberg, Sheelagh Carpendale, Nicolai Mar-quardt and Bill Buxton (2011). Sketching User Expe-riences: The Workbook. Elsevier.
[17]
Tovi Grossman and Ravin Balakrishnan. 2005. The bubble cursor: enhancing target acquisition by dy-namic resizing of the cursor's activation area. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '05). ACM, New York, NY, USA, 281--290.
[18]
Rubaiat Habib Kazi, Fanny Chevalier, Tovi Gross-man, and George Fitzmaurice. 2014. Kitty: sketching dynamic and interactive illustrations. In Proceedings of the ACM Symposium on User Interface Software and Technology (UIST '14). ACM, New York, NY, USA, 395--405.
[19]
Rubaiat Habib Kazi, Tovi Grossman, Nobuyuki Umetani, and George Fitzmaurice. 2016. Motion Amplifiers: Sketching Dynamic Illustrations Using the Principles of 2D Animation. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems (CHI '16). ACM, New York, NY, USA, 4599--4609.
[20]
Christopher Michael Hancock, 2003. Real-time pro-gramming and the big ideas of computational litera-cy. PhD Dissertation, Massachusetts Institute of Technology, Cambridge, MA.
[21]
Björn Hartmann, 2009. Gaining Design Insight through Interaction Prototyping Tools. PhD Thesis, Stanford University.
[22]
Björn Hartmann, Leith Abdulla, Manas Mittal, and Scott R. Klemmer. 2007. Authoring sensor-based in-teractions by demonstration with direct manipulation and pattern recognition. In Proceedings of the SIGCHI Conference on Human Factors in Compu-ting Systems (CHI '07). ACM, New York, NY, USA, 145--154.
[23]
Björn Hartmann, Scott R. Klemmer, Michael Bern-stein, Leith Abdulla, Brandon Burr, Avi Robinson-Mosher, and Jennifer Gee. 2006. Reflective physical prototyping through integrated design, test, and analysis. In Proceedings of the ACM Symposium on User Interface Software and Technology (UIST '06). ACM, New York, NY, USA, 299--308.
[24]
Ken Hinckley and Hyunyoung Song. 2011. Sensor synaesthesia: touch in motion, and motion in touch. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '11). ACM, New York, NY, USA, 801--810.
[25]
Lars Erik Holmquist. 2005. Prototyping: generating ideas or cargo cult designs?. interactions 12, 2 (March 2005), 48--54.
[26]
Kasper Hornbæk and Antti Oulasvirta. 2017. What Is Interaction?. In Proceedings of the SIGCHI Confer-ence on Human Factors in Computing Systems (CHI '17). ACM, New York, NY, USA, 5040--5052.
[27]
Stephanie Houde, and Charles Hill. 1997. "What do prototypes prototype?" Handbook of Human-Computer Interaction (Second Edition), 367--381.
[28]
Steven Houben and Nicolai Marquardt. 2015. WatchConnect: A Toolkit for Prototyping Smart-watch-Centric Cross-Device Applications. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '15). ACM, New York, NY, USA, 1247--1256.
[29]
J. F. Kelley. 1984. An iterative design methodology for user-friendly natural language office information applications. ACM Transactions on Information Sys-tems. 2, 1 (January 1984), 26--41.
[30]
Ju-Whan Kim and Tek-Jin Nam. 2013. EventHurdle: supporting designers' exploratory interaction proto-typing with gesture-based sensors. In Proceedings of the SIGCHI Conference on Human Factors in Com-puting Systems (CHI '13). ACM, New York, NY, USA, 267--276.
[31]
James Lin and James A. Landay. 2008. Employing patterns and layers for early-stage design and proto-typing of cross-device user interfaces. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '08). ACM, New York, NY, USA, 1313--1322.
[32]
David Ledo, Fraser Anderson, Ryan Schmidt, Lora Oehlberg, Saul Greenberg, and Tovi Grossman. 2017. Pineal: Bringing Passive Objects to Life with Embedded Mobile Devices. In Proceedings of the SIGCHI Conference on Human Factors in Compu-ting Systems (CHI '17). ACM, New York, NY, USA, 2583--2593.
[33]
David Ledo, Saul Greenberg, Nicolai Marquardt, and Sebastian Boring. 2015. Proxemic-Aware Controls: Designing Remote Controls for Ubiquitous Compu-ting Ecologies. In Proceedings of the 17th Interna-tional Conference on Human-Computer Interaction with Mobile Devices and Services (MobileHCI '15). ACM, New York, NY, USA, 187--198.
[34]
David Ledo, Steven Houben, Jo Vermeulen, Nicolai Marquardt, Lora Oehlberg and Saul Greenberg. 2018. Evaluation Strategies for HCI Toolkit Research. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '18). ACM, New York, NY, USA.
[35]
Germán Leiva and Michel Beaudouin-Lafon. 2018. Montage: A Video Prototyping System to Reduce Re-Shooting and Increase Re-Usability. In Proceedings of the 31st Annual ACM Symposium on User Interface Software and Technology (UIST '18). ACM, New York, NY, USA, 675--682.
[36]
Toby Jia-Jun Li, Amos Azaria, and Brad A. Myers. 2017. SUGILITE: Creating Multimodal Smartphone Automation by Demonstration. In Proceedings of the 2017 CHI Conference on Human Factors in Compu-ting Systems (CHI '17). ACM, New York, NY, USA, 6038--6049.
[37]
John H. Maloney and Randall B. Smith. 1995. Di-rectness and liveness in the morphic user interface construction environment. In Proceedings of the ACM Symposium on User Interface and Software Technology (UIST '95). ACM, New York, NY, USA, 21--28.
[38]
Nicolai Marquardt, Till Ballendat, Sebastian Boring, Saul Greenberg, and Ken Hinckley. 2012. Gradual engagement: facilitating information exchange be-tween digital devices as a function of proximity. In Proceedings of the ACM International Conference on Interactive Tabletops and Surfaces (ITS '12). ACM, New York, NY, USA, 31--40.
[39]
Nolwenn Maudet, Germán Leiva, Michel Beaudouin-Lafon, and Wendy Mackay. 2017. Design Break-downs: Designer-Developer Gaps in Representing and Interpreting Interactive Systems. In Proceedings of the ACM Conference on Computer Supported Co-operative Work and Social Computing (CSCW '17). ACM, New York, NY, USA, 630--641.
[40]
Andrew McCaleb Reach, and Chris North. 2017. "The Signals and Systems Approach to Animation." arXiv preprint arXiv:1703.00521 (2017).
[41]
Joseph McGrath. 1995. "Methodology Matters: Do-ing Research in the Behavioral and Social Sciences". Readings in Human--Computer Interaction. 152--169.
[42]
Jan Meskens, Kris Luyten, and Karin Coninx. 2009. Shortening user interface design iterations through realtime visualisation of design actions on the target device. In IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC 2009), pp. 132--135.
[43]
Jan Meskens, Kris Luyten, and Karin Coninx. 2010. D-Macs: building multi-device user interfaces by demonstrating, sharing and replaying design actions. In Proceedings of the ACM Symposium on User In-terface Software and Technology (UIST '10). ACM, New York, NY, USA, 129--138.
[44]
Jan Meskens, Jo Vermeulen, Kris Luyten, and Karin Coninx. 2008. Gummy for multi-platform user inter-face designs: shape me, multiply me, fix me, use me. In Proceedings of the Working Conference on Ad-vanced Visual Interfaces (AVI '08). ACM, New York, NY, USA, 233--240.
[45]
Brad A. Myers. 1998. Scripting graphical applica-tions by demonstration. In Proceedings of the SIGCHI Conference on Human Factors in Compu-ting Systems (CHI '98). ACM Press/Addison-Wesley Publishing Co., New York, NY, USA, 534--541.
[46]
Brad Myers. 2002. Mobile devices for control. In International Conference on Mobile Human-Computer Interaction, Springer, Berlin, Heidelberg 1--8.
[47]
Brad Myers, Choon Hong Peck, Jeffrey Nichols, Dave Kong, and Robert Miller. 2001. Interacting at a distance using semantic snarfing. In International Conference on Ubiquitous Computing. Springer, Berlin, Heidelberg pp. 305--314.
[48]
Brad Myers, Scott E. Hudson, and Randy Pausch. 2000. Past, present, and future of user interface soft-ware tools. ACM Transactions on Computer-Human Interaction. 7, 1 (March 2000), 3--28.
[49]
Brad Myers, Sun Young Park, Yoko Nakano, Greg Mueller, and Andrew Ko. 2008. How designers de-sign and program interactive behaviors. 2008. In Proc. Visual Languages and Human-Centric Compu-ting, 2008. VL/HCC 2008. IEEE Symposium on, pp. 177--184.
[50]
Dan R. Olsen, Jr. 2007. Evaluating user interface systems research. In Proceedings of the 20th annual ACM symposium on User interface software and technology (UIST '07). ACM, New York, NY, USA, 251--258.
[51]
Robert Penner. Robert Penner's Programming Mac-romedia Flash MX. McGraw-Hill, Inc., 2002.
[52]
Tristan Richardson, John Levine. 2011 The Remote Framebuffer Protocol (No. RFC 6143). Technical Report. https://tools.ietf.org/html/rfc6143
[53]
Marco de Sá, Luís Carriço, Luís Duarte, and Tiago Reis. 2008. A mixed-fidelity prototyping tool for mobile devices. In Proceedings of the Working Con-ference on Advanced Visual Interfaces (AVI '08). ACM, New York, NY, USA, 225--232.
[54]
Ramik Sadana and Yang Li. 2016. Gesture morpher: video-based retargeting of multi-touch interactions. In Proceedings of the 18th International Conference on Human-Computer Interaction with Mobile Devic-es and Services (MobileHCI '16). ACM, New York, NY, USA, 227--232.
[55]
Dan Saffer. 2013. Microinteractions: designing with details. O'Reilly Media, Inc.
[56]
Valkyrie Savage, Colin Chang, and Björn Hartmann. 2013. Sauron: embedded single-camera sensing of printed physical user interfaces. In Proceedings of the ACM Symposium on User Interface Software and Technology (UIST '13). ACM, New York, NY, USA, 447--456.
[57]
P. Frazer Seymour, Justin Matejka, Geoff Foulds, Ihor Petelycky, and Fraser Anderson. 2017. AMI: An Adaptable Music Interface to Support the Varying Needs of People with Dementia. In Proceedings of the 19th International ACM SIGACCESS Conference on Computers and Accessibility (ASSETS '17). ACM, New York, NY, USA, 150--154.
[58]
Wolfgang Stuerzlinger, Olivier Chapuis, Dusty Phil-lips, and Nicolas Roussel. 2006. User interface fa-çades: towards fully adaptable user interfaces. In Proceedings of the ACM Symposium on User In-terface Software and Technology (UIST '06). ACM, New York, NY, USA, 309--318.
[59]
Desney S. Tan, Brian Meyers, and Mary Czerwinski. 2004. WinCuts: manipulating arbitrary window re-gions for more effective use of screen space. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM, New York, NY, USA, 1525--1528.
[60]
Frank Thomas, Ollie Johnston, and Disney Anima-tion. 1981. The Illusion of Life. Abbeville Press, New York, NY, USA.
[61]
Nicolas Villar, James Scott, Steve Hodges, Kerry Hammil, and Colin Miller. 2012 .NET Gadgeteer: A Platform for Custom Devices. In Pervasive 2012.Lecture Notes in Computer Science, vol 7319. Springer, Berlin, Heidelberg. 216--233
[62]
Jo Vermeulen, Kris Luyten, Karin Coninx, and Nico-lai Marquardt. 2014. The design of slow-motion feedback. In Proceedings of the SIGCHI Conference on Designing Interactive Systems (DIS '14). ACM, New York, NY, USA, 267--270.
[63]
The Tools Designers are Using Today (2015 Survey) http://tools.subtraction.com/ -- Accessed April 01, 2018.
[64]
The 2017 Annual Design Tools Survey https://uxtools.co/survey-2017 -- Accessed August 27, 2018.
[65]
Adobe Animate (formerly Adobe Flash) https://www.adobe.com/ca/products/animate.html -- Accessed September 20, 2018.
[66]
Arduino http://arduino.cc -- Accessed April 01, 2018.
[67]
Microsoft MakeCode https://makecode.com/ -- Ac-cessed September 20, 2018.
[68]
Nintendo Labo - https://labo.nintendo.com/ - Ac-cessed September 20, 2018.
[69]
TeamViewer http://www.teamviewer.com -- accessed January 11, 2019

Cited By

View all
  • (2024)Enhancing Mobile Interaction: Practical Insights from Smartphone and Smartwatch IntegrationAdjunct Proceedings of the 26th International Conference on Mobile Human-Computer Interaction10.1145/3640471.3680451(1-9)Online publication date: 21-Sep-2024
  • (2024)GlucoMaker: Enabling Collaborative Customization of Glucose MonitorsProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642435(1-21)Online publication date: 11-May-2024
  • (2024)ConnectVR: A Trigger-Action Interface for Creating Agent-based Interactive VR Stories2024 IEEE Conference Virtual Reality and 3D User Interfaces (VR)10.1109/VR58804.2024.00051(286-297)Online publication date: 16-Mar-2024
  • Show More Cited By

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences
DIS '19: Proceedings of the 2019 on Designing Interactive Systems Conference
June 2019
1628 pages
ISBN:9781450358507
DOI:10.1145/3322276
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 18 June 2019

Permissions

Request permissions for this article.

Check for updates

Author Tags

  1. design tool
  2. interactive behaviour
  3. mobile interfaces
  4. prototyping
  5. smart objects

Qualifiers

  • Research-article

Conference

DIS '19
Sponsor:
DIS '19: Designing Interactive Systems Conference 2019
June 23 - 28, 2019
CA, San Diego, USA

Acceptance Rates

DIS '19 Paper Acceptance Rate 105 of 415 submissions, 25%;
Overall Acceptance Rate 1,158 of 4,684 submissions, 25%

Upcoming Conference

DIS '25
Designing Interactive Systems Conference
July 5 - 9, 2025
Funchal , Portugal

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)18
  • Downloads (Last 6 weeks)2
Reflects downloads up to 17 Feb 2025

Other Metrics

Citations

Cited By

View all
  • (2024)Enhancing Mobile Interaction: Practical Insights from Smartphone and Smartwatch IntegrationAdjunct Proceedings of the 26th International Conference on Mobile Human-Computer Interaction10.1145/3640471.3680451(1-9)Online publication date: 21-Sep-2024
  • (2024)GlucoMaker: Enabling Collaborative Customization of Glucose MonitorsProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642435(1-21)Online publication date: 11-May-2024
  • (2024)ConnectVR: A Trigger-Action Interface for Creating Agent-based Interactive VR Stories2024 IEEE Conference Virtual Reality and 3D User Interfaces (VR)10.1109/VR58804.2024.00051(286-297)Online publication date: 16-Mar-2024
  • (2023)GestureCanvas: A Programming by Demonstration System for Prototyping Compound Freehand Interaction in VRProceedings of the 36th Annual ACM Symposium on User Interface Software and Technology10.1145/3586183.3606736(1-17)Online publication date: 29-Oct-2023
  • (2023)Escapement: A Tool for Interactive Prototyping with Video via Sensor-Mediated Abstraction of TimeProceedings of the 2023 CHI Conference on Human Factors in Computing Systems10.1145/3544548.3581115(1-14)Online publication date: 19-Apr-2023
  • (2023)ProObjAR: Prototyping Spatially-aware Interactions of Smart Objects with AR-HMDProceedings of the 2023 CHI Conference on Human Factors in Computing Systems10.1145/3544548.3580750(1-15)Online publication date: 19-Apr-2023
  • (2023)Ubi Edge: Authoring Edge-Based Opportunistic Tangible User Interfaces in Augmented RealityProceedings of the 2023 CHI Conference on Human Factors in Computing Systems10.1145/3544548.3580704(1-14)Online publication date: 19-Apr-2023
  • (2022)SensorViz: Visualizing Sensor Data Across Different Stages of Prototyping Interactive ObjectsProceedings of the 2022 ACM Designing Interactive Systems Conference10.1145/3532106.3533481(987-1001)Online publication date: 13-Jun-2022
  • (2022)Investigating Cross-Modal Approaches for Evaluating Error Acceptability of a Recognition-Based Input TechniqueProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/35172626:1(1-24)Online publication date: 29-Mar-2022
  • (2022)ProGesAR: Mobile AR Prototyping for Proxemic and Gestural Interactions with Real-world IoT Enhanced SpacesProceedings of the 2022 CHI Conference on Human Factors in Computing Systems10.1145/3491102.3517689(1-14)Online publication date: 29-Apr-2022
  • Show More Cited By

View Options

Login options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Figures

Tables

Media

Share

Share

Share this Publication link

Share on social media