Lei Zhang
Steve Oney
ABSTRACT
Immersive authoring is a paradigm that makes Virtual Reality (VR) application development easier by allowing programmers to create VR content while immersed in the virtual environment. In this paradigm, programmers manipulate programming primitives through direct manipulation and get immediate feedback on their program's state and output. However, existing immersive authoring tools have a low ceiling; their programming primitives are intuitive but can only express a limited set of static relationships between elements in a scene. In this paper, we introduce FlowMatic, an immersive authoring tool that raises the ceiling of expressiveness by allowing programmers to specify reactive behaviors---behaviors that react to discrete events such as user actions, system timers, or collisions. FlowMatic also introduces primitives for programmatically creating and destroying new objects, for abstracting and re-using functionality, and for importing 3D models. Importantly, FlowMatic uses novel visual representations to allow these primitives to be represented directly in VR. We also describe the results of a user study that illustrates the usability advantages of FlowMatic relative to a 2D authoring tool and we demonstrate its expressiveness through several example applications that would be impossible to implement with existing immersive authoring tools. By combining a visual program representation with expressive programming primitives and a natural User Interface (UI) for authoring programs, FlowMatic shows how programmers can build fully interactive virtual experiences with immersive authoring.
Supplemental Material
Available for Download
srt
ufp2615pvc.srt (593 B)
Preview video captions
zip
ufp2615aux.zip (147 MB)
The zip file contains three example applications that were created using our immersive authoring tool.
- Cycling '74. 2018. Max 8. (2018). https://cycling74.com/products/max/Google Scholar
- A-FRAME. 2015. A-FRAME. (2015). https://aframe.io/Google Scholar
- Rahul Arora, Rubaiat Habib Kazi, Danny M Kaufman, Wilmot Li, and Karan Singh. 2019. MagicalHands: Mid-Air Hand Gestures for Animating in VR. In Proceedings of the 32nd Annual ACM Symposium on User Interface Software and Technology. 463--477.Google ScholarDigital Library
- Engineer Bainomugisha, Andoni Lombide Carreton, Tom van Cutsem, Stijn Mostinckx, and Wolfgang de Meuter. 2013. A survey on reactive programming. ACM Computing Surveys (CSUR) 45, 4 (2013), 52.Google ScholarDigital Library
- Don Brutzman and Leonard Daly. 2010. X3D: extensible 3D graphics for Web authors. Elsevier.Google ScholarDigital Library
- Jeff Butterworth. 1992. 3DM: a three-dimensional modeler using a head-mounted display. (1992).Google Scholar
- Ricardo Cabello. 2010. three.js. (2010). https://threejs.org/Google Scholar
- Rikk Carey and Gavin Bell. 1997. The annotated VRML 2.0 reference manual. Number BOOK. Addison-Wesley.Google Scholar
- Evan Czaplicki and Stephen N Chong. 2013. Asynchronous functional reactive programming for GUIs. In Proceedings of the 34th ACM SIGPLAN Conference on Programming Language Design and Implementation-PLDI'13. ACM Press.Google ScholarDigital Library
- Conal Elliott and Paul Hudak. 1997. Functional reactive animation. In ACM SIGPLAN Notices, Vol. 32. ACM, 263--273.Google ScholarDigital Library
- Carmine Elvezio, Mengu Sukan, and Steven Feiner. 2018. Mercury: A messaging framework for modular ui components. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. ACM, 588.Google ScholarDigital Library
- Barrett Ens, Fraser Anderson, Tovi Grossman, Michelle Annett, Pourang Irani, and George Fitzmaurice. 2017. Ivy: Exploring spatially situated visual programming for authoring and understanding intelligent environments. In Proceedings of the 43rd Graphics Interface Conference. Canadian Human-Computer Communications Society, 156--162.Google ScholarDigital Library
- Google. 2016. Google Tilt Brush. (2016). https://www.tiltbrush.com/Google Scholar
- Daniel D Hils. 1991. Datavis: a visual programming language for scientific visualization. In Proceedings of the 19th annual conference on Computer Science. ACM, 439--448.Google ScholarDigital Library
- National Instruments. 2017. LabVIEW. (2017). http://www.ni.com/labview/Google Scholar
- Bret Jackson and Daniel F Keefe. 2016. Lift-off: Using reference imagery and freehand sketching to create 3d models in vr. IEEE transactions on visualization and computer graphics 22, 4 (2016), 1442--1451.Google Scholar
- Annie Kelly, R Benjamin Shapiro, Jonathan de Halleux, and Thomas Ball. 2018. ARcadia: A rapid prototyping platform for real-time tangible interfaces. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. ACM, 409.Google ScholarDigital Library
- Takayuki Dan Kimura, Julie W Choi, and Jane M Mack. 1986. A visual language for keyboardless programming. (1986).Google Scholar
- David Lau-Kee, Adam Billyard, Robin Faichney, Yasuo Kozato, Paul Otto, Mark Smith, and Ian Wilkinson. 1991. VPL: an active, declarative visual programming system. In Proceedings 1991 IEEE Workshop on Visual Languages. IEEE, 40--46.Google ScholarCross Ref
- David Ledo, Steven Houben, Jo Vermeulen, Nicolai Marquardt, Lora Oehlberg, and Saul Greenberg. 2018. Evaluation strategies for HCI toolkit research. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. 1--17.Google ScholarDigital Library
- Gun A Lee and Gerard J Kim. 2009. Immersive authoring of Tangible Augmented Reality content: A user study. Journal of Visual Languages & Computing 20, 2 (2009), 61--79.Google ScholarDigital Library
- Gun A Lee, Gerard J Kim, and Mark Billinghurst. 2005. Immersive authoring: What you experience is what you get (wyxiwyg). Commun. ACM 48, 7 (2005), 76--81.Google ScholarDigital Library
- Gun A Lee, Claudia Nelles, Mark Billinghurst, and Gerard Jounghyun Kim. 2004. Immersive authoring of tangible augmented reality applications. In Proceedings of the 3rd IEEE/ACM international Symposium on Mixed and Augmented Reality. IEEE Computer Society, 172--181.Google ScholarDigital Library
- Microsoft. 2018. Microsoft Maquette. (2018). https://www.maquette.ms/Google Scholar
- Mark Mine. 1995. ISAAC: A virtual environment tool for the interactive construction of virtual worlds. UNC Chapel Hill Computer Science Technical Report TR95-020 (1995).Google ScholarDigital Library
- Mark Mine, Arun Yoganandan, and Dane Coffey. 2014. Making VR work: building a real-world immersive modeling application in the virtual world. In Proceedings of the 2nd ACM symposium on Spatial user interaction. ACM, 80--89.Google ScholarDigital Library
- Mozilla. 2018. Mozilla Hubs. (2018). https://hubs.mozilla.com/Google Scholar
- Marc A Najork and Eric Golin. 1990. Enhancing Show-and-Tell with a polymorphic type system and higher-order functions. In Proceedings of the 1990 IEEE Workshop on Visual Languages. IEEE, 215--220.Google ScholarCross Ref
- Michael Nebeling and Katy Madier. 2019. 360proto: Making Interactive Virtual Reality & Augmented Reality Prototypes from Paper. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems. ACM, 596.Google ScholarDigital Library
- Michael Nebeling, Janet Nebeling, Ao Yu, and Rob Rumble. 2018. Protoar: Rapid physical-digital prototyping of mobile augmented reality applications. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. ACM, 353.Google ScholarDigital Library
- Oculus. 2016. Oculus Medium. (2016). https://www.oculus.com/medium/Google Scholar
- Randy Pausch, Tommy Burnette, AC Capeheart, Matthew Conway, Dennis Cosgrove, Rob DeLine, Jim Durbin, Rich Gossweiler, Shuichi Koga, and Jeff White. 1995. Alice: Rapid prototyping system for virtual reality. IEEE Computer Graphics and Applications 15, 3 (1995), 8--11.Google ScholarDigital Library
- Thammathip Piumsomboon, Gun A Lee, Jonathon D Hart, Barrett Ens, Robert W Lindeman, Bruce H Thomas, and Mark Billinghurst. 2018. Mini-me: An adaptive avatar for mixed reality remote collaboration. In Proceedings of the 2018 CHI conference on human factors in computing systems. 1--13.Google ScholarDigital Library
- Kevin Ponto, Ross Tredinnick, Aaron Bartholomew, Carrie Roy, Dan Szafir, Daniel Greenheck, and Joe Kohlmann. 2013. SculptUp: A rapid, immersive 3D modeling environment. In 2013 IEEE Symposium on 3D User Interfaces (3DUI). IEEE, 199--200.Google ScholarCross Ref
- RxJS. 2015. RxJS. (2015). https://rxjs-dev.firebaseapp.com/Google Scholar
- Nazmus Saquib, Rubaiat Habib Kazi, Li-Yi Wei, and Wilmot Li. 2019. Interactive Body-Driven Graphics for Augmented Video Performance. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems. ACM, 622.Google ScholarDigital Library
- Arvind Satyanarayan, Ryan Russell, Jane Hoffswell, and Jeffrey Heer. 2015. Reactive vega: A streaming dataflow architecture for declarative interactive visualization. IEEE transactions on visualization and computer graphics 22, 1 (2015), 659--668.Google Scholar
- Udo Schultheis, Jason Jerald, Fernando Toledo, Arun Yoganandan, and Paul Mlyniec. 2012. Comparison of a two-handed interface to a wand interface and a mouse interface for fundamental 3D tasks. In 2012 IEEE Symposium on 3D User Interfaces (3DUI). IEEE, 117--124.Google ScholarCross Ref
- Tiago Boldt Sousa. 2012. Dataflow programming concept, languages and applications. In Doctoral Symposium on Informatics Engineering, Vol. 130.Google Scholar
- Anthony Steed and Mel Slater. 1996. A dataflow representation for defining behaviours within virtual environments. In Proceedings of the IEEE 1996 Virtual Reality Annual International Symposium. IEEE, 163--167.Google ScholarCross Ref
- William Robert Sutherland. 1966. The on-line graphical specification of computer procedures. Ph.D. Dissertation. Massachusetts Institute of Technology.Google Scholar
- Steven L Tanimoto. 2013. A perspective on the evolution of live programming. In Proceedings of the 1st International Workshop on Live Programming. IEEE Press, 31--34.Google ScholarDigital Library
- Balasaravanan Thoravi Kumaravel, Fraser Anderson, George Fitzmaurice, Bjoern Hartmann, and Tovi Grossman. 2019. Loki: Facilitating Remote Instruction of Physical Tasks Using Bi-Directional Mixed-Reality Telepresence. In Proceedings of the 32nd Annual ACM Symposium on User Interface Software and Technology. 161--174.Google ScholarDigital Library
- Unity. 2005. Unity. (2005). https://unity.com/Google Scholar
- Unreal. 2020a. Unreal Blueprints. (2020). https://docs.unrealengine.com/Engine/Blueprints/Google Scholar
- Unreal. 2020b. Unreal Engine 5. (2020). https://www.unrealengine.com/Google Scholar
- Valve. 2020. Half-Life: Alyx. (2020). https://www.half-life.com/en/alyx/Google Scholar
- The Verge. 2018. Microsoft is bringing the SharePoint work environment to virtual reality headsets. (2018). https://www.theverge.com/2018/5/21/17376422/microsoft-sharepoint-spaces-mixed-reality-virtual-reality-featuresGoogle Scholar
- Neos VR. 2018. LogiX. (2018). https://neosvr.com/Google Scholar
- Zhanyong Wan, Walid Taha, and Paul Hudak. 2002. Event-driven FRP. In International Symposium on Practical Aspects of Declarative Languages. Springer, 155--172.Google ScholarDigital Library
- Baltasar Trancón y Widemann and Markus Lepper. 2014. Foundations of Total Functional Data-Flow Programming.. In MSFP. 143--167.Google Scholar
- Lei Zhang and Steve Oney. 2019. Studying the Benefits and Challenges of Immersive Dataflow Programming. In 2019 IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC). IEEE, 39--47.Google Scholar
Index Terms
- FlowMatic: An Immersive Authoring Tool for Creating Interactive Scenes in Virtual Reality
Recommendations
XRDirector: A Role-Based Collaborative Immersive Authoring System
CHI '20: Proceedings of the 2020 CHI Conference on Human Factors in Computing SystemsImmersive authoring is an increasingly popular technique to design AR/VR scenes because design and testing can be done concurrently. Most existing systems, however, are single-user and limited to either AR or VR, thus constrained in the interaction ...
Immersive authoring of Tangible Augmented Reality content: A user study
Immersive authoring refers to the style of programming or developing content from within the targetexecutable environment. Immersive authoring is important for fields such as augmented reality (AR) in which interaction usability and user perception of ...
Highlighting Techniques for 360° Video Virtual Reality and Their Immersive Authoring
Advances in Visual ComputingAbstractHighlighting important elements is a fundamental task to guide the user’s attention in Virtual Reality (VR) applications. Besides the authoring process of a VR application, the creation of these cues for highlighting in 360° video VR already is a ...
Comments