[IN]Musicality: A Collection of VR Drawings and Music as an Artistic Application of Hybrid Immersive Models

Abstract

A recent work called I’m Watching You/Me (IMWYM) introduced a technical apparatus for live VR visualization of 360 drawings made on-the-fly. The exhibition [IN]Musicality uses and improves on the techniques presented in this first prototype but focusing on their artistic application. In fact, [IN]Musicality explores the artistic side of Hybrid Immersive Models through the interaction with spherical perspectives drawn to live music.

The article enumerates the issues from the first version of the artefact, presents possible ways to solve them, describes the solutions adopted for [IN]Musicality, and establishes milestones for the future roadmap of developments.

Appendix A – Table of Issues and Solutions

1.1 A.1 Narrative/Development

Issue	Criticality and draft solution
1 - Missing concept. IMWYM 1 focused on a technical advancement. There was neither a clear artistic concept nor a defined narrative. As the goals of the research changed, the new edition focuses on the artistic application. Consequently, both concept and narrative were critical improvements for integrating IMWYM with the new goals.	Critical issues. IMWYM 2 would not have any sense without the definition of a concept and a narrative. Solution. Elaboration of a concept and a narrative for redirecting and integrating the artefact to/with artistic purposes, and for highlighting the particularities of its components. For example, the artefact has a strong focus on the interaction with 360 content, which might open a wide range of possibilities for exploring VR content using gamification.
2 - Missing narrative. IMWYM 1 was included within the shared exhibition “Contingency” [24], yet neither limited nor oriented towards any specific direction by it. This lack of narrative was both with the general exhibition, and within the prototype.

1.2 A.2 Interaction/Evolution

Issue	Criticality and draft solution
3 – Lack of a front-end interface. An interface was critically necessary for a more efficient and simplified operation. For example, one characteristic of the artefact was the possibility of switching or adding inputs, and then interacting with them. Yet, visitors would not know how to do it since the available screen was showing pure visual code, something not inviting to interact with.	Critical issue. The interface is the connection between the content and the interaction. As so, it is very important for a simplified operation and for gathering visitors’ feedback. Then, the data is the base for understating the phenomenological reading of the interaction with the artefact. Furthermore, the artist’s guidance is required without the interface (issue 5). Solution. Elaboration of a front-end interface that enables a fluent interaction with the content, gather data from it, and show a menu for visitors to get help and read information.
4 - Dependence on the artist’s presence. Due to issue 3 and missing additional graphic material (flyers, posters), the interaction was very hard to understand, which required the authors’ presence for operating the artefact and explaining its functioning.	Critical issue. The artefact should stand by itself, with a minimum of explanations. Solution. Addition of supplementary material explaining the artefact distributed either digitally with an external device and/or with printed material
5 - Fixed FOV. Currently, the field of view can be set in Touchdesigner but remains constant at runtime. It might be useful to add a function for setting it dynamically so as to zoom in and out while looking at details.	Low priority. This issue is not connected with the main functioning but with an advanced fruition and visualisation feature. Solution. Add a button/function on the interface to change the FOV dynamically.

1.3 A.3 Experience/Fruition

Issue	Criticality and draft solution
6 – Software incompatibility. The first problem arose migrating from Mac to Windows. Using Touchdesigner on both OS, one might have expected a better integration. Yet, I needed to rework some key components. The second incompatibility was with the external camera (IN-5) since the one provided was dated.	Medium priority. Compatibility issues are a big problem if we are using a local computer. Solution. Have access to the local facilities with more anticipation to test extensively the artefact. Otherwise, it can be solved using the original laptop with which the artefact was built. In any case, it would be more opportune to have a web-based solution to avoid any OS’ limitations.
7 - Navigation glitch. Compass and Gravity sensors determined the camera’s position. A glitch occurs during the navigation: there is a jump of the view under certain rotations.	Low priority. The navigation is less trustable, but still functioning. Solution. Use of Quaternion-based calculations instead of Euler angles.
8 - Zig Sim app. The app needed to be maximized and onscreen to send the OSC data. Plus, the wireless connectivity was also an issue due to OS’ restrictions. Consequently, the artefact used a wired connection both to keep the phone’s battery loaded, and to simplify the net connection.	Medium-high priority. The problem avoids a smoother interaction with the content. Solution. A mobile application able to send OSC data running on the background. The creation of a dedicated application might be useful for other purposes (e.g., using the phone within a VR headset).
9 – Limited inputs’ projection. The presented version was limited to the equirectangular projection.	Medium priority. It is important to integrate better the theory of the research. Solution. Add cubical and azimuthal-equidistant perspectives as inputs.
10 - Calibration of the camera (IN-5). The calibration of the camera was very fragile, which forced a recalibration very often.	Medium-high priority. The live capturing should not rely on a physical calibration, but on a dynamic calibration based on the analysis of the image. Solution. Use of markers on the paper or board and live image-processing tools.
11 - IN-2 and OU-2 were not live tested. Due to spatial and equipment constraints in the room.	Low-medium priority. The functioning of both components was verified during the programming stage. However, IN-2 might result difficult to whom is not familiarised with the device and spherical drawing. Hence, much likely IN-2 might be only on the interest of dedicated artists or teachers. OU-2 is needed for separating IN-3 and the interface. Solution. Larger space and equipment availability.
12 - Setup IN-1 (physical drawing)/AR-1 (artist)/IN-5 (camera). This configuration generated a rather obvious problem: being the physical drawing hanged on the wall, the artist casted shadows while drawing and the visual captured by the camera was partially blocked.	Low priority. The problem is more aesthetical since does not block the functioning nor avoids the normal interaction with the artefact. Solution. Consider a setup for drawing onto a glass (or something transparent), and a camera capturing the drawing from the rear side.
13 - Setup IN-1 (physical drawing)/OU-1. During the first presentation, IN-1 and OU-1 were almost parallel and within the same line. Therefore, it was very complicated for the artist to watch the VR results and having instant visual feedback.	Low priority. The issue is only for the artist using the artefact and not for regular visitors. Solution. Build a flexible setup considering the installation with and without the live drawing performance and pay attention to the perspective that both artist and visitors have.