Desiderata for a Performative Hybrid Immersive Drawing Platform

1.1 The State of Art: Drawing a Spherical Perspective

Spherical perspectives are an emerging art media. There have been strong advancements towards immersive perspectives in the last few years, including both theoretical studies [1], [2], [3], [4], [5], [6], [7] and practical applications.

Among the practical applications, a growing number of architects, artists, and enthusiasts, including the present authors, have created works using these perspectives as a way of overcoming the limited field of view of classical perspective. Among the spherical perspectives, equirectangular perspective has been the most used in recent times by artists and architects such as G. Michel, B. Sucurado, M. Scherotter, A. Hartman, C. Masiero Sgrinzatto, L. F. Olivero., A. B. Araújo (among many others) [2], [6], [8], [9], [10], [11], [12], [13], [14]. This popularity is due to its seamless connection with VR panoramas.

Before the popularization of equirectangular panoramas, the azimuthal equidistant perspective was the most in use, with a long history of very expressive drawings (see Barre and Flocon [5], R. Termes [15], [16], G. Michel [6], F. Casas [4], A. B. Araújo [1] for instance), but other perspectives have been occasionally used as well.

For instance K. Adams has proposed a quasi-polyhedral perspective, so-called tetraconic perspective [17]; a recent work by J. and M. Santoyo [7] has called renewed attention to the Mollweide projection first used by Barre and Flocon in a single illustration of their seminal work [5, pp. 194–195], and M. Lopas and J. Lima has explored an extensive range of spherical perspectives based on classical map projections [18], [19]. Recently, cubical perspective has also received renewed interest due to its easy integration with VR and its intimate connection with classical perspective [20], [21], being systematically solved in [3] as a special case of spherical perspective. Although its systematic solution is recent, cubical perspective shows a wide spectrum of possible applications in the fields of architecture, digital arts, design, engineering, fashion and product design [14], [22], [23].

While many of these authors have taken a hacker’s approach to drawing in spherical perspective, using trial-and-error methods to get at their intended expressive effect, others have approached the problem as a new chapter in technical drawing, calling for its own rigorous methodology, so that while some authors will do guesswork sketching on top of qualitative understanding of the principles, or work over pre-calculated grids, or even trace over photographic panoramas, other authors obtain accurate projections and vanishing points through descriptive geometry constructions [21, Ch. I.6] (Figure 1 and Figure 3).

Figure 2

Room 102. Equirectangular drawing and VR visualisation. © Lucas Fabian Olivero, 2017.

Figure 3

Equirectangular panorama of the courtyard at ISEL school of engineering, Portugal and VR visualization. © António Bandeira Araújo, 2018.

In the theoretical front, a rigorous definition of spherical perspective was proposed by A. B. Araújo in [1, p. 150] along with a general strategy within which every spherical perspective can be approached as a special case [24, pp. 23, 42].

This scheme was first implemented for the azimuthal-equidistant case [1], as a generalization of Barre and Flocon’s original methodology to the 360-degree view. Then it was applied to the equirectangular case [2] and finally to the cubical case [3].

The connection between spherical perspective drawings, their immersive anamorphoses, and the digital realization of these, leads to a notion of Hybrid Immersive Model (HIM), as a novel artistic medium [25] that connects the act of drawing by traditional processes with the new possibilities of digital visualizations.

Hybrid Immersive Models can be seen as part of an artistic current that, joining the theoretical developments of spherical perspectives and the technological developments of digital immersion, aims at technology that “reinvigorates rather than replaces the discipline of sketching” [24], building immersive environments through a reimagining of traditional perspective drawing, where the operator rather than the machine is the one building the illustration [21, Ch. II]. This requires adequate methods of geometric abstraction to reduce the complexity of the representation to a set of lines, planes, geodesics, and vanishing points, so that what the computer does in millions of pixel-level operations can be done by the draughtman in a manageable number of hand and brain operations. It also requires technology that properly facilitates the seamless transition from the handmade drawing to the visualization.

1.2 Features from Existing Software

Spherical perspective drawing software can be divided in two classes:

1.2.1 Adaptions of General-Purpose Software

Adobe Photoshop used to feature the “SphericalMap viewport”, a tool that allowed the user to draw directly on the VR rendering (Figure 4). This tool, officially discontinued from version 22.5 on [26], was useful for editing nadir and zenith of photographic panoramas. As an extended adaptation, it was used for spherical drawing as well, using all the raster-based tools of Photoshop such as pencils, brushes, masks, and so on. It was possible to switch between the VR and the full equirectangular projection and draw in both. However, such a switch was not instantaneous: the new view used to take some seconds to load, depending on the computer’s performance and the characteristics of the drawing.

Figure 4

VR viewport tool of Adobe Photoshop v22.4.3.

In any case, there were no tools for treating the equirectangular view as a proper perspective (e. g., with vanishing points), so the possibilities of this global view were underused. Furthermore, this tool was not optimised for showing dynamically the relationship between the spherical perspective and the VR navigation during a live drawing performance.

1.2.2 Dedicated Software for Equirectangular Drawing

Presently there are two main options for dedicated equirectangular drawing software:

Eq A Sketch 360 [27] presents itself as a serious toy for drawing and learning spherical perspective. It introduced an innovative system called Eq A Snap for drawing any equirectangular line (in fact a geodesic) using a sliding grid and a snap-to ruler (Figure 5). It boasts rather unique perspective tools for functions, such as finding antipodes of vanishing points and the apex of geodesics [27, p. 1], but is otherwise very barebones, even using an external rather than an integrated VR viewer. The program assumes either a previous knowledge of the fundaments of spherical perspective – in which case it can be used as a production tool – or it assumes the user is in the process of learning those fundamentals, with the aid of some external materials. It can be baffling for the uninformed, casual user that approaches it unawares.

Figure 5

Drawing using Eq A Sketch 360 on MacOS.

For production work, the lack of smooth brushes, shadows or gradients means that the program is best used as part of a larger pipeline, and the resulting perspective drawings should be finished in other applications such as Inkscape/Illustrator or Krita/Photoshop (Figure 6). Eq A Sketch 360 runs in Windows, MacOS and Linux, although it is mostly supported in Windows. A web-based JavaScript version is forthcoming.

Figure 6

Equirectangular drawing using Eq A Sketch 360 and Adobe Illustrator. © Lucas Fabian Olivero, 2020.

Microsoft Sketch 360 [11] is an app dedicated to journaling and multimedia. Like Adobe Photoshop it has an integrated VR viewer and allows the user to draw in both the perspective and the immersive view. The integrated VR viewer automatically points the camera at the position of the last stroke. After a recent collaboration, Sketch 360 also has true perspective tools, having adopted the Eq A Snap system from Eq A Sketch 360, i. e., the combination of Eq A Sketch 360’s equirectangular snap-to system (here styled as the “two-point stencil tool”) with its color-coded sliding grid.

Of the two programs, Sketch 360 is the more adapted for casual drawing, with more refined interface, integrated VR and drawing tools such as layers and proper brushes, while Eq A Sketch 360 remains the most apt for learning and drawing formal perspectives, with more elaborate perspective tools and a more flexible and unintrusive grid system.

1.2.3 Dedicated Software for Azimuthal-Equidistant Drawing

As far as we are aware, there is a single drawing tool for this spherical perspective, a prototype based on GeoGebra platform [28]. The tool is expected “to be useful for researchers, teachers, and artists (…) both to learn spherical perspective methods, and as a production tool, in a pipeline that may feed into vectorial programs for further rendering” [29, p. 159]. The tool computes the flattened geodesic between two dynamical input points. Then it is possible to trace the arc of geodesic between two further dynamically controlled points inside the chosen geodesic. By “dynamic” we mean that when the reference points are moved, the geodesic is automatically updated (Figure 8).

Figure 7

Drawing boxes in Sketch 360 using the 2-point stencil, i. e., the Eq A Snap tool and sliding grid adopted from Eq A Sketch 360.

Figure 8

Drawing an azimuthal-equidistant perspective with GeoGebra. © António Bandeira Araújo, 2021.

The drawing program makes use of the dynamic geometry functions of the GeoGebra platform on which it is based. It is in fact made as a simple set of JavaScript functions on top of this software. This gives it a very special character as the most technical drawing oriented of the programs we have discussed so far. Like Eq A Sketch 360, the program has an intimate relation with the formal technical drawing methods of spherical perspective, but there is a difference: the methods described for the 180-degree fisheye perspective by Barre and Flocon [5], and their generalization by A. B. Araújo to the 360-degree case [1, p. 150], assume an intimate relation between auxiliary orthographic, descriptive geometry diagrams and the final spherical perspective renderings.

This connection, much more intimate than in the equirectangular perspective methods, is well expressed in GeoGebra, with its native dynamic, vectorial tools for Euclidean geometry.

Even more than is the case with Eq A Sketch 360, this drawing app requires the user to be aware of proper spherical perspective techniques. It can also be helpful in learning such techniques. When these are known, it can be a powerful tool. In figure 8 we can see it being used to draw a 360-degree view in azimuthal equidistant perspective. The tiled floor was made using the integration with the top orthographic view, and on top of this chequered floor several geometric forms have been established, such as cubes and ramps. The relative sizes of the objects are perfectly controlled through the use of vanishing points and orthographic projections.

This program is currently just a prototype, as suggested by the fact that it remains unnamed. It has shown itself useful in illustration work (see the cover drawing of the July 2021 issue of the magazine of the European Mathematical Society) but it is slow, methodical, and cumbersome, and completely unfit in its present form for our purposes. Not only it would require a very patient audience to appreciate the deliberate geometric methods, but GeoGebra tends to add its own layer of slowness as the number of elements in memory grows. Also, the fact that the program works best in a pipeline (as GeoGebra has limited artistic rendering tools) adds another layer of awkwardness to a live performance. The program needs to be made standalone and optimized to be useful for our needs. Also, conversion from fisheye to VR is somewhat more finicky that conversion from equirectangular, as the circular outline of the perspective projection requires a more careful alignment.

1.2.4 Dedicated Software for Cubical Drawing

The plugin 360° Art from Oniride for Photoshop 2015.5 [30], allowed raster-based drawing within a cubical map.

The plugin has been already deprecated and is not functional anymore in newest versions of Photoshop. It had guides and colours, and a separate window for the VR visualisation (Figure 9). The side viewport showed a classical perspective and could be fully navigated around the observer. However, the viewport needed to be updated manually after every modification of the content within the cubical map.

Figure 9

Oniride 360 Art plugin for Photoshop 2015.5.

We insert this option here for completeness although it is a borderline viable alternative. It allows cubical drawings in a dedicated interface without however having any tool for treating the drawing as a proper, formal perspective, much less as a spherical perspective. This plugin does not really fit into our requirements, but we include it since we are unaware of other software for cubical drawing, so far.

1.3 Summary of Current Software Options

From 1.1 and 1.2 we see that the existing literature covers several methods for drawing either physical or digital spherical anamorphoses (equirectangular, fisheye or cubical) with different levels of accuracy.

In terms of software, for the equirectangular and the cubical formats there are just raster-based solutions. For the former there are stand-alone software while for the latter a plugin limited to one specific version of Photoshop. For the azimuthal equidistant format, a vector-based proposal attached to the GeoGebra platform. All proposals excepting Sketch 360 can cover both Windows and macOS operative systems.

In terms of parallel visualization anamorphosis/VR result, both Sketch 360 and 360° Art allow a simultaneous view of the perspective drawing and the VR result in a parallel viewport. Yet, in either case the viewport must be operated by the draughtman himself, which may not fit the requirements of a performative dynamic app as we want to produce. The GeoGebra tool for azimuthal equidistant format and Eq A Sketch 360 have no direct VR visualisation available. In the GeoGebra tool case, it must be converted to the equirectangular or the cubical format before navigation is possible (the azimuthal equidistant is not currently a standard format for such a task).

As for purely handmade drawings (say, pen on paper), currently, most spherical perspective software forces the user to follow a chain of steps (often cumbersome) before seeing the VR results. It is hard to guess the immersive view without a high level of knowledge on spherical perspective. This discourages the beginner and limits advanced artists from using equirectangular perspectives in live presentations, as the untrained audience certainly cannot easily decode the perspective picture in their minds.

Indeed, in some exhibitions already held [25], [31], [32], [33], visitors experimented many difficulties understanding the relation between the plane spherical perspective drawing and its correlated 3D rendering. For simple occasional enthusiastic visitors, the explanation might result too theoretical. Instead, when the user experimented directly the “sphere from within” with the help of VR glasses, the comparison was more straightforward, and this resulted in a better appreciation of the correlation between the physical anamorphic drawing and the classical perspective shown on the screen.

Figure 10

Image input: Eq A Sketch 360’s interface (top), handmade drawing (centre) and existing panoramas (bottom).

2.1 Components, Interactivity, and Functionality

We recall the setup and functionality of the artefact as presented in [34]:

Interactivity and Functionality

The artist (AR-1) chooses a medium between IN-1, 2 or 3 (Figure 13 and Figure 14). It is possible to choose one or to mix several inputs. This latter option allows the possibility of a drawing interactively composed with other artists or even with the public (one per each medium).

An external visitor (VI-1) interacts with IN-4 sending position data. The program defines the position of the camera through Compass and Gravity sensors, thus filtered to channels “Gravity 3” and “Compass 1” (Figure 11). Every new position of the phone updates the camera within the visual virtual sphere, displaying a new section of the immersive drawing to the audience.

OU-1 shows the classical perspective resulting from the VR modality, that is, the spherical anamorphosis is dynamically converted in a linear perspective shown in the viewport according to the camera’s position within the visual sphere. In the picture’s experimentation, the Field of View was set at 90° (Figure 16).

Figure 15

Mixing inputs (IN-2 + IN-3).

Figure 16

Live VR result of the mixing.

In short, a first subject (AR-1) live draws in equirectangular projection using either traditional or digital techniques. At the same time, a second subject (VI-1) defines the position of the camera sending data through a mobile phone (IN-4). Both AR-1 and VI-1 (along with an audience) watch the results through OU-1.

2.2 Novelties

Thanks to this artefact, the artist can focus on the drawing while members of the audience are free to choose their preferred point of view. The artist has a wider and more complex view, perceiving the macro and the micro at once, keeping both the whole and the detail in view during the entire process. General visitors can instead appreciate the result by seeing the image directly in VR. Nevertheless, the curious or knowledgeable visitor can also perceive the connection between all the elements in one same intellectual, interactive, and dynamic exercise.

The artefact presents some important novelties regarding the previously analysed software:

1. it offers the possibility of mixing several inputs into one unified output (Figure 15 and Figure 16), integrating inputs from other already existing programs.

2. it opens the possibility of using spherical perspectives within live performances, showing an on-the-fly dynamic and interactive visualization of the result.

3. it covers to the needs of both neophyte and experts: it introduces the former to spherical perspectives through an instant trial-and-error game; while the latter can draw the most complex constructions and use the artefact to share, explain and communicate results to the public.

2.3 Live Test

The artefact was presented during the ARTECH 2021 congress, held between 13 and 15 of October 2021 at Aveiro, Portugal [36] (Figure 17). The installation was shown at the “Contingency” digital media art exhibition of the congress, curated by Maria Manuela Lopes. During the first day, the authors explained the functioning of the artefact to the visitors and performed a small demo (Figure 18). During the rest of the congress, the artefact was freely tested by the public. The authors collected testimonies from visitors to outline the overall reception of the artefact.

Figure 17

The artefact during the exhibition at ARTECH 2021. Ph: Lucas Fabian Olivero.

Figure 18

António Bandeira Araújo and Lucas Fabian Olivero explaining and performing a live demo. Ph: Juliana Wexel.

We talked to Chiara Masiero Sgrinzatto, who is an illustrator specializing in immersive drawing. Chiara was a key person to test the usability of the artefact since her career is strongly based on equirectangular drawing [12], [37]. She tested the artefact thoroughly (Figure 19), and reported on it as follows:

“Thanks to the ability to display the drawing both in equirectangular and in interactive format, it is easy to understand how the immersive drawing is taking shape to people who are not familiar with spherical perspective. In a future development as a software, the system could be very helpful for education projects, for live painting performances, and for all the purposes involving a general public that could not understand clearly the distortions of the spherical drawing. It would be an interesting feature to add the possibility to interact on the tridimensional re-projection: a tool for sketching or taking notes on the output canvas, allowing a bi-directional participation to stimulate design processes and ideas sharing.”

Chiara Masiero Sgrinzatto

Independent artist, illustrator and designer, Venice, Italy

We also gathered reviews from people with different backgrounds, not necessarily related to the field of drawing. We wanted the artefact to be useful for live performances, in such a way that it could allow the artist to freely draw in equirectangular perspective while the visitor might be just watching and interacting with the VR results. Therefore, we talked to random visitors, asked them about their background and interests, and then about the impact produced by the artefact:

“I haven’t had many experiences with spherical perspectives before, maybe we learned a bit about the theory in school, but nothing I would clearly remember. I, therefore, do not regard myself as savvy regarding this subject. I am an interaction designer and researcher working in the field of smart textiles, specifically in designing ways to communicate this new medium to everyday users.

Although this may not actually be accurate, I understood the drawing and the artifact as “theory” and “practice” of the project. As the drawing showed how it is done, the artifact allowed me to see the immersed representation of what it is. This hugely increased my understanding of the drawing and the project itself.

The whole setup was not very intuitive, I initially perceived it as a performance, and without a call to action would probably never discover the artifact on the table. However, once my attention was brought to it, it was immediately clear what the purpose of the artifact is and how it can be used. It added a big wow effect already, it might be even more impressive and immersive with a VR headset instead of the phone. What I particularly enjoyed was that the immersive representation through the artifact was changing in real-time, as the drawing was being drawn.”

Sara Mlakar

Media Interaction Lab, University of Applied Sciences Upper Austria, Hagenberg, Austria

“I am a multimedia artist, journalist, researcher and doctoral student in Digital Media Art. I hold a Masters’ Degree in Literature and Culture. I lectured at the Department of Languages, Literature and Modern Cultures at the University of Bologna, Italy, and I had experiences as an Assistant Professor of Journalism at Universidade Veiga de Almeida, in Rio de Janeiro, Brazil.

In the live action experience, it can be seen that the artifact transforms something mathematical into a performative exploration. Through interactivity, the application provides an immersive and playful experience at the same time, and also allows the lay public in general to understand how the drawing is generated. It felt like having an ‘Escher effect’ literally, in the palm of your hand.”

Juliana Wexel

Centro de Investigação em Artes e Comunicação (CIAC), Universidade do Algarve, Portugal

“The experience allowed me to get closer to spherical perspectives. I liked that it linked traditional drawing methods with new technologies. I could say that the experience suggests an intimacy between the viewer and the work, however, I would like to have a device that allowed me to be more immersed and intimate with the artist’s work (maybe a VR headset).”

Anonymous

2.4 Improvements and Tests to Be Made in the Short Time Frame

Figure 19

Chiara Masiero Sgrinzatto testing the artefact during the exhibition at ARTECH 2021. Ph: Lucas Fabian Olivero.

Based on the current state of the artefact, plus the feedback received during the exhibition, we list important points to be reviewed and some necessary improvements to be made before entering the pure coding stage:

1. Camera’s position is determined using Compass and Gravity sensors. Presently, a glitch occurs during the navigation: there is an inversion/discontinuity of the view under certain rotations, probably due to the use of Euler angles; this will be solved by using Quaternion-based calculations.

2. OSC data is provided by a wired connection and the use of Zig Sim forces the phone to be constantly on with the application open. It will be useful to develop an open-source app to send wireless data in the background and free the phone screen for other uses, e. g., to stream back the output content and test a forced stereoscopy with VR glasses. This way, a visitor wearing the VR glasses can determine the camera’s position through the movement of the head and be free to move about in space at the same time.

3. There is no front-end interface available for switching among inputs. This is necessary for a more efficient and simplified operation.

4. Currently, the Field of View can be modified within TD, but it remains constant while the artefact is running. We plan to add a functionality for changing it during runtime directly from the front-end interface.

5. This first application works with equirectangular projection, yet we aim to integrate cubical and azimuthal-equidistant perspective inputs as well.

2.5 Other Possible Integrations?

Some projects have used the CAVE system for 3D sketching in ways that might integrate well with our purpose. For example, CavePainting from Daniel Keefe [38] and Sketching in Space from Habakuk Israel [39] have been exploring CAVEs implementation within early conceptual design. These proposals explore a paperless sketching tool analysing the impact of drawing inside the 3D environment with ad-hoc tools. They are viewer-centred perspective artefacts [40, p. 65], that is, the pictures on the walls of the CAVE change with the position of the viewer so as to maintain an immersive anamorphosis at all times with regards to the position of the observer.

In this case, the workflow goes directly from the designer’s thoughts to the 3D digital model, capturing early conceptual design ideas and sculpting their digital model on-the-fly [38], [39], [41]. These artefacts consider 3D space as a different and more direct medium for externalising thoughts, a medium in which modelling happens automatically and there is no need for projecting anything, either on paper or at all [39, p. 4]. All this enhances the usefulness of digital interfaces for binding two stages of design: sketching and 3D modelling.

Also of interest is the project SketchTab3d, which proposes a 2D sketching tool combined with a 3D sketch library [42]. It uses a tablet with a regular dashboard for 2D sketches input, and it is connected with a CAVE system [40]. Once inside the immersive 3D environment, the user can navigate the 3D libraries and perform some actions such as moving, deleting, multiplying, or scaling 2D sketches. Other important features are multiple collaboration and synchronized sketching.

The 2D sketching tool offers the possibility of creating, modifying, or deleting sketches. From a web-based interface that uses an HTML canvas, the user can access the dashboard, enter the CAVE mode, create a new sketch, or logout. When drawing mode is selected, the dashboard imitates a plane paper, without grids or references to any specific perspective system.

We can envision an integration of SketchTab3d with our own purposes by adding the option of choosing a spherical perspective system when using the 2D sketching tools. That way, the program might incorporate and preload the chosen perspective’s functionalities, such as the Eq A Snap tool from Eq A Sketch 360 if using the equirectangular perspective. One goal of this incorporation is to enhance the relationship between 2D immersive drawings artworks and their direct navigation.

Such project integrations might help disseminate the recent advances in spherical perspective drawing techniques by providing automatic feedback that visually verifies the correctness of the immersive constructions, and, on the other hand, these immersive drawings might provide CAVE tools with an additional range of applications to explore.