New Digital Realities – Blending our Reality with Virtuality

1 Introduction

We live in a complex world, in which physical and digital environments, media, and interactions are woven together. The lines between natural and artificial, reality and virtuality, human and machine intelligence seem to blur. We observe that those originally distinct semantic differentials are no longer clearly distinguishable. Instead, they form a continuum in which these two aspects define the endpoints, but the much larger area is given by a mixed space in-between.

Let us consider some examples of such blended or mixed spaces: About 300.000 years ago, our world consisted only of natural surroundings. Then, the Homo sapiens started to produce artificial objects ranging from stone tools, the wheel, to calculators. Nowadays, most objects in our urban areas are artificially made by humans, and even naturally appearing objects are often artificial ones or at least partly artificial. For instance, even natural trees have been typically planted and pruned by humans, and thereby they move along the continuum towards more artificial objects.

Next, in classical computer systems (based on binary representation), data is encoded in fundamental units called bits, where each bit represents a distinctive state, which is either 1 or 0. With this technology, human intelligence has already been simulated in some tasks (e. g. image recognition or speech analysis and synthesis) and sometimes even extended for other tasks (e. g. numerical data processing) by computers. However, we observe a shift from this classical model of computing towards other approaches such as quantum computing. Here, data is encoded into quantum bits (qubits), which can represent a one, a zero, or a coherent superposition of both states simultaneously. The probabilistic approach of superposition together with quantum entanglement, which allows qubits to affect the state of other qubits, promises enormous performance improvement for several complex tasks. Again, with qubits, the area between 0 and 1 can be considered as a continuum.

Another example can be found in virtual reality (VR), which has gained enormous attention recently. VR denotes an immersive computer-generated multisensory stimulation, in which users can dive into a fully artificial reality. Again, VR can be seen as a semantic differential to the real world. However, other techniques such as augmented reality (AR) or augmented virtuality (AV) combine virtual and real objects in a coherent and consistent space – denoted as a reality-virtuality continuum – along which the different mixed reality (MR) technology can be described.

These new digital realities with an increasingly blended space of mixed realities require new paradigms for the interaction and communication between humans and technology. One major goal of HCI research is to support humans in using technology so that they can solve their tasks in an effective, efficient, and satisfying way. In this context, the analysis, design, implementation, and evaluation of future human-computer interactions (HCI) play an essential role.

2 A Brief History of New Digital Realities

Digital transformation and new digital realities affect nowadays almost every aspect of our daily lives and will further shape how we work, learn, communicate, and live in the future [13]. In particular, two emerging technologies with the potential to have a long-term impact on human-computer interaction are VR and AR. One of the first fictional descriptions of Virtual Reality occurred in 1935 by the American science fiction writer Stanley G. Weinbaum. In the story, a professor invented a pair of goggles that enabled “a movie that gives one sight and sound [...] taste, smell, and touch. [...] You are in the story, you speak to the shadows (characters) and they reply, and instead of being on a screen, the story is all about you, and you are in it.” [18]. With his explanation, Weinbaum perfectly describes a psychological state known as telepresence, which is key in defining Virtual Reality [15]. The invention of VR, which denotes the generation of completely synthetic environments using computer technology [9], dates back to the 1960s when Morton Heilig patented one of the earliest examples of immersive computer-generated environments, the Sensorama [5]. The motivation here was to make people feel like they were in the movie simulating a motorcycle ride through a city environment and letting people see the road, hear the engine, feel the vibration, and smell the motor’s exhaust. While the Sensorama was a static device in which people had to put their head into, Comeau and Bryan created 1961 the first head-mounted display (HMD) called the Headsight, consisting of two video screens, one for each eye, as well as a magnetic tracking device. Around the same time, Ivan Sutherland envisioned the Ultimate Display as a “room within which the computer can control the existence of matter” [16]. In 1968, it was Sutherland who created the first partially see-through head-mounted display (HMD) [16] known as the Sword of Damocles, which is widely considered to be a precursor of current AR technology. Though having its origins in the same decade as VR, it took almost 30 years before AR emerged as an independent research field (for an extensive historical background of both VR and AR see [3]). In contrast to immersive VR technology which fully replaces the view to the real world with computer-generated content, AR aims at enhancing real-world views by embedding additional virtual content [2]. Both technologies became particularly popular in the gaming and entertainment sector [12]. According to the Gartner hype cycle 2015 [17], both technologies already passed the phases of technology breakthrough and the peak of inflated expectations, and are estimated to reach mainstream adoption by 2025. In the report of 2019, both VR and AR were even excluded from the list of emerging technologies because, according to Gartner, they already reached a mature state [13]. However, related technologies such as immersive workspaces, for instance, for online meetings, or AR clouds for sharing virtual content, have entered the innovation trigger and will further boost AR and VR technologies.

A more differentiated analysis of the strengths and weaknesses of both technologies led to a set of customized applications beyond the entertainment industry. VR can immerse users in environments, which represent a different place or time while being isolated from the real world. This can be beneficial for applications such as therapy, military training, and design review [6]. In contrast, AR technology is preferable to VR systems if users are required to see and interact with their real environment, for example, for navigation, maintenance tasks, and computer-assisted surgeries [12]. Regarding the virtualization of real-world objects, VR and AR are only two stages within a continuum, which was introduced by Paul Milgram in 1994 [9]. While purely real environments (REs) and completely virtual environments (VEs) mark the extremes of this continuum, a range of Mixed Reality (MR) environments, i. e. spaces that combine real and virtual elements, lies in between. Whitton et al. [19] describe fundamental challenges to enable such MR environments, including the merge of real and virtual scene elements, the tracking of physical objects, as well as the simulation of plausible virtual-physical interactions. If the MR environment is predominantly real and only single physical objects are embedded, the overall state is denoted as AR. As opposed to this, Augmented Virtuality (AV) refers to a primarily VE to which some amount of real objects has been added [9].

Though Milgram’s continuum provides a basis for the continuous virtualization of an environment, it is limited in the sense that it only considers augmentation, i. e. the addition of virtual content, as a possibility to increase the virtuality of objects. Related research projects, for example, by Broll et al. [4], demonstrate that REs can also be modified by (partially) removing real content, resulting in a Diminished Reality (DR) state. Analogous to the correlation of AV and AR, Diminished Virtuality (DV) is the equivalent to DR as it refers to a predominantly VE, from which some of the virtual elements are removed and replaced by visualizations of real-world objects. While augmentation and diminishment are opposite to each other on a conceptual level, they both increase the overall virtuality of the affected objects. This is because technologically the diminishment of real-world objects also requires the superposition of virtual content, which in this particular case represents the portion of the scene that was previously obscured by these objects. Instead of adding virtual or removing real content, an alternative to achieve such a (partial or full) virtualization is the modulation of existing objects, for example, by using image filters in order to highlight specific features, or by virtually transforming visual properties of the physical object.

To integrate augmentation, diminishment, and modulation into a comprehensive taxonomy, Mann et al. [7] introduced an orthogonal axis to Milgram’s continuum, called Mediality. However, we argue that modulation of REs also decreases the amount of unaltered real objects, and therefore can be understood as a third path to increase the virtuality of such environments. Consequently, we extend Milgram’s continuum by two additional paths between reality and virtuality resulting in (i) augmentation, (ii) diminishment, and (iii) modulation of physical and virtual elements within an environment. The three operations should not be understood to be mutually exclusive since they can be applied simultaneously to different parts of a physical object to increase its overall level of virtuality resulting in new digital realities in which reality and virtuality are blended for novel forms of HCI. Hence, we will live in new digital realities in which physical and virtual objects are blended together. However, physical objects are not limited to the traditional augmentation, modulation, and diminishment, but all objects can change their state along the RV continuum, i. e., also virtual objects might materialize, for instance, by digital fabrication or 3D printing technologies. Furthermore, real, mixed, and virtual objects can interact with each other in a physically plausible way. And finally, humans will be able to seamlessly experienced blended spaces without requiring them to switch the display technology. The blending of reality and virtuality will eventually become our new digital reality.

3 About this Special Issue

The previous sections described the main ideas of blending reality and virtually into new digital realities. The following section will briefly introduce the different articles being part of this special issue and explain how they blend with this trend. The articles that are part of this special issue demonstrate, for example, how researchers can understand the current state, learn from specific contexts to improve the current state, explore novel interaction concepts, and look at visionary approaches for the next generation digital realities.

4 Conclusion

This issue presents work on Digital Reality, including VR, AR, and usability measuring using emotion recognition. The presented works show that the future, the way we live, work, and are entertained heavily depend on the advancements in computer science, which help adding to the quality of displayed content, and allow for the creation of photorealistic, vivid, and more life-like VR experiences as well as for integrating digital media and computation in our physical world. The natural interplay of virtual objects and their real environment was already subject to several research projects, which thereby reveal future prospects of Blended Spaces (e. g., the simulation of realistic illumination effects [11] or shadows [10] in MR environments). Considering the virtual content separately, the film industry demonstrates algorithms to create computer-generated content that is already close to being indistinguishable from a real video recording.

While the rendering of these graphics still takes a lot of resources in terms of time and computational power, it is reasonable to assume that real-time renderings with an equal quality will be achievable with improving computing systems that support digital reality [1], [14]. Beyond computer graphics and technology evolution, New Digital Reality also touches areas of analogue life, especially when technology captures us in the physical world.

Therefore, if we change the way we communicate, play, work, and recognize others emotions, new challenges occur in terms of ethical questions and users’ privacy protection [20], [8].