Keywords

1 Introduction

The immersive filmwork is required to be a panorama video, and the finished work needs to be experienced by the audiences with head-mounted display. Compared with the plane images familiar to us, the immersive film are brand new from the shooting and post production tools to record shooting idea, which shall be explored and thought in terms of technology, expression language and work positioning from a brand new perspective. Undoubtedly, the immersive film with the head-mounted display as the hardware platform is a new audiovisual medium, just as what Frank Rose, the author of the Art of Immersion: How the Digital Generation is Remaking Hollywood, Madison Avenue, and the Way We Tell Stories, thought about, “every new medium brings about a new narrative form” [1], in my opinion, apart from narrative form, the experience appeal of immersive film are also different from those of other media.

The way the story is presented and the form to accept the story change accordingly as the narrative media transforms. The literary narration of the humans has a long history, from dictation to instant communication on the Internet, from frescoes in the temple to book printing and from cave painting to virtual images. For the communication pattern and acceptance experience of the story, the development of the science and technology promotes the transformation of the narrative media. The humans evolve from listening to stories to watching stories and then to experiencing stories. The receivers of the story may cover the listeners, readers and audiences. As far as the narrative experience of immersive images is concerned, several types of receivers, who provide more comprehensive synthetic perceptual experience or can be said to be the complete experiencers of the priming and undergoing of the story of the virtual world, could be covered.

Reading, listening and watching and other measures to accept the story have their respective experience modes and effects. Relative to reading, the readers unscramble the abstract writing language and proactively motivate their pre-existing life experiences to integrate and imagine the content of the book and their personal affective experience in the brain. However, painting and film with the way of experience based on watching provide the audiences with more intuitive visual effect. Most of the audiences are undergone the passive feedback within the specified content domain of the images. The interactive experience of immersive images lies in that the audiences (users) need to be part of the story in terms of the visual perception experience and the story emotional experience.

2 A Free Perspective Creates a New Mode of Narrative Experience

Immersive image is a kind of new mode of graphic narration, as it provides a free and mobile perspective to observe the world. The tradition painting and images both have the visual boundary, however immersive images attempt to eliminate this boundary, that is, the media is eliminated when the audiences receive the narration and the audiences then obtain the audio visual experience similar to the real world.

The process to experience the immersive interactive film works is characterized by the elimination of the existence of the media.

Comparatively speaking, such narrative media as fresco, book and film fully stress on the existence form of the media itself in terms of the design of narrative experience. The temple frescoes require the decorative design of the temple space, the books would take into account the texture and thickness of the paper for printing and the films would emphasize more on the environmental construction of the cinema. To a large extent, the experience of the design content and effect of the narrative media is an important part of the narrative experience. However, the experience of immersive images prefer to stress on the experience appeal to eliminate the existence of the media itself during the process of narrative experience.

All of the images face a certain problem—how to make the time and space reappear. All the narrative media organize the time and space in an orderly manner, based on which the story is presented to the audiences. Interaction creatively designs the time and space and triggers different narrative structures through the watch and selection of the story-telling scene space. For example, the audiences turn the neck and focus on the content of a certain scene space with the plot development and changes or the scene control of the actor-driven performance. The film may trigger different plots depending on the focus of the audiences on the plot and the time duration, while the maker of the film can lead the audiences to focus on some plots intentionally through the audio-visual elements, so as to push the narration forward according to the director’s intentions. The leadership of attention by the maker and the subjective selection of the audiences jointly form the process of the interactive narration. Comprehending from this perspective, the self-directed interactive narration experience space provided by the immersive interactive film is up to the maker to a large extent. The criterion of the spatial scale of the interactive narration that the audiences independently control, the control of the intensity of the leading narration with the maker at its core and the relationship between both of them, namely the treatment of the triadic relation between the audiences and the maker, are the most subtle and best embodiment of the artistic level of the maker on narration by the immersive interactive video art creation.

Hockney, a famous artist, once said [2], “This is the case with all pictures, all of which are drawn from a particular perspective. We view the world in the way of our own, even if it’s a small square object. For example, different people have different opinion on a small box. When walking into a room, every one notices different things by his unique feeling, memory and interest.”

When we are in the borderless story space as a whole that has not been cut, we would observe this scene and find our own interest point by following our heart, to further build the story world of our own, therefore the story truly lives in our heart. The experience relation among audience, image media and image content have be rebuild in immersive film art.

2.1 Free Perspective of Panoramic View Promote Natural Experience

Generally, immersive images do not edit the time and space in a single plot. Without the picture frame and the directional cutting and combination of montage, it in turn provides a more intense time-space reality. This natural reality, in addition to the real experience of the sense organs in a single vanishing point of perspective on screen texture, light and resolution offered by the traditional painting and 2D images, provides the sense of reality on space-time dimension, more importantly. The immersive images provide the audiences with the time-space dimension that allows them to freely move the perspective to watch the screen and receive the narration in a panoramic image space, and such an experience is closer to the real world.

The realistic perspective is in line with the physiological and psychological characteristics of watching. While looking around and staring, the point of sight (attention) moves with the inner experience and the trigger in the scene, forming the moving first perspective. In 2D video, we watch the screen in the way of focus perspective born with the video camera. However, we may watch in the psychological way in the immersive interactive film works and stay out of the control of field of shooting scale and film editing, thus receiving and promoting the narration from a free perspective. At this time, the image composition changes due to the change of the audience’s perspective, forming a mobile experience process. In reality, instead of looking at the world from a fixed point of view like a video camera, we keep turning our eyes and body to search the important and interesting information one after another. Because of the panoramic visual space, the immersive images provide the audiences with the habit to see things, haul the audiences into the story scenes and enhance the sense of immediacy that the audiences experience the story.

The traditional Chinese painting enjoys the way of experiencing the story similar to that provided by immersive film artistic works through a mobile perspective. The picture formed by “cavalier perspective” adopted in the traditional Chinese landscape paintings requires the appreciation of the moving focus. The viewer’s eyes move on the picture as if traveling between the landscapes. Like the way we usually look at landscapes, we always look at landscapes in landscapes. Viewers enter in to the scenes of the picture with the unfolding of the Chinese hand-rolled landscape paintings, as if the viewers walk in the mountains and experience a tour crossing time. However, the audiences are always excluded from the scene of the picture, in the paintings and images with a fixed vanishing point of perspective.

The immersive interactive images provide the audiences with a free moving perspective to observe and experience the story in a natural way. The art of video pursues “natural realism” in time and space and creates experiences of specific scenes through narration. Compared to painting and 2D video, the immersive interactive film works is more realistic in terms of the degree of freedom of viewing perspective and the integrity of time and space of the story. As we use the way of perspective to make the spatial performance effects of the picture closer to the real environment that we see with one eye in terms of art of painting, the immersive images pursue the similarity between the sense judgment of the works and the experience of the audiences for the real world as far as possible.

Guo Xi (nearly 1001–1090) pondered over painting completely from the perspective of mountains. In his opinion, the mountains are high and far, meaning “looking up to the top of the mountain from the bottom”, deep and far—“peeping the back of the mountain from the front of the mountain” and flat and far—“looking at the distant mountain from the nearby mountain”.

Hockney once said [3], “Naturally, there is a problem on what you see first, what you see secondly and what you see thirdly when people look at scenes. For a photo, all things are seen at the first sight, and the camera shoots from a perspective. People do not look at things in this way. It takes time to see, and the space is thus created.”

2.2 The Combination of Immersion and Interaction Makes the Experience of Immersive Film Closer to Natural Reality

With the behavioral reality, the users drive the occurrence, change and development of the event and obtain the narration experience in the anticipation and surprise, as the plot logic is in line with the audience’s psychological rhythm. The user is placed in the story itself with the trigger of the event and the promotion of the narration. Compared with being an onlooker, it’s the best narrative experience to live in it. The synaesthetic experience pursued by interactive design works is realized in the narrative art to the utmost extent. In the process of designing empathy, the artist’s design of the experiential relationship between the works and the audience is of great importance. In the interactive experience of the immersive images, synaesthesia is the artistic experience generated by the substituting relation of the interactive design results based on empathy in the audience and works. This kind of sense of synaesthesia close to the elimination of media is just the most unique charm of the immersive interactive images.

The realistic sense of art experience is one of the most important experiences pursued by visual art creation. And the art form that stresses the intuitive expression visually requires the sense of reality in language arts and effects as a matter of course. The more true the more intuitive, naturally, the stronger the sense of substitution. Visual art, which is characterized by intuitive experience, can only form a complete artistic expression if it’s true and thus the audiences can accept and recognize the narration psychologically and be touched, so as to achieve the original intention of the artistic works. In the experience design of video art creation, it is not only the realistic effect of the audio-visual and sensory effects presented by the medium of the works, but also the degree of emotional engagement in the experience process of the works by the audiences, as well as the audio-visual perception, emotion and sentiment experience relationship between the audience and the works of art. In terms of the experience form of the works by the audience, is it the face-to-face relationship between the eyes and screen? Or in terms of the audience’s psychology, the individual is in the same entirety of the time and space of the works? The experience form of immersive interactive images realies the latter and the sense of reality achieved by the experience result is far more than other existing narrative media. At the cinema, the mood of the audience will not change the image on the screen anyway. No matter how sad or joyful, the emotion of the reader will not change the changing story and the role behaviors in the book. However, in the immersive interactive artistic works of video, the panorama images with immersion wrap the audience, and the visual boundary of the medium is eliminated and the real environment is replaced by the world of immersive images, while the emotion and reaction of the audience will have a mutual relation and affection with the rhythm and perspective of the plot of the works. The sense of reality, through the language of interaction and immersive images, integrates the audience and the works as a whole and realizes the seamless joint of emotion and the works. The audio-visual sense and the boundless perspective realize the immersion of sense organs, and achieves the realistic experience of a higher level.

The form to experience the works by the audiences is interaction rather than watching. And it’s more than watching from a certain distance. Because the audiences will interactively control the works according to the change of the narrative process of the immersive images and the will of subjective emotion, rather than accept the audio-visual information of the works passively, the narrative form and language of immersive interactive images are bound to be different from those of novels and films and other watching and reading narration works. At this time, we need to understand the original significance of narration, that is to say, we try to empathize the audiences through the artistic media and language and achieve the goal that the audiences obtain the thematic affective experience. In the artistic works of immersive interactive images, on the one hand, the audiences accept the elements used for the purpose of empathizing and realize the emotional substitution visually and behaviorally through the interaction via self-behavior, and forget oneself in reality in the immersive image space that is panoramic to be “me” in the narrative plot of the works. This “me” is the art of me, not infected with someone and the “me” in the works. This me is the performance of me in the works context who is formed in real life by life experience and values, and the me defined by the significance of the specific narrative scenes of the works of immersive interactive images. Viewed from the perspective of receiving and feed backing the narrative information of the artistic works, the audiences of the artistic works of immersive interactive images have a meaning different from the audiences of 2D video and paintings.

2.3 Scene—Narrative Space of Immersive Film

In 2D images, the narrative logic and the sense of order of information are formed by field of shooting scale, layout and montage. With specific grammars and audio-visual elements, the audiences can comprehend and receive the narration. How to build the logic of narration in the panoramic immersive images is actually that the immersive images need to conduct a complete narration to build the perfect visual order in the case that the audiences can freely move the point of sight. Just like the traditional Chinese roller landscape paintings and Suzhou Gardens of China, one step makes a difference while the scenes are connected with one another, forming a complete narrative space that is of artistic conception—“scene”, which is the scene scheduling of the narrative factors around the point of sight of the audiences.

It has its own uniqueness for the performance of the panoramic space. Panoramic space is a brand new concept of image space for the frame-type shooting scale of image, which is a conversion from plane image to three-dimensional space. This immersion film present of panoramic view space brings about a real immersion experience. The pursuit of immersion experience is exactly our original intention to constantly improve the quality of the cinema and go to the cinema to watch the movies and VR is just a phased step for this original intention. The traditional movie scene, under the constraints of the shooting scale, presents a dynamic picture, and the audiences watch the scene picture face to face, while the VR panoramic presentation provides the audience an audio visual space, and the audiences are just in this space. Both of these are essentially different in the visual representation of narrative.

In the narrative design process of panoramic visual space of immersion film, it is quite necessary to establish the principle of the main view to tell the story with the characteristics of time process. If there is no shooting scale to limit the audiences’ attention when watching the movie, the main visual object in the main view range of the panoramic space is the main content carrier to narrate, and what seen from the main view is the design of the narrative content expressed by the creator in the image space, and then the switch connection rule between the main view and the main body content is particularly important, and the main view has become the core content of immersion film audio-visual narrative language.

In addition, as the current mainstream film production process, the scenery, the lighting settings and the actor position are designed around or in front of the lens. The concept of camera lens is weakened when the immersion film are dispatched on the scene. The directly considered is the positional relationship between the panoramic story space and the eyes of the audience, and even the problem of “jump axis” contraindicated in camera stand design emphasized to ensure the smooth narrative for traditional movies is inexistent in the panorama narrative. Therefore, the principle of scene scheduling changes when shooting a VR movie. However, based on the main view narration principle described above, it is necessary to emphasize the connection between the main views and the logical relationship of the camera stands in the entire panoramic space, since the camera stand still represents the view of audiences to watch the entire story, and the audiences are “present” in the immersive experience. The main view is set in the narrative space of the panorama. Since the space is sufficiently close to the real life environment, it is naturally possible to set the secondary views to improve the richness of the narrative in the panoramic space, which is also the unique charm of the panorama view of immersive film narrative.

When the immersion film showing with head-mounted display, the “story” scene will be presented in front of you in depth, and it seems to run across the time and space and reach the event spot. This is the theoretical experience effect of panorama image media and also the objective of the work experience during the creation. The plane image works is realized to edit the story time based on the scene with the features of shooting scale in the expression logic of “looking at here, and look at there…” said by the director to the audiences. For VR head-mounted equipment, the experience brought by the panorama image is closer to “loading” the users on the spot. When the users wear the head-mounted display, the work tells the users that “you are here, and you are here…”. In the panorama image, the sense of immediacy and the narrative rhetoric of the immersed experience are corresponding, and the integrity of panorama view is not to realize the integrity of the narration plot through the time-space cutting and recombination. More importantly, the narrative relationship between creators (recorders), works, recorded objects, and audiences is not “peeped” as described in the classic movie theory but “present in scene” the new relationship.

In general, the narrative method of panoramic view in immersive film still needs to follow the narrative as the basic principle of time art and objective physical characteristics of human eyes forward to see things. On the basis of this, how to play the narrative value in “extra” space is one of the directions that creators can explore in immersive film art creation, which is also the key of how to enhance immersion and how to construct the unique experience of immersive film narrative. Immersive film creation provides a different requirement on the director’s ability to schedule the scene and the actor’s ability to perform from that of the current mainstream movie art creation. The cameraman shall still consider the angle when setting the camera stand and scene sequence, but the relationship between one camera with visual space position and rhythm brought by multiple lens shall be considered, which can be seen that the difficulty of making immersive film art is enhanced, but this is the demand for art experience innovation of immersive film art in creation (Fig. 1).

Fig. 1.
figure 1

Recording and shooting based on panoramic image and panoramic actual sound

Full size image

3 Construct the Narration of Immersive Images with Interactive Behaviors

3.1 Interactive Narration Is the Key to the Artistic Experience Innovation of Immersive Film

In the works of immersive images, we no longer use the scenes with border and the mode of focus perspective to watch the images, while interaction becomes an important tool to push forward the narration. When the narration is conducted by the combination of immersive images and interaction, the audiences will obtain the artistic experience to enter another time and space from the works. The means of interaction becomes more diverse and mature with the development of technologies. The forms of immersive interaction—spatial gestures, voice and face recognition and even the involvement of AI make the interactive experience more natural. Interaction of the dynamic behaviors of the audiences will bring the fun of experience to the behavior itself, such as the popular VR games. However, the involvement of narrative images requires more abundant interaction fun. The purpose of interaction can be to satisfy the visual senses, music appreciation or emotional experience of joy, anger, sorrow and happiness, and even the judgment and choice of values.

Necessity of interaction as the art experience mode of immersive film.

The immersion film can inevitably prompt to create a totally different artistic experience than the traditional movie art, rather than merely enhancing immersion in hardware. The immersion film can carry a complete rich narrative to express the profound artistic significance, which can be called movie art; otherwise, it still remains in the level of “special movie”. On the one hand, this requires an increase in the production equipment, software and playback equipment, on the other hand, the narrative innovation is more important.

From the morphological characteristics of the immersion film, the effect of the immersive experience brought by the panoramic view of the immersion filmspace is not that the audiences watch the lens picture as the spectators, but that the audiences are just in the space of the movie, which must inspire the audiences different experience appeal, therefore it is the best choice to interact with the event in the space. With the interaction, the audiences can experience the give-and-take consciousness narrative rather than the audio visual experience and the audiences are changed from the receivers to the interactors in the narrative space. The interactive narrative is the best experience to promote or match the immersion film, and the sublimation from audiovisual experience of traditional movie to the behavior percipience experience integrating audio visual and operation is bound to further enhance the immersion of the audience in the experience of VR video.

3.2 Construction of Interactive Narration of Immersive Film

In immersive images, the promotion of narration depends on the progress of interaction. But how to promote the narration with interaction? The narration formation of the artistic works of immersive images is totally different from those of paintings and 2D films. The interactive narration forms from the interaction with the film conducted by the audiences in the process of experiencing the works. The audiences play a constructive role in the narration construction of the works of immersive interactive images. It can be said that the formation of interactive narration is formed by the medium of the works and the experience process of the audiences. The process of interactive experience is a story building process. In the artistic works of immersive interactive images, “the story is viewed as a way for the people to build a concept world composed of all kinds of processes. People present as different roles when participating in different processes” [4]. We experience the story and enjoy the wonders when watching the film, enriching the emotion of oneself and creating an artistic me in the works.

Inevitably, the design of interactive narration needs to consider many factors such as narrator, medium of interaction, interaction behavior, and narration structure and narration results. And such factors are based on the audiences as the center, that is, to consider the experience results of the audiences with empathy. In terms of the nodes of interaction, the design of narration conditions (such as “what do I have”), process of thinking (how do I choose), role relationship (who am I), operational behavior (what did I do), language feedback (my suggestion) and feedback results (results obtained) is the main considerations.

With the help of the theoretical methods of cognitive narratology, we can understand what kind of thinking tools, processes and activities make it possible for us to construct and understand the narration. Taking the narration itself as a tool for understanding and experiencing, the cognitive narratology uses the methods of cognitive science, including cognitive psychology, social psychology and other disciplines, to study how the audiences organize and understand the having experiences and the relationship with the narrative plots in the process of narration. “Among many forms of the cognitive narratology, one type puts the emphasis on the importance of frameworks (or schemas) and scripts for readers and authors. The framework is a holistic concept that can be used to edit and limit to include and better understand the experience; scripts are recurring action patterns and sequences” [5]. Like other narrative arts, when we enter the time and space of narration of immersive interactive images, the film tells an overall world outlook and forms an overall narration framework, which is the cognitive base by which the audiences experience the story of interactive narration. Based on the narrative framework and the cognitive life experience of the audiences, the audiences know what can be done or what interaction is required. As we come to a museum, we will appreciate the objects we are interested in carefully and pay more attention to the collection strongly recommended by the museum. As shown in Fig. 2.

Fig. 2.
figure 2

The museum space presented by immersive interactive images, and you can further understand the stories behind the collection by activating the hot spots via interaction.

Full size image

3.3 Experience Essence of the Interactive Narration of Immersive Images

The illusion of immersion of immersive images allow the audiences to better participate in the narration in the process of experiencing the works, that is, the form of immersion enhances the sense of reality of the interactive narration experience. Interactive narration, based on immersive images, attempts to achieve the same experience in real life to some extent. The story-telling process is the interaction process between the environment and human behaviorally and emotionally; the developing changes of the story is the message sending and message feedback process between the audiences and the objective environment. The difference between real life and works lies in the inconsistency of time and space. Based on realistic space-time and life experience, the perception of the audiences is put into the virtual story world.

The medium form of images decides the form of interactive narration, and the interactive design of narration experience will be made on the content and process of interaction. The design of interactive narration experience of immersive images is unique for being in the virtual story scenes wholeheartedly. As we interact with things and people face to face in the world of real life, we usually use real tools through five sense organs and arms and legs, rather than control the keyboard and touch screen expressionlessly, to hide the emotion and behavior behind the medium. That is to say, the interactive narration of immersive images would, due to the characteristics of immersive medium, naturally lead the audiences to use the means of natural interaction. Of course, due to the limit of technology implementation possibilities in the actual interactive works of immersive images, natural interaction and gamepad interaction will make the move together.

The interaction process that immersive interaction affects the experience of artistic works is a choice, an exploration, a recognition and an acquisition in terms of audience thinking. How this interaction behavior occurs would be influenced by such factors as the interactive experience scene provided by the image works, the life experience, values and aesthetic standard of the audience and so on. The interaction narration experience of immersive images is a kind of virtual practice of the audiences in the time and space with a specific theme in immersive images, and the choice, judgment, analysis, deduction and reasoning of the audiences in the context for the development of the story plots as well. From the perspective of the play of immersive interactive images, the development of the story and the plot changes are related to the audience’s feedback.

In the traditional narration media, the audiences, listeners or readers learn a story and get to know its existence through the medium, so as to obtain a kind of satisfaction of seeking novelty. However, knowing a thing or seeking novelty is not the only goal of interactive narration. Its goal is to participate in the wonders and even practical activities to create the wonder. As each behavior tells the story of life in the daily life of the audiences, in the immersive interactive images, the artist offers the audiences a specific environment and thematic scene and the audiences obtain a certain narration experience from it.

On the other hand, Hockney said “Watching uses memory, so it’s not the same feeling when looking at strangers and acquaintance” [6]. Moreover, different people have different memory. Even if we stand in the same place, we see a completely different world. Watching is affected by other factors, having never been to this place before and the degree of familiarity of you who have been here before for this place will affect your view here.

The design of interactive narration usually starts the interaction with the audiences from three aspects, namely plot, role and perspective. Because the audiences obtain the possibility and chance to interact with the works in the process of experience, then important evidences to judge whether the works is good or bad are if the design of the interactive narration experience satisfies the audiences, the degree of freedom of interaction provided by the works, the mode of experience of interaction, the presetting of the key points of interaction content and the effect setting of interaction feedback.

In terms of narration, the design elements of immersive interactive images are mainly role, scene and plot. Role interaction mainly refers to the interaction between the audience and the role of the play, or the control of the behavioral expression of the role of the play, or even the replacement of the visual modeling of the role, just as the player’s control of the game in the RPG role-playing game. Scene interaction mainly refers to the design of the story scene and the atmosphere control, to achieve the visual atmosphere and spatial structure that the audience can personalize the story scenes, and thus ultimately realize the personalized expression of the story. Plot interaction usually pursues the personalized plot design, so that the audiences can create their own stories. In the present circumstances, it is often the case that several plot clues are designed in a film to provide the audiences with independent or uncertain plot selection possibility on the preset plot points.

Walking in to an artistic works of immersive interactive image is like entering a story world. However, the story hasn’t happened yet and awaits the audiences and interaction practice to reconstruct the story world. In such a story world reconstructed by the audiences, who does anything to anyone, when and where, and why and in what way, such interactive factors are the conceptual treatment of the time, space and perspective by the maker of the images, enabling the audience to realize the creation and recognition from experience in the immersive video world.

4 Conclusion

Today, the panoramic view and interactive narrative of immersive film have aroused everyone’s interest, the visual experience are shocking us. However, people who have hundreds of years of visual experience in movie have more expectations in the immersive film currently in its infancy. These expectations are just the motivation for immersive film innovation.

In the artistic works of immersive interactive images, Monika Fludernik, a theorist of cognitive narration stands for a generalized narrative theory which can explain the narrative way to construct the story jointly by immersive interactive images and the audience. In his opinion, “the basis of narration does not lie in the narrator or the event sequence, but the so-called experience, namely our practical experience. The key to narration lies in those characters who act and think. In this way, the cognitive framework based on our embodiedness makes the action and thinking become the activities of great importance.” [7] These actions and thinking are just the innovation of the artistic experience of immersive interactive image. The immersive images give people a way to observe and experience the world, which is in line with the human kind nature. The interaction behavior of the audiences in the process of experiencing the works reconstructs the story world of immersive images. People get a more natural narrative experience in the new way to observe and display the world, namely immersive interactive images.