Keywords

1 Introduction

The history of film is essentially one of the parallel development of film technology and art. Every leap in film art is inseparable from the breakthrough of technology, i.e. high-resolution images and sounds, as well as the research and development of modern digital technology. Based on such continuous evolvement and innovation of film technology, this form of art has grown from the black-and-white and silent one to the color and sound one. What change will happen to films in the upcoming years? Filmmakers have raised varied opinions based on current audio-visual technologies. The concept of the third-generation films proposed by Beijing Film Academy seems to herald the future model of films.

Among the three keywords of the third-generation films, interactivity, along with digitalization and stereoscopy, is the nucleus. Interactivity is also the focus of VR films that we are mainly studying. The third-generation films, a byproduct of the development of computer digital technology in the new era, have three prominent features, that is, digitalization, interactivity and stereoscopy. Digitalization means that the films involve digital photography, production, editing and effects; interactivity indicates the viewers’ participation and controllability i.e. views can control the plots and take part in the group interactive games during the film; stereoscopy means to focus on the development of stereoscopic video content and the stereoscopic projection where crystal-clear stereoscopic films are presented to the audience through the stereoscopic technology which is at the cutting edge globally.

Interaction is the uniqueness of third-generation films and has given rise to more features. The interactive control technology has brought about subversive changes to film viewing and aesthetics. They mainly involve three respects – narrative modes, story development and film viewing experience. Specifically, the linear narration where audience are in a passive position has been transformed into a proactive interactive mode; the single storyline has developed into a multidimensional one while the predictable ending has been extended to a number of unpredictable ones; in terms of film viewing experience, the individual passive viewing has been escalated to group interaction by which the audience lead the story. From the perspective of film creation and aesthetics, the most prominent breakthrough of third-generation films is that the free creation space of directors is completely opened for the audience’s free choices. This liberates the audience’s mind and vision and allows them to have unprecedentedly free experience and perception of the dream world.

2 History of VR Technology

VR is the word most talked about in media, entertainment, culture and other industries. This technology surged with Facebook’s acquisition of Oculus in 2017 followed by the presence of HTC and SONY. A number of industries, including game, film, education, tourism and healthcare, expect to seek new opportunities through VR technology. All those phenomena occur in the context of the functional expansion of VR in recent years. New manufacturing processes and display modes have appeared with the support of computer science and the Internet and are likely to be commercialized. Particularly, the design and release of the wearable device for panoramic viewing, i.e. VR glasses, have given rise to a new media channel different from film, television and smartphone. The combination of VR technology and the panoramic display will stimulate people’s unlimited imagination of artistic creation with VR.

Utilizing computer technology, VR can generate a digital space highly similar to the real environment, the objects in which can be felt by users in real time without limitation. In spite of the debates on the specific approaches to and time of its actualization, the augmented experience brought about by such space is represented by the viewing experience that is completely immersive and interactive. Thanks to wearable devices and mobile terminals, the audience can be immersed or even involved in the stories as a role therein, trying different stories and endings. The audience is empowered with more proactive control and more realistic experience of the film’s progress. The linguistic form and thinking of film creation are thus reformed. Such a film where the audience can interact and influence the progress is essentially a game with a cinematic structure and dramatic property, a “game film”. From this point of view, developing animation films into VR ones can exert unique advantages, as animations and post-production can end up with the best effect with the most fluent and strongest sense of immersion. The reason for this is that they are not produced in a live-action manner but a computer-aided one, allowing producers not to bother camera movement, lighting and other technical issues. However, just as any other milestone revolution in the film history, the story contents, film languages, narrative methods and the styles of presentation necessitate a great adjustment. Change in any element can distinguish an animation film from others, giving rise to a typical film genre – genre animation films. Films with clear styles are better memorized by the audience with an enhanced charm of expression.

After the continuous development and popularization, the term VR technology has penetrated into military, medicine, education, media, games and other various industries and fields. Its application in the field of film, television and animation is no exception, as reflected by the constant emergence of animated films produced based on VR technology. The massive sales of VR head-mounted devices and the promotion of offline experience stores have impelled VR animation to become a new type of film familiar to the public.

As the audience have extreme freedom to choose, the film must take into account every angle that the audience can reach. Only in this way can the audience have the most realistic experience. VR specifically requires the director to adjust the overall style based on a game-designing way of thinking, so as to ensure the story is interpreted in line with the modern aesthetics, hi-tech sound and image design, scene transfer and realism. Two points should be paid attention to – the sense of impact and recollection. As to the first point, the perceptive impact is mainly achieved via audio-visual elements while the spiritual one via the story and progress. Recollection, on the other hand, arises from the integration of the thoughts of the character and the audience realized by driving the audience to enter and be integrated into the spiritual status of the character. Whereas entering is a gradual process of environment building, suspense and foreshadowing, as well as story development, integrating means to integrate the audience with the character and his thoughts via narrative methods and character building.

As the animation technology progresses, animation creation has gradually separated the performance of the animated characters from the audience, who could only enjoy the animations recorded on films without taking part in the creation. Thus they could not experience the excitement of participation. In contrast, VR technology focuses on immersion and interactivity, two crucial features for the combination of the VR technology and animated films. Nevertheless, many creators are restricted to the immersive feature while ignoring or avoiding interactivity. In a certain sense, the birth of VR films, for the first time in history, pushes down the fourth wall of films and brings the audience directly into the scene so that the audience may enter the film with the identity and perspective of a character. In other words, the viewing experience of VR animated films will be greatly impacted if the audience can only be immersed through watching and experiencing, totally unable to make a difference or participate in the interaction and thus deprived of the sense of presence. Centering on storytelling, creators should design carefully, starting with all elements of the film and properly conveying the story-based interactive design to the audience, so as to enable them to interact with the story happening in the virtual world with a real sense of presence and participation. Only in this way can VR animation really drive the audience into a new dimension of film viewing. Animated films based on the VR technology have emerged constantly in large. The massive sales of VR head-mounted devices and the promotion of offline experience stores have impelled VR animations to become a new type of film familiar to the public.

The animation technology so far has gradually separated the performance of characters and audience in creation. The audience may only watch the images recorded on films and are deprived of the right to participate in the creation and the joy of participation. Immersion and interaction, on the other hand, happen to be the two most prominent features of VR technology and will exert great importance in the combination of VR and animated films. However, many filmmakers tend to focus on immersion while ignoring or avoiding interaction. To a certain extent, VR films, for the first time, have torn down the fourth wall and push the audience right into the scene, make them enter the film with the identity and perspective of a character. What VR animations present to the audience are a point of view without limitation and an observation angle that can be randomly moved. By such means, viewing is granted with a new connotation – narration. The narration has become a process where the audience are no longer bystanders but participants. Progressing the story and set up guiding information thus are the most important part of VR audio-visual language. The guidance is usually completed via sounds, light, movement of objects, or key characters to tell the story or express the emotions. However, limiting to viewing and immersive experience without influencing and interacting with the story avoids the sense of presence and compromises the experience of VR animations. Filmmakers need to elaborate their creations by appropriately focusing on all elements of the film and delivering to the audience the interactive design reasonably based on the story. On the basis of sound storytelling, the audience need to interact with the story happening in the virtual world. Only with the real sense of presence and participation acquired therefrom can VR animations drive the audience to a new dimension of film viewing. There have been abundant animated films based on VR technology and these VR animations, as a new genre, have been popular to the public thanks to the massive sales of VR headsets and the promotion of experience stores. Traditional filmmaking process actually leaves limited space for the interactive design of VR videos and animations. In an era of advanced real-time rendering, optimal options of VR animation production lie in the combination of game engines and traditional film making processes. Companies such as Unity and Unreal have produced real-time rendered VR films. Despite the fact that the quality is still far from line rendering, low costs and timeliness are what help game engines prevail (see Fig. 1). These two advantages are even greater in terms of VR animations. In the works of Oculus Story, for instance, the audience may experience the stories from every angle thanks to the utilization of game engines. They directly go through the fourth wall and enter the world of the characters. The conventional offline rendering is three degree of freedom behind this approach. Plus, the production of panoramic animations is known for its difficulty and the hairball theorem strongly impacts the immersion effect. Compared to this, the real-time rendering of game engines perform much better for it completely adopts the real-time computing of GPU and the rendering quality is getting closer to that of CPU. Attracted by this, a number of companies are researching how to apply game engine computing to the visual effect of animations. In addition, the powerful interaction of game engines exerts unmeasurable possibility for the interactive design of future VR animations.

Fig. 1.
figure 1

(Source: www.soomal.com)

Experience schematic of VR game “The Lab”

Full size image

VR interactive animations, which were born from the step-by-step integration of traditional films and animations, are produced in close connection with the development of the film industry. Since its birth, animation art has been in an exploratory period in terms of the theme connotation, the means of expression, the form of art and the medium of communication, in particular, the creative medium of animations. An early form of animations is a pottery bowl found in Shahr-i-Sokhta, Iran, which was painted with five frames of images, allowing people to see a goat jumping to a pear tree when rotating the bowl. This dynamic scene formed by the interaction between men and a carrier can be deemed as the origin of interaction-driven animations. Digital technology has been integrated into animation creation along with the development of computer science, driving animations to evolve from a paperless flat stage through digital stereoscopy and VR animation.

3 Exploration of the Interactive Modes in the Combination of VR and Animations

Immersion and interactivity, the two most prominent features of VR technology, are still important for judging the quality of a VR animation that combines VR technology with traditional animation techniques. However, given the uneven level of studios on the market that produce VR animated films and the lack of detailed and standardized specification requirements for films and television as in the traditional film industry, the quality of VR animations currently viewed and experienced by the public varies greatly, and so do the immersive and interactive experience, giving rise to countless problems, such as screen-door effect, feeling of vertigo, confusion and solitude, which greatly compromise the audience’s experience in watching. Some of the problems result from hardware problems, yet most of them are attributed to the defects in interactive design. Some VR animations, which are essentially a spherical panoramic video, where the only thing the audience can do is to passively follow the already-designed animation. By rigidly copying the traditional film and television production process, these animations do not take into account users’ need to experience the differences between VR images and traditional plain ones, let alone the interactive design. The audiences tend to be dizzy watching these animations, not knowing what they are about, which is really a torment. Some other ones make the audience confused, as they often miss a lot of plots due to their attention is far from sufficient to catch everything. The reason is that wearing the VR glasses, the audience’s attention is distracted to a 360° panoramic environment, where they cannot concentrate on the story, for the instinct of human beings as animals decide that they need to first figure out where they are and what is around them by looking around. If the story is presented before the audience get used to the environment, the audience will probably miss a lot, and fail to keep up with the story development even though they finally are focused. To avoid these elementary mistakes, we must scientifically observe the feedback and needs of the audience watching VR videos, from physiology to psychology. This indicates the necessity of studying the interactive design of VR animation, only by which can we avoid mistakes and bring excellent interactive experience, better serving the audience and telling stories.

Without a doubt, the interactive design of VR animations is more than simply slowing down the pace to give the audience a longer reaction time. A series of issues should be considered, such as the role and the perspective of the audience during the viewing process, the elements of the films with which the audience can interact, whether the audience should interact with such elements, when such interactions should take place, and the appropriate frequency of interaction. If these issues are well handled, the VR animated film will not only tell the story well but also make the audience experience the charm of immersion and interaction. Otherwise, the film may even hardly convey the story. Thus the significance of interactive design for VR animations is seen.

The reform of film viewing conceived in third-generation films is specifically about the transformation from individual passive viewing to group interactive games where the audience determine the story development. In this concept, the audience experience group interactive games in cinemas as a group entertainment distinguished from the conventional films. At the entrance of cinemas, each viewer receives a terminal device of interactive control. The audience are divided into three or more groups and voluntarily become interactive controllers of the group game at the gaming points of the film, each group unites to interact or compete with the others. Such a fun approach will form a lively, happy and united atmosphere and help the audience get closer to each other. The group with the strongest cohesion will be entitled to choose the story direction as wished, directly impact the ending. Game modes are also applied – the selections in RPG games. The interactive feature of RPG, i.e. players control the hero to lead the plots and change the story, is experienced by the viewer and thus become another characteristic of interactive VR films with game playing.

The combination of VR and animation through game technology can provide the audience with a more realistic experience. However, this idea just focuses on experience, while the deeper interaction with the missions in the film is the key to the success of VR animation. It is believed that the following aspects of VR interaction are worth exploring:

3.1 Multi-line Development of the Story

Just like men doing multiple choice questions in their lives where each choice will have an impact on the final outcomes, various endings in games may be introduced to the story organization as an analogy of the butterfly effect. In the story development of interactive animation, the audience can act as an external force to drive the story to the ending as they wish. That is to say, the audience are prompted to make several rounds of selections in order to see the ending they expect. Instead of creating the story in a linear order, different nodes of the story can be set in the animation script, leading to distinguished endings. Currently, the “mind map” is used in creating the script of an animated story as a conceiving approach (see Fig. 2), i.e. the characters, events and other elements of a certain story are taken as the starting point of creativity to trigger different storylines. The steps in developing a map generally include establishing a central idea, initiating a structural map, integrating and concatenating the materials, and clarifying the narrative streams and sequence. This creative way of mapping emphasizes a professional training of divergent thinking, stitching together irrelevant characters, scenes and story elements in the animated story to form the storylines of different fractals before creating an animation with interactive narration.

Fig. 2.
figure 2

The sketching approach using “mind map” (starting from elements such as characters and events of a story, varied storylines will be triggered).

Full size image

In the past, the narration of films was usually dominated by the director’s thoughts and followed a certain fixed and linear narrative approach, allowing the audience only to passively accept the contents of the film. To a large extent, the audience watched completely passively and learned the events and the ending that the director had organized. The audience may either agree or disagree with some of the contents, yet they could only quietly watch the contents of the film already planned by the director. This film narrative mode is typically linear and fixed, with the ending even completely predictable. Now, however, the audience can experience completely different interactive contents and savor different lives through the lifestyles they choose during multiple views of one same film. It can be seen that the divergent narrative method, which differs from the traditional films where the director monopolizes the narrative style, allows the audience to have a freer and richer life experience.

The interactive technology has revolutionized the conventional narrative model of films. Owing to the multiple versions of the story underlying in the viewing process, the narrative approach of the film becomes opener and the ending more unpredictable. The storyline also has developed from a single and predictable one into a number of completely unpredictable plots and endings. Film developers can set a few different storylines to make the ending more unpredictable. Based on the various story directions selected by the audience, different endings will occur.

3.2 Enhancement of Gameplay

Traditional films, television and animations are mostly displayed in a dark cinema environment to help the audience enter the aesthetic status of the animation from daily life. With their attention focused on the storyline of the animation, the audience are completely immersed in the animation story, experiencing the ups and downs of the story, the joys and sorrows of the characters. Since the advent of animations, the audience has been eager to experience the animation story, imagining that they are the protagonists of the story who overcome the difficulties and harvest successes. Unfortunately, traditional animated film and television programs can do nothing in engaging the audience in animation creation to help them truly experience the emotions of animated characters. However, interactive animations as a new medium allow the audience to determine the direction of the story to a certain extent via interaction and thereby realize the perfect ending in their minds as much as possible. Under the intervention of these interactive means, the audience can gain the aesthetic pleasure of the animated story on the one hand and the joy of a game player on the other hand. Therefore, in addition to cultivating the traditional sense of aesthetics, professionals of VR animation creation should put further efforts in integrating reality and virtual world, sensory experience, creation and participation.

In the wake of the emergence of smartphones and other mobile smart terminals, and the integration of human somatosensory devices, the medium of animation creation has spanned from traditional to interactive practical dimensions. Today, VR technology is broadly used in both films and games, more abundantly in games. VR games such as Fancy Skiing, Mercenary and Furballs have been very popular among players. Steam recently unveiled the Top Selling VR Games of the year. Five games, which are either VR monopoly and or VR support, were nominated, namely The Elder Scrolls V: Skyrim VR, VRChat, Beat Saber, Fallout4 VR and SUPERHOT VR. It is reported that the authority of Dubai has launched its first VR training center that is equipped with high-end equipment and tools for VR technology. With its software developed by the GIS Center team in Dubai, the training center offers training courses in a wide range of areas such as urban planning, land surveying, and construction supervision. In the future, more projects are expected to be added to cover other relevant professional fields. The training center will be accessible for various departments and agencies.

The innovation of VR contents and their industry application have brought new ideas to the development of the cultural industry, for they enhance people’s preference of the industry while enhancing the entertainment feature. In the field of games, most VR games are currently stuck in the DEMO phase as limited by hardware, venue, technology and other conditions. To highlight the advantages of VR, the development idea is to emphasize the experience. Most contents are developed in clips, ranging from car racing, mountain climbing to natural exploration. Though not mature enough, these clips give users a strong sense of impact. The latest trend is to make real complete VR games. EVE Valkyrie, a space combat shooting game developed by the Icelandic developer CCP, has attracted much attention thanks to its high quality and complete game mode and map. What the game does is to ensure the 100% participation by players by allowing them to talk, fight and kill, and even choose to change the direction of the story.

In the field of VR animated films, VR technology also reinforces the entertainment of the story. For instance, when there is a battle scene, the audience can control the game character to fire an arm, which makes the original animation story more entertaining. Oculus Story Studio has produced several animated shorts including Henry, a story of a hedgehog. Great breakthroughs have also been made in immersive documentaries, such as Waves of Grace, which describes the cruel reality in Liberia. When the volunteers in protective suits are putting the corpses of Ebola patients into the graves, the audience will feel they are next to the grave, surrounded by more graves. Such an immersive feeling is more touching than any news report to the audience. Discovery, BBC and so forth have also attempted virtual natural documentaries in the first-person perspective. In China, Caixin Media has launched its latest VR documentary Kindergarten in the Mountain Village; while Sightpano produced Blind Circle.

In addition, VR technology has penetrated into the news service industry as well. It has been found in thorough researches and explorations that VR technology is suitable for producing news in the form of a panoramic video for such form and its unique point of view can bring about strong immersion. New York Times, BBC, ABC News, Associated Press and Chinese media like Xinhua News, People’s Daily, Caixin Media and Sina have all made their attempts and achieved satisfactory results.

When appreciating a VR animation, the audience is eager to participate in the creation and to seek a realistic experience in the virtual story. This requires the animation works to give feedback in time after the interaction, that is, the interactive effect, which can be a cool visual animation, a voice comment, and a tactile experience such as shakes or vibrations. This kind of interactive effect is more applied in new media animation games. For instance, the birds in Angry Birds use the huge inertia of the slingshot to strike the castle and express their rage at the King Pig’s stealing of eggs in an interactive effect triggered by the finger drawing the slingshot. Such effect gives the audience a pleasant feeling of interaction. In another game Talking Tom Cat, the player can interact with Tom Cat by directly touching the screen. The cat may meow, or break wind and make other sounds with the corresponding button pressed. These interactions will interest the audience with a realistic feeling of playing with a cat.

4 Exploration of the Communication Models of VR Interactive Animations

Despite the rapid development of VR technology, it is usually costly to produce an interactive animation. In view of the need of multiple story versions for the audience to freely choose in interaction, greater input of manpower, material and financial resources is demanded than traditional animations. What’s more, in addition to directors, key animators and post-production staff required in traditional animations, experts in interactive design and technology are required in the production team. This implies that talents of interdisciplinary creative applications are called for in animation creation, the function division in the creation mode should break through that in traditional animations which include character design, scene design and key animation, and more emphasis should be laid on the learning and innovation of interactive technology and forms. In the meantime, unlike the previous animations whose interactions are designed based on tactile sensings such as multi-touch and gravity-induced tilt, interactive designs in the future will apply more motion sensing and limb motion techniques.

Hence it can be seen that interactive animations undoubtedly have much higher production costs and at the same time a wider coverage of technologies compared with traditional animations. Given the fact that many excellent interactive animation projects have been suspended due to fund insufficiency, it is necessary for film designer to consider the communication model that the film can rely on. For example, cloud transmission and other Internet viewing modes provide convenience for the audience to download the resources from the cloud at a high rate; costumes and props available for the characters can be designed in the films as in games; advertising may also be placed appropriately in the films. When watching the film, the audience can click and buy their favorite props or costumes. Surely there are other communication models worth exploring. Such a combination of contents and communication models is a prerequisite for the sustainability of the costly VR interactions, and represents a sound exploration.

The emergence and development of every new medium constantly create the conditions and opportunities for the progression of animation art, while the animation art itself is also constantly creating miracles in the era of high-tech new media. In the future, new media animations will create dynamic images in an artificial manner based on digital technology and Internet technology, embodying the fusion of science and art. New media animation as a major involves art and industry and other disciplines, inclusive of the intersection and integration of drama and film science, art, design, computer graphics, interactive technology and media communication. It is the special professional background that makes possible the diversified new media animation works, such as game animations, interactive films, interactive displays and interactive picture books.

The interactive design of some children’s books provides a good experience for the readers. For example, the electronic picture book The Fantastic Flying Books allows the reader to use gravity sensing to control Mr. Morris’ flying direction in the sea of books by tilting the iPad; Dandelion allows the reader to simulate the physical characteristics in the natural life utilizing the gravity sensing system of the smart mobile terminal, such as blowing against the microphone to scatter the dandelion on the screen. Such an interactive experience can effectively spur children’s interest and enhance the realism of the experience. The “Arts Future Lab” launched in January 2016 is a novel art platform built by the “Art Plus” innovative model. The laboratory is aimed to explore the future forms of art with big data and cloud computing as the core, and intelligent computing, intelligent interaction, smart devices and smart applications as the means. This is also the direction of new media animations in the future – to introduce more interactive devices and forms that trigger more sensual experience to create a kind of new media animation where the audience can immerse themselves, control the story development and go through a variety of experiences and feelings of the characters.

In the future, the production of VR interactive animation films should not only address the audience experience but also handle commercial advertising, in a bid to solve the problem of high production costs and limited channels of capital recovery. Only by doing so will more outstanding, groundbreaking and revolutionary interactive films emerge.

5 Prospects of VR Interactive Animations

Apart from the escalated immersion via hardware approaches, VR technology is destined to cultivate the artistic experience of animations that is completely different from traditional ones. VR images are capable of carrying complete and meaty narrations. Only those bearing deep artistic meanings are qualified for video arts, otherwise, they are only “special effect films”. In this sense, production devices, software and film players are among the aspects to be improved. Given the multiple storylines of VR films, a video database for HD films which has storage over 10T and enables videos not lower than 1080P is needed. In the meantime, a Next Generation video interaction engine will be introduced and developed based on the latest research achievements of digital image processing and an interactive control system of video stories on an interaction engine platform for high-resolution and high-detail Next Generation game videos. This engine requires the close collaboration of a CPU with a high rate, a GPU with an image coprocessor and the HD video interaction engine system. The audience will experience the high-resolution VR films based on the advanced automatic separation and display system of stereo images from nVidia.

The technical evolvement promotes the transformation of interactive narration, to which there are a lot of approaches. Multiple storylines are one of them and have been mentioned a lot. This approach itself, however, is essentially the same with traditional narrative modes of films. In addition, multiple storylines are less feasible in the case of live action production. On the other hand, if the digital virtual production and real-time rendering by digital engines are applied, there will be abundant possibilities of interactive narration. The design experience of game interaction can be transplanted to VR films, helping realize deeper interactive narration. For example, the choice of story volume, the control of narration rhythm and the selection of ending all establish a personalized 3-dimensional interactive narration and the story is truly handed over to the audience. The audience will be granted with free choices and the right to think independently, which are the real charm of VR films.

6 Conclusion

In addition to representing innovations in technology, the innovative development of VR also provides the cultural industry with a new form of media and a new way of development thinking. In the future, VR will further evolve from 3Is (immersion, interaction, imagination) to 4Is (the 3Is plus “intelligence”). It is conceivable that the cultural industry will develop following such a trend, leading to a future worth looking forward to.