Media Art Environment Geist: Integrating Traditional Painting into 3D AR Storytelling Scenario

Abstract

This paper describes an approach at ZGDV Darmstadt to combine experimental use of traditional means (paintings) and VR/AR technology for Storytelling based edutainment applications. The illusion of light and depth banned to paper or canvas was the mission of generations during history of art, when paintings and drawings were the only media to transfer the imagination of space and atmosphere to the beholder.

In the context of the edutainment application Geist providing a Storytelling based learning and gaming environment enabling users to become part of a story and interactively explore history of Heidelberg, our goal was to elaborate how painted images could serve as an effective surrogate of reality within a computer-generated 3D-world: Within the Geist demonstrator there are three locations where 3D-objects and virtual characters are surrounded by a texture that simulates a natural environment. It containes men, buildings, vegetation but also important phenomenons based on light, perspective and colour. So the virtual reality is enlarged by a second level of visual reality based on paintings. In order to embrace this sort of approach, it is necessary both to reflect reality and represent it in an artificial way. Because atmosphere in this case involves not only the reproduction and transforming of perception our eyes are used to but to create an adequate expression and prevailing mood: On the three locations the user is lead into the expectation of imminent occurrences that struck the town of Heidelberg during the 30 years war. The preliminary results allow to analyze the functional, cognitive and aesthetic nature of this crossbreed and to draw further conclusions concerning further developments. The first part of the paper provides an overview of the Geist project and the resulting demonstrator as reference example for an edutainment application. Afterwords the two major technologies Storytelling enhanced by VR/AR technology and traditional painting are introduced before the integration process of these two antithetic approaches is describes. Finally a short summary points out first results of this integrated approach and discusses further development directions.