Interactive environments for music and multimedia

Abstract.

Multimodal Environments (MEs) are systems capable of establishing creative, multimodal user interaction by exhibiting real-time adaptive behaviour. In a typical scenario, one or more users are immersed in an environment allowing them to communicate by means of full-body movement, singing or playing. Users get feedback from the environment in real time in terms of sound, music, visual media, and actuators, i.e. movement of semi-autonomous mobile systems including mobile scenography, on-stage robots behaving as actors or players, possibly equipped with music and multimedia output. MEs are therefore a sort of extension of augmented reality environments. From another viewpoint, an ME can be seen as a sort of prolongation of the human mind and senses. From an artificial intelligence perspective, an ME consists of a population of physical and as software agents capable of changing their reactions and their social interaction over time. For example, a gesture of the user(s) can mean different things in different situations, and can produce changes in the agents populating the ME. The paradigm adopted for movement recognition is that of a human observer of the dance, where the focus of attention changes according to the evolution of the dance itself and of the music produced. MEs are therefore agents able to observe the user, extract “gesture gestalts”, and change their state, including artificial emotions, over time. MEs open new niches of application, many still to be discovered, including music, dance, theatre, interactive arts, entertainment, interactive exhibitions and museal installations, information atelier, edutainment, training, industrial applications and cognitive rehabilitation (e.g. for autism). The environment can be a theatre, a museum, a discotheque, a school classroom, a rehabilitation centre for patients with a variety of sensory/motor and cognitive impairments, etc. The ME concept generalizes the bio-feedback methods which already have found widespread applications. The paper introduces MEs, then a flexible ME architecture, with a special focus on the modeling of the emotional component of the agents forming an ME. Description of four applications we recently developed, currently used in several real testbeds, conclude the paper.