Keywords

1 Introduction

Ubicomp as an interaction paradigm and technological platform has been developed with the intention to allow computers to move away from the center of attention into the periphery. Inserted into everyday objects and blending with the environment, they would conceptually and perceptually “disappear”. Even the original formulation of ambient displays and calm computing as expressions of ubicomp interfaces has been recently criticized by Hansen as a too functional, “downright instrumental” perspective [1].

When placing displays in the periphery, the assumption remains that ambient displays like dangling strings and slowly changing colored lights can be re-centered and interpreted by an observer, thereby moving from the periphery back to the center of attention. Hansen in contrast argues for an autonomy of the peripheral, making ubicomp environments inherently affective and atmospheric [2]: “We must conceptualize the coupling of human and techniques beyond the figure of the ‘technical object.’ In the wake of computational technologies that distribute sensibility beyond consciousness, the correlation between human-implicating individuation and techniques has moved beyond what we might think of as its objective stage […] and has entered a properly processual stage in which techniques directly intensifies sub-perceptual dimensions of human experience.” Instead of a technical mediation of perception, today’s concern is “the more indirect technical mediation of an environmental sensibility.”

In our work we set out to explore the atmospheric potential and the affective connection between humans and their instrumented environments and to develop corresponding artistic design strategies. We have chosen to follow an artistic research approach [3], as it enables us to explore the atmospheric effects and consequences of the use of responsive environments from a critical perspective, beyond pure application and usefulness.

2 Related Work

Substantial research on the close coupling of people and their interactions with a generated, responsive environment is conducted in the area of computer games. Games are highly engaging and are driven in their appearance and difficulty by the interaction with a player. A common approach is to measure physiological signals to assess a player’s emotional state and to use these measurements to control game difficulty, which is an easily accessible parameter of the environment in most games [4]. Parnandi et al. [5] extend this idea by introducing control theory to model the interaction between human physiology and game difficulty during game play. They use electro dermal activity (EDA) as a physiological correlate of arousal and couple the player’s response to control car speed, road visibility, and steering jitter in a racing game to manipulate difficulty. A similar approach has been proposed for training in virtual environments by Wu et al. [6]. Here the aim is to identify and maintain a state of “optimal arousal”, measured by psycho-physiological responses, in a military training task.

We think that close couplings between people and a responsive space in closed feedback loops have more in common with the psycho-cognitive processes found in the rubber hand illusion and other, technically induced out of body experiences [7, 8]. For this we use the metaphor of the reactive environment as an “extended skin” [9].

From the point of view of an artistic approach to the development of responsive spaces the architectural works of Schnädelbach and Beesley are close to our work. Schnädelbach et al. [10] aim to connect physiological measurements (breathing and EDA) and the fabric of a tent-like structure (controling light, projection and sound) to externalize a person’s physiological data in an immersive and visceral way. In later work [11] they use this adaptive architecture to support Yoga practices. Beesley’s Protocell Mesh from the Holozoic Series [12] represents a responsive architecture that uses light and moving, biomorphic structures to stimulate and engage the visitor to evoke a “[…] humane response [to] the contemporary condition of ecology.”

3 Technological Approach

Ubiquitous Computing [13] is an interaction paradigm and technological approach where digital sensors and actuators are incorporated into physical space by introducing a networked layer of intelligence into things. Compared to Virtual Reality which immerses a visitor in a digitally constructed synthetic world, isolating her from the physical surroundings, ubicomp seamlessly blends virtual sensing and control into the real world [14]: “…the singularity of “the interface” explodes into a multiplicity of more or less closely aligned, dynamically configured moments of encounters within sociomaterial configurations, objectified as persons and machines.”

Biofeedback technology and corresponding body worn sensors measure personal body functions and physiological parameters in sports and daily life, with the aim to improve health and perhaps even more importantly to increase mental and physical performance. Generally spoken, biofeedback was developed to gain more awareness and control of physiological functions influenced by thoughts, emotions and behavior. In our project we measure biophysical signals for breathing (in-point, out-point, regularity), heart activity (beats and rate) and body movement of visitors in a responsive space and record primary, preconscious reactions and emotions. These parameters enable a spectrum of control: While body motion is under direct and voluntary control, breathing is much less direct and not commonly used for explicit interactive control. However, humans have some degree of voluntary control over their breathing [15]. In contrast, while heart rate is coupled to body motion and deep breathing, we have no awareness and no common experience of voluntarily controlling our heart beat.

We combine ubicomp and biofeedback technologies to immediately and affectively connect an artistic environment with a person being related to it. Thereby we consider both sides – the human and the technological system – as equal actors [16] that are connected and interplay in real-time in a human-in-the-loop system (Fig. 1). Although the technical system has no independent intelligence, its physical responses follow dynamic mappings to physiologic human reactions, manifested through light, sound and wind. These responses are expressive and can cause new physiologic reactions, creating the human-machine interdependency.

Fig. 1.
figure 1

Schematic of the cybernetic human-in-the-loop system

Full size image

The technical system is composed of a connected sequence of three modules: input sensors, signal processing and mapping, and output media or actuators. The intervening human visitor is the most volatile element of this system, closing the feedback loop between the sensors and actuators. She is affected by the reactive surrounding media and reciprocally influences the parameters of it by real-time processed biofeedback. In other words: The measuring of heartbeat, respiration and movement of the human permits to capture psycho-physiological reactions to external stimuli which emerge when being exposed to the artistically staged environment. Mutually, the environment changes its appearance according to the received human biofeedback data.

4 Designing the Place

Benyon [17] discusses aspects of Human-Computer Interaction, interaction design and user experience in the age of ubiquitous computing. In the examination of physical space and place he quotes systematic proposals of experience of place by Lentini and Decortis, Gustafson, Relph. Relph [18] describes the identity of environments in terms of three components, which prove to be decisive for the following description of our project:

  1. 1.

    the place’s physical setting;

  2. 2.

    its activities, situations and events (afforded by the place); and

  3. 3.

    the meanings created through people’s experiences and intentions in regard to that place.

Relph’s classification has been often criticized for ignoring historical, cultural, social, educational and individual qualities and circumstances. For the focus of our approach we excluded most of those aspects as well and tried to overcome the complexity of the human world for our design by setting up an artistic scenario in a fully controllable studio situation with limited association potential and by carefully selecting the evaluation participants.

4.1 The Physical Setting

Since our project is an artistic research project and not a media art project, we did not begin development with a clearly defined script nor an intention of expression or statement. Instead, drawing inspiration from our own initial sketches and a large image collection that was categorized by concepts, aesthetics and structures, we followed an iterative process of conception-investigation-implementation-evaluation-elimination/approval. Step by step we built a large environment of about 4 m wide, 8 m long and 5 m high. It is composed of a single type of white, semi-transparent, non-woven textile with interesting texture and tactility, hanging loosely from the ceiling, shaping a cocoon-like isolated space resembling organic natural structures. The resulting homogeneous and very reduced setting is instrumented with spatial, parametrically controlled light (brightness, color, duration, number, spatial composition), spatialized sound (type, volume, position, duration) and controlled air stream (intensity, duration, position, direction) that causes the textiles to move softly. These control channels constitute the dynamic atmospheric parameters of the installation.

Accelerometers, an elastic respiratory sensor and a photoplethysmograph (for measuring heart rate) are used as input modalities. Since the dynamic atmospheric parameters of the installation change directly according to the physiological reactions of the participants, it’s appearance and behavior is non-linear in time and does not follow a predefined choreography. Table 1 gives an overview of the fully controllable sensor-actuator mapping design of the installation. Different types of information can be processed and interpreted from one single biofeedback signal and different parameters of different actuators can change accordingly. To create an atmosphere that can be perceived as emotionally responsive, we developed a complex setting of composed couplings starting from familiar patterns and modes of media design of the physical world, western media literacy or studies as for example color psychology. We had to take decisions to arising questions like: Does the air stream feel more personal if it increases when breathing in or out? Is it accurate to represent breathing in (oxygen-rich air) with colorful bright light and breathing out (carbon dioxide rich air) with a dull, desaturated light situation? Is the heartbeat sound – well established in cinematic culture – too platitudinous to represent heartbeat, does it create the wrong associations or is it a necessary hint for the participants to detect the human-computer connectivity? Is the color blue the accurate representation for calmness and red for an agitated state of mind? To further investigate these design issues we will make an evaluation with experienced designers, who will get direct control and influence on the design of the system, but for that we have to first develop a suitable and effective process with techniques to evaluate the decisions and implementations.

Table 1. Sensor-actuator mappings
Full size table

4.2 Afforded Activities

The resulting homogeneous and abstract, yet responsively mediated space, removed participants from their familiar context, whereby they became very attentive of their surroundings. After initial attempts to include cushions or other elements to be explored, we decided to not implement any distractive features of affordance [19] as they appeared to imply behaviors and tasks that appeared too suggestive and obvious. This cognitive reduction did not only make the visitors attentive but also self-aware of their situation in the environment. We assumed that the minimalist design approach would make people become more aware of their affects and the reactivity of the surrounding space.

4.3 Meanings Created Through People’s Experiences

One of the initial motivations to develop the present project was the challenge to investigate the design of an environment that causes bodily identification. We asked ourselves: is it possible to build a responsive environment that makes an immersed person interpret it as part of their proper body, as an extended organ or a second skin. Or the other way round: Does the involved person to some extend accept the own identity as part of the surrounding system? Malafouris [20] elaborates the difficulty of drawing the boundaries between a person and the environment by using the famous example of the blind man’s cane: Where does the blind man’s self end and the world begin? Gins and Arakawa [21] intensely focused on the relation organism-person-environment and shaped the terms sited awareness and the situated body and proposed to architecturally build the questions themselves to gain insight.

Since we did not give the visitors any narrative to follow nor tasks to complete they came up with changing, ambivalent associations, started to create meanings and to set up their own experiments and challenges. The visitors intend to create mental models [22] to interpret and manage their surroundings.

5 Evaluation

5.1 Procedure and Interviews

We have conducted a series of experiments with a total of 17 participants (9 male, 8 female, ages between 22 and 54 years). Before they enter the installation setting, we briefly inform participants about the procedure without explaining technical details or the goal of the evaluation. We ask them to take off their shoes and to put on socks which improves the sense of tactility and serves as a “rite de passage”. They put on an elastic chest belt with the breathing sensor, the motion sensor and the wireless transponder. A photoplethysmograph is attached to the index finger of the right hand.

We explained to the participants that they should enter the room and freely behave as it suits them, that there was nothing they had to achieve or that could go wrong. The exposure before participants would leave the space again lasted between 7 and 12 min and was video recorded. We also recorded the measured bio-physiological sensor data. The recordings are merged with the video and audio track in the Evaluation Viewer (Fig. 2), allowing us to analyze the recorded behavior together with the appearance and sound of the installation at a glance.

Fig. 2.
figure 2

The Evaluation Viewer shows video, sound, heartbeat and breath on the same time-line

Full size image

After exposure we conducted 10 to 15 min semi-structured interviews with the participants, asking them to recount their experience. During the interview we encourage them to talk further about feelings and emotions connected to the experience. At a later point in the interview we would ask if they saw any connection between their actions, their body and the space, and further if they felt like they could interact and if they could describe any relationships and correlations. Finally we would ask them to describe the setting again as they would explain it to an outside person.

The transcribed interviews were further analyzed to reconstruct the temporal structure of the experience and behavior of the participants. To extract chronological order, we used the video and data recordings, together with the statements in the interviews.

5.2 Stages

We conducted evaluations in four stages, each with participant groups with different profiles. The first stage of the evaluation was run as a formalized pretest and was conducted with experts in design and media (designer, filmmaker, theater director). We analyzed this phase also to further develop the installation and to improve technical solutions, where necessary. Before the second stage we would carry out changes to the installation. For the following three stages the installation would remain unchanged. The second group of participants consisted of students and designers. A third group was made up of participants with no background or affinity to technology or design.

A fourth evaluation was conducted as the research setting became part of a public theater event at the IXDM Lab, created together with the Theater Basel. For this we added a second wireless biofeedback interaction device to enable the participation of a second performer. The Max/MSP interface we normally use to check and control the real-time setting was displayed to an audience on a large projection screen and became an alternative way of presenting our research approach. The director and two actresses used the installation in a live performance to address the topic of future forms of community building in responsive environments. In interviews with the actors and the director after the performance we explored questions concerning the potential of the responsive environment for performative expression, their behavior compared to a conventional stage situation and the communication and expression through biofeedback signals in this type of staging.

6 Results

6.1 Exchange and Control

The interviews produced a number of insights toward our core research questions concerning the transfer of control and emotion in this type of environment. At the current stage of the design and implementation the participants do not perceive the environment as an externalization of their own body, but interpret their surrounding as a feedback system or at the most as an encounter with something affective. One participant noticed: “…the pulse I recognized a bit later, in the sound, that there is a feedback of the pulse, that the light reacts to it”. Another participant explained that the experience was “…calming and enjoyable because of the (externalized) heartbeat”. Further examples: “When I noticed that I could control it myself, I quite liked it. It was fun to play with it. It became more joyful, because oneself was in charge.”, “… then that frightening, abrupt, heteronomous part – there is something coming towards me and I have to react – changed into: I can control and influence the situation.”

6.2 Ambivalent Associations

A number participants described their experience as ambivalent and therefore interesting. They would name specific media components (light, sound, air stream, tactility) that over the course of the experience they at some point conceived as pleasing, frightening, boring or irritating. The changing associations were also caused by the interplay of the media components and changed the participant’s state of mind: “It was a state that could switch very quickly from “very pleasant” to “rather uncomfortable”, but exactly this made it interesting”. “Concerning the light, it was from friendly to hostile, concerning the movement of the noises, also from friendly to hostile, then dark-bright and smooth-hard, I found that interesting”.

Since we renounced recognizable narrative elements and only introduced noise as an abstract audio element, the interplay with other media components caused varying associations. Participants interpreted the noise as sound of the sea, water, wind from the Antarctica, or similar. In two cases the relaxing sound of water changed into the noise of a highway – which could have to do with the 3D-sound moving and going in circles around the installation. In one case the bright light source at the ceiling without an association was first considered a dazzling technical device. When the participant laid down on the floor and looked at it while the wind from the ventilator moved the textiles and the water noises came in, the lamp all of a sudden became a romantic star.

6.3 Affection

We could find a strong affective relation that evolved between the participants and the (technical and non-technical) artifacts. Affection is used here in a twofold dimension: Firstly, it is understood in the sense of emotion: All participants developed an emotional relation towards some of the technical and non-technical artifacts. The quality of these emotional ascriptions varied, but one dominating pattern could be observed: technical elements (like fans, lights and the abstract sounds) caused ambivalent feelings, while ‘organic’ artifacts like ‘wind’ from the fans and the white and light tissue were correlated with positive feelings. Affection in the second sense means that humans are affected by the environment and vice versa – in the sense that they change each others behavior. This relation proved to be inherently bound to the question of power: As long as the participants didn’t realize that they were able to control the environment, they often felt uncomfortable and alienated. As soon as they discovered the options of influence, they began to feel at ease.

6.4 Phases of Behavior and Exploration

When we analyze the chronological order of the experience, similar patterns of emergent activities across participants can be identified. As they describe their impressions and their own actions in the environment, several phases emerge: When they enter, participants express a situative distance from the space and immediately follow that with formulating an explorative strategy. They would then go on to identify (known, familiar) elements in the environment and very often immediately rate or classify the objects. After that, they introduce and describe ideas of systems, functional or formulated as antagonisms (nature vs. technology). Later they go back to appraise the overall experience, often with an element of self reflection. In the end, most participants formulate concrete goals, make plans and show reflection in relation to the installation.

The emergence of this common, strong pattern is quite surprising and did not appear in the original, narrative structure of the interviews. The chronological order enables us to see the early adaptation of the participants to the environment and the strong influence of such a process on the experience. Two well established models describe strategies of experience and action in dynamic situations: Situation awareness and Sensemaking. Situation awareness (SA) formulates three distinct stages that form a basis for planning and action in complex situations [23]: The first stage calls for an identification and classification of elements in the environment and the description of their attributes and dynamics. The second stage connects the elements and their states and puts them in context with an available model. A third stage is concerned with the prediction of states of elements and their behavior in the environment, enabling proactive decision making. We can clearly correlate the processes exhibited by the participants with the three phases of SA.

We also observe more complex, interwoven patterns: Sensemaking has been applied to understand how people behave in unstructured situations and describes strategies of model creation and adaptation as questioning, matching and validating frames. McCarthy and Wright have extended the original formulation to describe experience as a continuous, active engagement with a designed environment. They identify six connected processes of sensemaking [24] that match with statements, themes and structures in our interviews:

  • Anticipating. Our experience is influenced by expectations: “…I realized I am curious what will happen in there.”

  • Connecting refers to an immediate, prelinguistic sense of the encountered situation that primes our interpretation of what follows: “It came to me like this: I felt secure.”

  • Interpreting is a narrative construction of an experience: “I like colored lights that change.” and includes speculation about what has happened and what will happen.

  • Reflecting. We develop attitudes and judgments of developing experiences: “…strange that we perceive the sound of a highway as noise and a waterfall as romantic?”, enabling us to see experiences in context.

  • Appropriating connects an experience with previous and future encounters and with our self-image: “That would be something I would do the next time I go in!”

  • Recounting leads beyond own experience and relates to the experience of others.

The obvious novelty and unfamiliarity that our setting has for the participants exerts a strong influence on the behavior, reflection and overall experience. Although the environment is calm and reduced and does not suggest a task or objective, it triggers a process of sensemaking, where visitors intensively engage with the environment.

7 Conclusions

Although the general structure of the feedback loop as it is implemented in our environment is very simple and the artistic design of the space is deliberately reduced – to the point of being associated with sensory deprivation – we can distinguish different mechanisms of control and influence: Ambivalence in perception and emotion makes the space interesting. To some degree participants experience affection, describing coupling between themselves and the designed environment. At the current state of development, participants do not perceive the environment as an externalization of the own body, but interpret their surrounding as a feedback system or at the most as an encounter with something affective. In a number of cases the question of control determines the overall attitude, often switching from negative to positive emotions.

When analyzing the temporal structure of participants’ behavior and their experience, identifiable phases emerge that match the three stages of situation awareness. More complex processes of sensemaking, that participants execute to form and adapt models in reaction to the unfamiliar situation, are also present.

The two dimensions of affection (coupling people with their environment) show the importance to extend the perspective on ubicomp technologies beyond a pure functional view towards the dimension of affective processes that partly take place on a primordial level of human perception. The coordinated effort the participants make to understand and structure novel environments points to a (higher than expected) cognitive engagement that is in particular directed at the underlying, behavioral structure of the environment.

8 Outlook

In future work we plan to increase dynamic complexity of the responsive environment by actively driving feedback loops and developing strategies for introducing interventions and disturbances. We will incorporate the knowledge of situation awareness and sensemaking in the encounter into the dramaturgy and the design of the space. Further, we will investigate the effect of time and repeated exposure to responsive environments, in particular on the affective state and the emergent activities of a visitor.