Keywords

1 Introduction

This paper will present the participants’ experience as a mode of personal and communal ritual and person-centered care in a common theatrical environment in ten life-review sessions held at a senior living facility in Richmond, Virginia, from March through June 2016. Therapeutic potential will be discussed in the user’s personal engagement with the avatar, sense of anonymity, distancing effects, and emotional release. Each life-review session was followed by lengthy community talks, building strong empathetic bonds between the storyteller, the artist (mediator) and the audience (residents). Moreover, the avatar movie screening brought everyone together reflecting on all those stories told via avatar. Later, the avatar movie was distributed to the participant’s family members and grandchildren enriching intergenerational relationships.

We observed striking patterns of interaction inflected by physical limitations of the older adults’ bodies and technical limitations of the life-review system, which shaped the users’ verbal and kinetic expression revealing surprising engagement during the storytelling process. The sentiment analysis and response features contributed to empathetic relationships among the storyteller, the artist, and audience.

Finally, this paper addresses the artist’s experience as an active participant in this life-review event. Originally inspired by Korean shamanic ritual, the avatar life-review process generates cross-cultural dialogues among American seniors and Korean-born artists all seeking enriched communication.

2 Virtual and Augmented Body

In Body without Organs (BwO), Gilles Deleuze defines the virtual as a potentiality that becomes fulfilled in the actual. Though not yet material, it is real [1]. The virtual body facilitates the discovery of potential selves. Maurice Merleau-Ponty describes the body as neither subject nor object [2]. The body constructs its surroundings virtually, operating as a subject through movement, but doing so within objective space. A phantom limb persists vividly for the amputee—a phenomenon that cannot be explained in solely psychological or physiological terms [3]. In the same way, the phenomenal body can reveal the situation of the augmented body we encounter in mixed reality prompting meditation on embodiment, perception, and human experience. The distinct modulation of time and space provided by virtual reality technology facilitates a dynamic mode of phenomenological experience that can help us examine what it means to be human in the digital age. Phenomenological embodiment can be understood as a quantum state of paradox—neither here nor there; neither subject nor object [4]. It parallels the distancing effect used in drama therapy and therapeutic puppetry to promote healing by mediation between virtual and actual bodies.

Virtual and augmented reality have become part of our daily lives; our bodies are constantly mediated, represented, and multiplied via technology. In the flux of multiple bodies, spatiality and temporality, it is possible and necessary to discuss the role of human bodily experience in complex relationship with our digital reflections.

The avatar life-review was designed to facilitate this discussion. Its lip-synchronization and gesture-tracking features facilitate simultaneous speech between the virtual and physical body. In this embodied speech process, the human storyteller is augmented into a virtual storyteller, facing each other, simultaneously speaking, moving, and reflecting in a spiraling flow.

3 Person-Centered Care and Anthropology

Carl Rogers developed his non-directive form of talk therapy with the aim of supporting the patient’s autonomy and self-exploration. He advocated for “unconditional positive regard,” a practice of accepting and supporting a person regardless of what he or she says or does [5]. Unconditional positive regard supports deep empathetic relationships by promoting trust and free expression. The avatar life-review system was designed to support the user’s autonomous storytelling, while providing digitally mediated reflective listening allowing the storyteller to see and hear the story as an actively listening other might. Through detection and analysis of the storyteller’s voice, words, gestures, and reflection of those behaviors via avatar animation, background images, music, and ambient sound, the system functions as an empathetic listener, but crucially also allows the user to function as an empathetic listener to herself simultaneously without judgment remaining unconditionally positive. The audience, artist, and storyteller build strong empathetic relationships mediated by emotionally responsive background sound and visuals.

The participant’s experience is the most critical element in avatar life-review, involving embodiment, self-perception, emotion, and phenomenological reality mediated by technology. All participants—storytellers, the artist/mediator, staff, family, etc., contribute to a shared experience.

In his book, Where the Action Is: The Foundations of Embodied Interaction, Paul Dourish problematizes the dichotomy between “user” and “designer,” noting that the designer is always also a user [6]. Grudin (1993) argues that these terms influence our understanding of systems according to the way computer engineering faced initial design problems [7]. The term “user” is commonly understood as a general category of non-experts, denying dimensions of the participants’ experience as genuine. But user experience is more than a pattern of behavior; it is phenomenological and subjective. We need to acknowledge their meaningful engagement with the system [8]. From an anthropological perspective, the designer (artist) is also considered an active participant with many dimensions of experience within system, including imaginary projections, expectations, interaction, etc. [9]. In our avatar life-review sessions, the artist actively participated as a designer, storyteller, mediator, observer and audience; meanwhile, the users actively contributed to the shaping of the emergent experience and to the ongoing development of the system.

4 Thematic Observation: Avatar Life Review

The technical design of the avatar life-review system was important, but planning for its use by older adults in the residential setting was crucial, requiring a careful anthropological approach [10]. Ten older adult participants volunteered to participate in avatar life-review sessions at a senior living facility in Richmond, Virginia. We gathered qualitative data via interviews, observations, video, field notes, and surveys. The following themes were discovered based on these data:

1. Personal Immersion 2. Social Engagement 3. Technology 4. Improvisation and Mediator

4.1 Personal Immersion

We had planned to test the project in private rooms to promote exposition of subjects’ personal stories. To our surprise, the elder home gave us access to their common theater space for avatar life-review sessions, which supported a different dynamic. In this theater setting, participants were public storytellers with an audience of staff, residents, participants, and team members. It was challenging for the participants to tell a personal story in this context; however, very interesting personal engagement emerged in the follow-up sessions.

When Mrs. C entered the theater, she began, “I am going to tell a sad story today.” It was surprising since previous stories told by residents had been positive. I knew telling a sad story would be even harder in this unexpectedly public environment. Mrs. C told her story about her father, from when she was 6 years old. He had scolded her badly when she surprised him in jest. She acted out a wonderful dramatic performance identifying strongly with the avatar, using broad physical gestures and an animated tone of voice. She concluded with a single sentence, repeated in different voices: “please help me to forgive him.” The first seemed to belong to the avatar, speaking forthrightly, but the second was different—Mrs. C’s voice, weeping and trembling. Her tears surprised us. She confessed she rarely cries. She spoke of her experience as “such a good release,” and described a sense of “anonymity” of being able to disclose something, not as herself, but as the avatar. This sense of anonymity is related to distancing effects in drama therapy, where participants feel safe relating hidden emotions and stories. The system allowed her to talk to herself, rather than talking to another such as psychiatrist in a potentially judgmental situation (Fig. 1).

Fig. 1.
figure 1

(© Semi Ryu. Photo: Brianna Ondris.)

An avatar tells her childhood story with sad and angry emotions

Full size image

Mr. W., a 90-year-old gentleman with a husky voice, also built a strong emotional relationship with his avatar. He started his life-review using a boy avatar to tell his story from elementary school. In the middle of storytelling, he began looking for his teacher. I changed the avatar from a boy to a young woman, upon which he started making his conversation with the avatar as if she were his teacher. He pleaded, “why didn’t you love me?” “come close to me,” “hug me.”

We paused, as we judged he was too immersed into the avatar. Then he stood up slowly with his walker, walked to the screen and tried to touch the avatar image. His own shadow in the projector light erased her image. It was a heartbreaking moment for the viewers.

Participants developed new modes of expression involving unique patterns of gesture and speech while engaged with the avatars. In part, the forms of these gestures were inflected by the theatrical environment; we look forward to further experiments in more personal settings to compare the variant dynamics.

4.2 Social Engagement

The avatar life-review was conducted in a common theater space building an interesting tension between private and public. The presence of an audience made personal engagement more challenging, but lengthy discussions followed each life-review creating a strong empathetic community. Cross cultural discussions emerged involving traditional Korean culture, different modes of communication, etc., as the artist became a part of this community. We celebrated similarity through empathy; difference through conversation (Fig. 2).

Fig. 2.
figure 2

(© Semi Ryu. Photo: Brianna Ondris.)

Residents watch a participant’s avatar life-review performance

Full size image

Although the system was designed around the avatar, participants were conscious of the audience’s presence and responses. Each story was followed by applause. We shared joy, grief, anger, and many feelings about fathers, husbands, mothers-in-law, children, and other relations. The avatar system’s sentiment analysis and enhancement algorithms amplified these empathetic relationships as the screen and sound responded to the emotional quality of the storytelling, reflecting the empathy of listeners.

We tried various setups to experiment with the relationship of audience, storyteller, and avatar: audience located before or behind the storyteller, central or diagonal screen placement, etc. In a central setting with the audience located behind, some participants felt uncomfortable; they felt it was poor manners to turn their backs on the audience, although the avatar was facing them. Some participants wanted to watch the audience’s response during their storytelling, to be engaged with audience more than the avatar.

The aspect of communal ritual was evident after each life-review session. Often, the audience surrounded the storyteller and dynamically expressed their impression, empathy, encouragements, and similar experiences, using warm body language such as hugs and hand-holding. This occurred consistently and involved everyone, including observers, mediator, residents and staff members. This social engagement was also highlighted on the screening day when we showed all of the recorded avatar video in the same theater.

The participants enjoyed watching their own avatar performance as well as others’. Often, they were surprised by witnessing their own stories as avatar performances. The distanced setting brought a fresh perspective on stories they told via avatar; some were surprised by their own words. Participants wanted DVD discs of their avatar video to watch again with friends and family members.

4.3 Technology

The system was adjusted based on the residents’ feedback. Avatar’s faces were updated to more cheerful expressions on the request of residents. They wanted to engaged with happier looking avatars and probably happier aspects of their memories. The avatar originally had head motion tracking and lip-sync only. But we observed the significant role of hand gestures in older adult’s speech. The residents also wanted full upper body interaction. The final prototype incorporated full upper body interaction with hand-gestures, lip-sync, and sentimental response.

The technical setup was challenging, due to the physical limitations of older adult users. The Microsoft Kinect sensor requires straight body registration. However, the participants were usually sitting in the chair in a reclined posture which made body registration difficult and unstable. We tried to make the system work optimally, but realized the participants’ comfort is more important for natural storytelling process. Wearing headsets created a connection between artist and participant, and built a sense of trust. It was like placing a crown or ceremonial headwear on each participant to guide them to an alternate world (Fig. 3).

Fig. 3.
figure 3

(© Semi Ryu. Photo: Brianna Ondris.)

A participant registering her body with the Kinect sensor

Full size image

Some participants’ jewelry conflicted with Kinect sensors due to reflections. But if jewelry was important to their identity, we couldn’t ask them to remove it. We had anticipated a moderate number of technical conflicts but, as always, the particular challenges surprised us.

Technical limitations of the prototype system prompted unexpected negotiations with the technology by the older adult users. For example, one participant was speaking the words “in my heart,” and trying to make her avatar touch her heart. Since finger detection had not been implemented yet, it was a challenging motion for the avatar—not quite detecting her hand on her chest. Suddenly, she stood up and repeated the phrase “in my heart,” trying to elicit an appropriate response from the avatar. She also tried to create a hole with her fingers, but when the avatar’s fingers did not follow, she spun her hands together, to express the notion of “hole.” New gestural language emerged to suit what the technology can support. The older adult users continually challenged the system, discovering ways to express their emotions despite its limits.

We didn’t expect older adults to be so kinetically animated in their storytelling. As noted above, their hands served as another speech organ, but with the avatar system, the entire upper body was engaged. I felt a strange sense of guilt when the avatar couldn’t fully support the user’s emotional expressions. They wanted to express emotions through gestures such as kissing, hugging, or touching the heart. There was something beautiful about their wishes, struggles, and humor in relationship with the avatar.

The sentiment analysis system can detect emotions in the user’s speech. Background images and sounds were modified based on detection of 6 emotions, to support an empathetic connection between storyteller and audience. These changes were managed by both automatic and manual means, creating a semi-automatic solution. Emotions in their stories were complex and changed rapidly. The artist/mediator often intervened ahead of the computer response in an effort to keep pace.

Often, participants and audience members interpreted the avatar’s appearance and behavior differently, even when it was unchanged. They seemed to base their interpretations on who is telling the story, and how she/he interacted with it, rather than according to an impartial, abstract analysis of its gestures.

4.4 Improvisation and Mediator

We demonstrated hand puppetry as a familiar analogy, to explain avatar interaction to the older adult participants. This 21st century puppet can be used to discover hidden memory and feelings through improvisational storytelling. However, most participants came with a prepared story. They seemed to have practiced the story before the sessions. But eventually they began to improvise, and volunteered to participate again, inspired by others’ stories. We offered a weekly topic to help them to prepare, but often residents prepared stories they wanted to tell, regardless of topic. Participants usually said their story would be very short, but often turned out longer than anticipated. Over time, the line between private and public was blurred as participants became more comfortable telling stories and improvising.

For the artist, actively participating as a mediator, designer, audience, and storyteller, the avatar life-review was an amazing sharing experience. For over ten years, she has explored virtual puppetry inspired by Korean shamanic ritual. She has explored cultural factors in the emotional psyche—in particular the Korean notion of Han—in paradoxical relationships between virtual puppet and puppeteer [11]. She has recently begun to apply these to the study of, and therapeutic interventions in, the aging psyche of the senior population.

This cultural background is uniquely connected with the therapeutic potential of avatar life-review. Korean culture differs from Western culture in many aspects. As a society with a strong Confucian heritage, we value our elders highly and seek them out for guidance and connection. Direct eye contact is regarded as a challenge towards one’s elders; averting one’s eyes is more respectful. Life-review sessions became an opportunity to mediate these cultural differences. The residents were fascinated by Korean culture—food, clothing, traditional costumes, Confucianism—often leading to cross-cultural discussions after life-review sessions.

5 Gerotranscendence and Future Direction

Life expectancy in Western industrialized nations continues to rise. Today’s retirees are likely to live another twenty-five healthy years or more. We must prepare for this emerging reality by adapting the culture to better integrate older adults into their communities [12].

Our work on the avatar life-review system was premised on the notion of gerotranscendence—a developmental stage that occurs when an individual who is living into very old age shifts perspective “from a materialistic and rational view of the world to a more cosmic and transcendent one, normally accompanied by an increase in life satisfaction” [13]. Older adults’ storytelling performances, filled with emotion, positive energy, and humor, provide excellent material for younger generations to listen and learn from those perspectives—for their own sake as emerging adults who will have to negotiate that same difficult balance of matter and spirit and for the sake of communal integration with elders.

The first prototype of the avatar life-review system focused on the basic structure and has not yet been tried with a variety of experimental techniques. We see broad potential for evolution to enhance future users’ experience. We value the opportunity this study afforded us to realize the importance of emotional expression in our stories. Even the systemic breakdowns brought about unforeseen returns: we realized how important it is to touch our own hearts, to hug and kiss our beloved others, real and virtual.

Further studies on avatar life-review will explore diverse challenges: early memory loss in older adult population, PTSD patients, and diversity in society, to increase awareness of, and adaptation to, racial, cultural, and gender difference through person-centered care and anthropological research.