Keywords

1 Introduction

Oral storytelling is the oldest form of sharing stories. It continues to be an excellent way to encourage socialization and communication for seniors and people with dementia, and has been found to increase life satisfaction and self-esteem [3]. A common form of modern oral storytelling is family narratives around family photographs. In recent years, digital storytelling has been growing as a field and using photos to frame that storytelling has been a common thread. Many different modes for digital storytelling have been explored; tablets, projected tabletop displays, cell phones, and digital paper and pen, to name a few, but many structure or limit the story. We hypothesise that placing a focus on free-flowing, independent oral digital storytelling will increase older adults’ motivation to tell and document their stories and will add a new range of possibilities to the existing digital storytelling work.

1.1 Background Information

Photo Storytelling for Seniors.

Several projects have already explored the area of storytelling with family photographs specifically for seniors. These solutions aim to increase socialization or memory in their older adult users. Recent research shows that a mobile application is useful in encouraging seniors with early-stage dementia and their families to engage in storytelling from photographs in the form of digital or physical postcards [7]. These postcards use photos annotated with a written, typed, or spoken message which the recipient can respond to in kind. Video-enhanced speech recognition is used to support speaking messages to be written onto the postcard, and speech or video media can be added to the digital postcards, a feature reported by users to be the most meaningful. This suggests that seniors may benefit from multimodal interfaces for storytelling.

Other research has explored how tools designed for family and caregivers can improve a senior’s socialization. One such intuitive approach researched a digital annotation system using paper photos and a digital pen that allows family and caregivers to collaboratively create an interactive photo album comprised of audio-enhanced photos to be viewed by seniors [9]. Family members can annotate family photos for the senior, through both written text and recorded audio, with memories of the photo or other details. Seniors are encouraged to share their memories by going through the album and playing the recordings. This use of pen and paper is familiar to seniors, but limits how far the system can go compared to using a tablet.

Picgo [5] is another system that is designed for caregivers that simplifies the work of annotating family photos by prompting for answers to five general questions (who, what, where, when, and why) about each photo and adding tags that correspond to the given answers. These annotated and tagged photos are meant to be used by the caregiver to design and implement reminiscence activities with the senior, so it is not meant to be a tool for creating and telling stories.

These examples all focus on brief messages or memories around a single photo. They enhance communication and socialization in a back-and-forth conversation, but do not encourage a user to spend a longer time to tell a complete story.

Audio-Enabled Digital Storytelling.

Other work has been done more generally into the field of digital storytelling around photos without a focus on older adults as users or recipients. These works provide more examples of how speech can be used to enhance digital storytelling, but much of the research is older and recent technological changes leave much room for improvement. Moreover, increased recent awareness of issues related to social isolation for older adults [8] and storytelling’s potential to reduce social isolation [3], demand that efforts be focused on the usability aspects of authoring tools that support older adults’ creation of complete oral stories augmented with photos.

Among projects that exploit the audio track of stories is an eavesdropping system that uses audio recording to derive information from storytelling [2]. As a story is being told, audio capture derives keywords, themes, and other metadata for the displayed photo. The main purpose of this is to index the photos for future text searches, and stories are saved so they can be shared through email. This research was solely into the speech-processing system that leverages storytelling for the user’s gain. This tool runs as a part of a thumbnail gallery and photo viewer that is built into the computer, so a user must choose to view their photos through that system in order for the audio capture to occur. Unfortunately, the system is not designed to support complete authoring.

Other work includes a custom-built tablet-like device [1] to build and view stories with the option of recording audio, a table-top projected interface [11] that supports telling a collective past with a focus on natural conversation and flexible storytelling, and the Cherish system [4] that uses digital photo frames to prompt storytelling based on visiting guests or special events. These systems all create a larger story across multiple photographs, and allow more freedom as to the story’s structure. Our research plans to bring a similar flexibility to digital storytelling for seniors and leverage such previous advances to improve the authoring process for complete multimedia stories.

Storytelling Design for Seniors.

This research is motivated by previous studies into how seniors want to document their stories and that propose guidelines for designing solutions for seniors’ reminiscence. One study found that seniors do enjoy reminiscing in the form of storytelling and that, though they often do not feel like they have knowledge worth sharing, a trigger (such as a photo or verbal prompt) would help to remind them and to make them feel comfortable in telling [12]. Similarly, another field study into the different ways that older adults are recording memories found a need for a malleable way to create stories that allows users to have control over their own content [6]. Both studies found that seniors want to construct narratives around their stories, and call for simple, intergenerational solutions that create story artefacts.

Opportunities to Focus on the Authoring Process.

Despite the research that shows a desire for ownership and flexibility in storytelling, most existing storytelling tools use a very structured approach in which a user is prompted to build a set of photos or answer a set of questions for a story, and often the user is a caregiver, not the senior. Based on the previous work surveyed here, we can hypothesize that a flexible, free-flowing, and independent approach will match older adults preferred storytelling methods by giving them more control over their stories and artefacts. Motivating an older adult to tell a story is a common problem as well. Therefore, our aim for this project is to research whether suggesting clusters of family photos, built from the user’s own stories, as triggers can encourage self-directed, free-flowing oral storytelling. Once a story has started, that speech can be leveraged to guide the user through a longer, unstructured story.

2 Proposed Solution

This research will investigate the needs of older adults with respect to supporting the capture of storytelling from photographs and how best to support these with advanced multimodal and automatic speech recognition interfaces. It is hypothesised that such storytelling will require a tool that is free-flowing, without enforcing any strict story structure. As well, such a tool should be designed for user independence so the senior can own the process and use it on their own, and should be speech-enabled so the themes of a story will motivate further stories suggestions. The large goal is that these features will create a tool to motivate seniors not only to tell and share their stories, but to encourage them to continue to tell that story and to create a larger narrative arc.

Design Motivation.

As surveyed earlier in this paper, research shows that seniors desire full control and independence over their own histories [6], but much previous work researches tools that family members or caregivers can use along with or to guide seniors. A storytelling tool that is designed with the older adult as the sole user and content producer/curator is an area that is still largely unexplored. Although storytelling is a collaborative process, the fact that the senior owns their stories and curates their own artefacts provides a stronger motivation. Ownership will also likely lessen the chore or obligation around documenting family history as they will choose when and how to use the tool.

Another common trend in existing applications of digital storytelling is to provide a forced structure for the story [1, 4, 5]. This can take the form of a set of basic questions to be answered or by requiring the user to manually build an ordered set of photos from which a story will be told. These structures limit how a story can be made. Telling stories as personal narrative is a culturally ingrained act, with a standard outline that is generally followed [10]. Therefore, it is not necessary to provide, for example, generic questions that would be answered organically if a complete story were encouraged. Instead, a more free-flowing approach can provide an initial trigger, built from information gained from previous stories told, and guide the user with suggestions of how to continue the story. This will allow for an open-ended story with no fixed beginning or ending beyond the user’s shared memories and will work well with natural human interactions such as speech.

Remembrance almost always involves speech. Stories are shared orally, and through oral storytelling, family histories are preserved and moral values passed on. This makes speech interaction a logical mode for digital storytelling, but also suggests that it must be a natural interaction that is inspired by how stories are shared between people. These oral stories can create rich multimedia artefacts through explicit recording and post-processing to determine story themes, for example. These themes become apparent as speech about a photograph can reveal much about the content of that photo.

Storytelling Artefacts.

The multimedia artefacts resulting from these stories create sharable outputs in the form of individual stories or a larger multimedia biography. The senior, as independent curator of their stories, is responsible for the management and sharing of these artefacts. Individual stories can be modified or rearranged in the larger story. One or multiple stories can be shared with others to encourage communication between distant family members. Together, digital stories create a living artefact that encompasses one overarching story holding the complex framework of a family’s history.

Motivation in Storytelling.

This research will bring together the concepts of free-flowing storytelling, senior user independence, and speech-enabled interfaces to investigate a tool that promotes storytelling in older adults. As the owner and curator of their storytelling process, they will be motivated to share their stories. This may leverage a sense of responsibility to document family history while lessening the work usually required to do so.

Another impediment to older adults sharing stories is a sense that others are not interested in their memories and so those memories are not valuable [12]. Providing a memory trigger, based on recently told stories, is a reminder of their stories’ value and will strengthen the original motivation.

That initial trigger will aid a senior to start a story. Continuing this story across a complete narrative arc will be done through the suggestion of new triggers based on the current story. This continues the motivation of the story on a longer, open-ended path, and it is believed that a free-flowing, independent, speech-enabled solution will achieve this goal of motivation across the entirety of a story.

3 Proposed Methodology

Older adults’ current methods of storytelling, family history documentation, and photograph organization will be investigated in one-on-one sessions using Contextual Inquiry to gain further understanding of common practices, desired outcomes, and physical interactions with these items. This will support previous research done in similar areas [6, 12] and will give insight into both how photographs are organized physically (e.g. in photo albums, boxes, digitally) and how the content is organized (e.g. by time, event, people). Also of interest is how people naturally cluster photos when telling a story and what gestures are used to interact with the physical photographs.

The information gathered in these sessions will be used to define the design for a tool using natural touch and speech interactions, based on what was observed from and reported by the participants, and leveraging the observed ways that the participants store, organize, and cluster their photos. The acceptance of the tool’s design will be measured in user studies conducted with each user’s own photographs and stories.

4 Conclusion

This research investigates how a speech-based tool can assist seniors with the production and editing of rich multimedia stories that can be shared with family members. A main focus of our fieldwork is discovering what will best motivate a senior to choose to tell a story, start telling that story, and continue it across a full narrative arc. There is an emphasis on independent and free-flowing storytelling, in agreement with the findings of previous research tools for seniors’ remembrance, and supporting speech interaction for oral storytelling is the preferred modality for sharing memories. These underexplored research areas show much promise for future work and contributions to reducing older adults’ social isolation through digital storytelling.