Keywords

1 Introduction

Augmentative and alternative communication (AAC) systems are tools to overcome communicative challenges, involving those pertaining to storytelling. Several AAC systems have been developed to present multimedia content. To develop these systems, the AAC system designer requires information regarding the in situ interactions within AAC-mediated collaborative and co-constructive activities of augmented speakers and their interlocutors. However, the efficacy of questions asked by interlocutors while presenting content such as photographs, movies, and songs as a stimulus to augment storytelling by older adults has not been a major research focus. This paper presents the findings of an analysis of peer conversations between geriatric health service facility residents with neurocognitive disorders and their care staff while showing photographs on a touchscreen.

The objectives of this research were, specifically,

  1. 1.

    to test the relationships between the presence or absence of specific types of questions, and their corresponding success or failure in eliciting life stories, and

  2. 2.

    to mine frequent sequential patterns of care staff utterances, including the specific types of questions in cases considered successful.

To accomplish these aims, we applied the chi-squared test and the cSPADE algorithm in R to conversation data.

The remaining paper is organized as follows: The subsequent section details related work. In Sect. 3, a brief overview of the conducted experiment and analysis is presented. Sections 4 and 5 provide the results and their discussion, and Sect. 6 concludes the main findings of this study.

2 Related Work

Regarding conversations with older adults, there are several studies investigating the effect of presenting content that serves as a clue to storytelling. Fried-Oken et al. reported that AAC was associated with greater use of targeted words during personal conversations [4]. Bourgeois et al. reported that using a memory book consisting of autobiographical, daily schedule-related, and problem resolution information improved the duration of speaking time of older adults [3]. These studies suggested that more detailed or a larger number of life stories can be obtained by presenting content that serves as a pointer for the conversation.

There are some studies that support storytelling using information and communications technology (ICT) owing to its advantages such as easy presentation of multimedia content. Alm et al. developed a system that could present content on a touchscreen and reported that the system proved remarkably effective in restoring a degree of equality in communication with a person with neurocognitive disorder [1]. Webster and Hanson developed a system for care staff to obtain information regarding older adults [8]. The contributions of their research include the identification of factors important in working with a care staff population, introduction and evaluation of a software tool for care staff in residential homes, and emphasis on the potential benefits of technology in assisting care staff. However, these studies did not consider how the care staff interacted with older adults.

Storytelling has been examined from many different perspectives. Nishida et al. conducted questionnaires for caregivers and reported that their communication skills can be explained based on three factors: consideration to receptive conversation, consideration to speech, and strong patience [6]. Stuart examined the storytelling patterns of older adults in order to understand the complexities of this type of communication [7]. Thereby, he reported that repetition of the same story includes three patterns or categories: construction of the repeated story, order of the stories told in repetition, and stories that were repeated with added “update” information. Our research is distinguished by the fact that its focus is on the questions asked by care staff during a conversation.

3 Method

All participants were treated according to a protocol approved by the Shimane University Institutional Committee on Ethics, and informed consent was obtained from each participants.

3.1 Participants

Four residents between the ages of 78 and 88 with neurocognitive disorders and six care staff from a geriatric health service facility participated in this study. In this study, we use “neurocognitive disorders” instead of “dementia” in consonance with DSM-5 [2]. The average score of the mini mental state examination undertaken by the residents was 17.75 (SD = 5.44). A communication peer was selected for each resident based on the staff member’s assignment as a care giver. This pairing is henceforth termed a dyad. Their conversations were in their native language (Japanese).

3.2 Equipment

A technology probe is an instrument with limited functionality that is deployed to explore unknown information and expected to obtain useful or interesting data [5]. Figure 1 shows screenshots of our probe, which is capable of browsing thumbnails of photographs, showing a selected photograph in a larger size, logging interactions, and recording speech. For this study, 190 common photographs and 46 personal photographs were installed.

Fig. 1.
figure 1

Screenshots of our technology probe

Full size image

3.3 Data Collection

Each dyad used our probe and engaged in approximately 5 min of unstructured conversation two times a week over a period of four weeks. During these conversations, each dyad sat shoulder-to-shoulder to share a touchscreen, as shown in Fig. 2. On some data collection days, the residents’ cognitive conditions were not sufficient to recognize photographs. Consequently, we collected audio recordings of a total of 27 conversations. By using the data logged by our probe, we extracted 121 photo segments from the transcribed audio recordings concerning the enlarged photographs and utterances.

Fig. 2.
figure 2

Experimental apparatus

Full size image
Table 1. A part of the 30 codes
Full size table

3.4 Analyzing Transcripts

First, we classified each photo segment indicating whether residents did (success) or did not (failure) tell their life stories. Second, we coded each sentence from both the staff and the residents using emergent coding. The final coding scheme contained 30 codes. Table 1 shows a part of the codes. Third, using the chi-squared test, we examined the relationships between the presence/absence of two specific codes associated with care staff utterances and success/failure. The codes that we focused on were (a) questions about things or events depicted in each photograph and (b) questions about things or events associated with, but not directly depicted in each photograph. Finally, we mined frequent sequential patterns including codes with significant differences using the cSPADE algorithm in R [9].

4 Results

We obtained 66 successful photo segments and 55 failures. In some cases, the same photograph and the same resident generated both success and failure. A significant interaction was found for (b) by the chi-squared test (\(\chi ^2(1)=25.548\), \(p<0.01\)), but not for (a) (\(p>0.05\)). Life stories were more likely to be elicited with (b) (67%) than without (b) (14%). The top 21 sequential patterns with a length of 2 items or more are listed in Table 2. After removing the sequential patterns of a single itemset and patterns with support less than 0.4, there were 37 patterns of successful elicitation containing (b). These patterns began with (a), (c) and/or (d); and progressed to (b); followed by (b), (c), (d), (e) and/or (f); where (c) represents an in-depth question about each resident’s utterances, (d) represents a repetition or recapitulation of each resident’s utterances, (e) represents an utterance expressing what the care staff member thought or felt, and (f) represents an utterance expressing the words that the resident found difficult to say.

Table 2. Top 21 sequential patterns with a length of 2 items or more
Full size table

5 Discussion

5.1 Results of the Chi-Squared Test

The results suggest that (b) may be effective to elicit life stories. Figure 3 shows the transcript of the conversation between Resident A and Care Staff B and the photograph presented during the conversation (in this paper, we present the drawings made from tracing photographs for ensuring the participants’ privacy). The black-and-white photograph shows Resident A and his teammates wearing baseball uniforms. The “*” in the transcripts represent utterances coded with one of the 30 codes in the final coding scheme other than (a) to (f), the focus of this study. Question (b) includes the keyword “company” which is not directly depicted in the photograph. Using question (b), Care Staff B elicited life stories from Resident A, including stories about a team from his area of the city, Yotsugane.

Fig. 3.
figure 3

Baseball photograph and transcript between Care Staff B and Resident A

Full size image

Figure 4 shows another transcript of a conversation between Resident A and Care Staff C, where A was presented with the same photograph. Using question (b) and the keyword “position,” A’s life stories were elicited, including his position as a shortstop and his effort to really stop the balls. Even with the same photograph, other life stories were drawn out by changing the keywords used in the question.

Fig. 4.
figure 4

Baseball photograph and transcript between Care Staff C and Resident A

Full size image
Fig. 5.
figure 5

Military shogi box photograph and transcript between Care Staff B and Resident A

Full size image

In contrast, a conversation without question (b) and a different presented photograph are shown in Fig. 5. This color photograph shows the military shogi box and some of the pieces. Shogi is a board game similar to chess. Military shogi is similar to Stratego, with pieces representing individual officer ranks (e.g., captain) and things associated with an army (e.g., a spy and a land mine). Only the “captain” or “a military shogi game with bonus playing pieces” that are directly depicted in the photograph were mentioned in the questions. In this conversation, questions by Care Staff B did not draw out a life story from Resident A.

Whether or not the residents tell their life stories is influenced by the questions asked by the care staff. Photographs that elicited life stories were again shown to participants on other data collection days. Although they were engaged in similar conversations, the care staff’s questions about things or events associated with, but not directly depicted in each photograph were able to draw out new and in-depth information.

5.2 Results of Mined Frequent Sequential Patterns

We determined that there were patterns that began with (a), (c) and/or (d), which progressed to (b), followed by, (b), (c), (d), (e) and/or (f) in successful photo segments. Figure 6 shows the transcript of a conversation following this pattern and the photograph presented during the conversation. In this photograph, Resident A is shown with a “Merry Tiller” and a Merry Tiller sign. Behind Resident A, we can see part of the company’s sign. “Merry Tillers” were rotary tillers and cultivators that were manually pushed. They were sold by Yokoyama Agricultural Machinery Company.

Fig. 6.
figure 6

Merry Tiller photograph and transcript between Care Staff D and Resident A

Full size image

First, Care Staff D asked question (a) to confirm whether Resident A can recognize the place where the photograph was taken. Care Staff D mentioned the company name in utterance (f) to invoke Resident A’s memory. Next, Care Staff D explored the life stories of A by question (b) which included “Yunbara” (the address of the dealer), “work” and “fix.” Then, D asked question (c), drew out A’s life stories, repeated A’s utterance (d), and drew out A’s life stories. As a result, A’s life story was elicited: A’s job was to fix cultivators. Finally, D repeated A’s utterance (d) a few times.

The pattern of care staff’s questions confirms the cognitive ability of the residents, explores their life stories, and focuses on specific topics.

5.3 AAC System Design

Through conversation data analysis, we were able to identify questions that are effective in eliciting life stories and patterns for the questions. The system that supports storytelling should add functions not only for older adults but also to support questions from the care staff. For example, we propose a function to present keywords related to photographs extracted from the conversation data of other days. With this function, we expect that the care staff will ask questions (b) including keywords about things or events associated with, but not directly depicted in each photograph. The keywords, such as the names of the things or events directly depicted in the photograph, will also support questions (a).

Fig. 7.
figure 7

Possible implementation for showing a photograph and keywords associated with it

Full size image

Figure 7 shows a possible implementation of this function. This screenshot shows the photograph and keywords can be extracted from transcripts of conversations with the figure on the other days. In this example, “Merry Tiller,” “Yokoyama Agricultural Machinery,” and “Cultivator” are extracted from the transcript in Fig. 6. Some of these keywords can be reused by the care staff to elicit life stories from the older adults.

5.4 Limitations

In neurocognitive disorder care, life story work is also conducted in groups. Our analysis focused on peer conversation. Therefore, care staff may use different types of questions and sequential patterns to elicit and share life stories in such cases.

6 Conclusion

We analyzed peer conversations between geriatric health service facility residents with neurocognitive disorders and their care staff while showing photographs on a touchscreen. We examined the relationships between the presence or absence of specific types of questions and success or failure in eliciting life stories. As a result, we found that it is important for care staff questions to include things or events associated with, but not directly depicted in each photograph. Furthermore, we extracted frequent sequential patterns including these questions in combination with confirmation, in-depth question, etc.

These insights may be used to enhance the design of AAC systems for storytelling. Designing such systems requires careful consideration of not only the provision of content as a stimulus for older adults but also the support for interlocutors to ask effective questions within the frequent sequential patterns. Functions to provide clues for asking questions should be incorporated to explore possible life stories, and focus should be placed on specific topics while showing the content. Future research is necessary to analyze group conversations and develop AAC systems with these functions.