Keywords

1 Introduction

Children with complex communication needs (CCN), including pervasive developmental and intellectual disabilities, often present with difficulties in expressing and/or understanding verbal messages. Augmentative and alternative communication (AAC) interventions can promote positive peer interactions of children with CCN [21]. For example, visual aids, such as pictorial symbols and photographs, are widely used for exchanging messages between the children and their communication peers [3]. A variety of AAC systems have been developed and used as assistive products to communicate by presenting visual aids on a screen and generating speech in daily activities [2]. In many cases, the vocabulary used in such systems should be selected by speech-language pathologists (SLPs), occupational therapists (OTs), educators, parents, and manufacturers prior to exposing them to children [22]. Although several research have been conducted to build core vocabulary lists for AAC systems [1, 10], vocabulary selection is still a challenging process. On the one hand, additional vocabularies are required to support communications in diverse settings by collecting communication samples in each context [18]. On the other hand, paradoxically, if the vocabulary becomes too large, the access to each word becomes overloaded. As a result, the selected vocabularies are often insufficient to respond to children’s interests, needs, and actions as they arise during interactions.

However, few studies have focused on “just-in-time” language acquisition for daily interactions through AAC systems [4, 17]. Hirotomi et al. developed the PC-based communication aid called “STalk” system [13]. Its principal objective was to augment the understanding of verbal messages. It is capable of recognizing voices, extracting words and chunks from verbal messages, and presenting visual aids stored in a local database and/or retrieved via image search on the web. In this study, we examined the change in the behaviors and attitudes of children with CCN and their communication peers when communication is mediated by the mobile application (app) version of STalk, named “STalk2.”

2 AAC and Persuasive Technology

The behavior and attitudes of the peers are one of the most crucial determinants of successful communication with individuals using AAC. Instructions for communication peers are highly effective because knowledge on how to facilitate interactions is not intuitive for most peers [15].

Interactive information technology designed to change the user’s behavior and attitude is known as “persuasive technology” [7]. The target behavior can be hindered by a combination of three factors: lack of motivation, lack of ability, and lack of a well-timed trigger to perform the behavior [9]. That is, increasing ability by providing appropriate instructions to the communication peer is not always the solution. The AAC system designer should consider ways to simplify behavior and increase motivation. Oinas-Kukkonen and Harjumaa [19] classified the principles used in persuasive technology. We adopted two principles for our system design:

  • The “reduction” principle is a system that helps users perform the target behavior by reducing complex behavior into simple tasks.

  • The “self-monitoring” principle is a system that supports users in achieving their goals by helping them track their performance or status.

3 STalk2 Mobile App

The STalk2 mobile app runs on Android OS; the “S” stands for Smart Assistive Technology (SAT). Hirotomi et al. [13] proposed SAT as an ICT-based assistive technology (product) that provides smarter services for individuals with special needs to improve their quality of life; reduce the heavy burden on caregivers, therapists, and other assistive technology practitioners in installing and maintaining assistive products through their smart functions; and record evidence for rehabilitation and further improvement of assistive products. To achieve these goals, STalk2 was designed to increase the use of visual aids in the dynamic process of interaction between children and their peers.

We gathered requirements for the design of STalk2 from SLPs, OTs, educators, parents, and other caregivers from February 2011 to August 2012. We conducted focus groups with 39 caregivers, while 49 caregivers answered an open-ended questionnaire after the demonstration of STalk and/or a prototype of STalk2. Consequently, we focused on reducing the efforts that users have to make in creating, finding, editing, and presenting visual aids; and providing visual feedback on the usage of the system. These are based on the “reduction” and “self-monitoring” principles of persuasive technology. Additionally, we decided to implement functions for synchronizing multiple devices, backing up data, and providing web access to the data.

Fig. 1.
figure 1

Process flow in STalk2

Full size image

Figure 1 shows the process flow. In STalk2, a message is represented in a “multimedia hieroglyph” format [11, 12, 14]. In this format, a set of visual aids, i.e., images with captions, are allocated into five columns for answering the following questions: when (time), where (place), who (subject), how (feeling, language, method, tool, transportation, or material), and what (action). Originally, this format had one more question, why (reason). However, this question was omitted as a result of the requirement gathering process because the potential user groups may have difficulty answering the question.

Each column has twelve cells. The user can input visual aids in these cells in three ways, as shown in Step 1:

  • The user can touch the title of the specific column and select visual aids from the most recently used list.

  • The user can touch an empty cell and search for a visual aid by speech recognition. First, STalk2 will try to retrieve it from a fringe vocabulary database containing visual aids created or edited by the user. If no results match, STalk2 will retrieve it from a core vocabulary database containing approximately 1700 droplet symbols (http://droplet.ddo.jp/). If no results match again, STalk2 will finally try to obtain the corresponding image from a web image search.

  • The user can perform a long press on an empty cell and create a new visual aid by taking a photograph or selecting an image from the gallery app.

If necessary, the user can edit or delete a visual aid by making a long press on the cell. As shown in Step 2, the user can edit the caption, replace the image, and place a question mark or slash symbol over it.

In Step 3, the user can touch a visual aid and present it to the peer with an enlarged image, caption, and synthesized speech of the caption. N2 Text-To-Speech engine (KDDI Research, Inc.) is used for speech synthesis. STalk2 also has a special presentation panel designed for choosing among several visual aids with a question mark. In this panel, three visual aids are also displayed to express the following opinions: “I don’t know,” “None of them,” and “I don’t want to answer.” After the presentation, the user can save these visual aids as a template message in Step 4. The template message can be selected and reused quickly, as shown in Step 1’.

User activities, messages, and visual aids are recorded into the database. In Step 5, these data can be analyzed and presented as quantitative evidence of communication assistance, such as the visualization of time series data for vocabulary development. The stacked bar chart segments the bars to show comparisons between the message columns. The user can touch the bar to obtain further details, such as a set of captions for visual aids that were added during a certain time period.

Steps 1 to 4, including Step 1’, aim to reduce the effort that users have to make to create, find, edit, and present visual aids. Step 5 aims to provide visual feedback to enable users to track their performance. Additionally, the database of STalk2 can be synchronized with the Evernote cloud service (Evernote Corp.). The service will synchronize STalk2 apps installed in multiple devices, back up data, and provide alternative access to data. For example, parents and school teachers can share visual aids and messages. They can also create low-tech communication aids, such as picture cards, by printing visual aids using Evernote web pages and apps.

4 Evaluation of the Effects on Changing Communication Behaviors and Attitudes

4.1 Participants

In our study, the participants were five children with CCN and eleven of their peers, including three parents, seven teachers, and one OT. All children received special education services. Table 1 shows their demographics. The Kyoto scale of psychological development [20] was used to assess the developmental age of children who were not deaf or hard of hearing (P1, P2, and P3).

Table 1. Children’s demographics
Full size table

Pictorial symbols, freehand drawings, and/or photographs were actively used for communication. Simple text messages were used in combination with visual aids for communication with P3, P4, and P5 children. P4 and P5 children have been able to exchange simple messages in sign language but often need visual aids to compensate for insufficient hand signing skills for expressing and understanding messages.

4.2 Procedure

We conducted the longitudinal study to evaluate the effects on changing communication behaviors and attitudes by using STalk2 in real world situations. Peers used STalk2 with the children at schools, home, dormitory, and child development support centers from November or December 2012 to March 2013. Prior to interacting with children using STalk2, we provided all peers with an approximately 30-min tutorial on STalk2. They were able to explore all the features of STalk2 within the tutorial.

At the end of the study, the communication peers completed the questionnaire shown in Table 2. All statements were originally in Japanese, the native language of all participants. The questionnaire consists of ten statements, and a five-point scale ranging from “strongly disagree” to “strongly agree” was used for each. Q1 focused on the changes in the frequency of the target behavior; Q2–Q5 are related to the children’s behaviors and attitudes; and Q6–Q10 are related to the peers’ impressions about the use of STalk2.

Table 2. A list of statements in a questionnaire on changing behaviors
Full size table

The peers were also asked to record in a diary or video actual instances of communication, regardless of the presence or absence of STalk2, and their impressions about each instance during the study. Recordings of 93 and 56 instances with and without STalk2 were obtained, respectively. These recordings were made to verify the results of the questionnaire.

5 Results and Discussion

Figure 2 shows the results of the questionnaire. The percentage of top-two-box scores, which refer to those who agreed with the statement, was 91%, 64%, 55%, 64%, 91%, 82%, 91%, 91%, 91% and 73% for Q1–Q10, respectively.

Fig. 2.
figure 2

Stacked bar chart showing different levels of agreement for each statement

Full size image

5.1 The Frequency of Presenting Visual Aids with Verbal Messages

STalk2 was designed to increase the opportunities to present visual aids with verbal messages to children with CCN. In Q1, all peers except one answered that the frequency of the target behavior increased. The peer who answered “3” actively used visual aids during class; for example, showing presentation slides prepared in advance or communicating with the child by drawing. In this case, the peer substituted the use of STalk2 with these behaviors.

The following are actual instances from the diaries and video recordings of peers who used STalk2 to communicate to the child:

  • Providing an assortment of potentially reinforcing items (photographs of toys, food, drinks, places to go, and play activities) for self-selection

  • Confirming daily schedules with/without changes

  • Providing class instructions

  • Teaching good/bad behaviors

  • Orienting a person with whom the child will perform activities with (taking a bath, driving to school, etc.)

  • Confirming a driving route

  • Confirming details of occasional events

  • Explaining the current situation to ease the child’s anxiety and confusion

  • Collecting and organizing information in the form of STalk2 messages with the child with explanations.

5.2 Children’s Behaviors and Attitudes

The opportunities to present visual aids with verbal messages were increased during the study. This peer’s behavior may affect children’s behaviors and attitudes. Q2–Q5 examined these effects from the aspects of vocabulary development, focus of attention, reduction of problem behaviors, and enjoyment. The results were interpreted based on the opinions expressed by peers in their diaries and video recordings.

Vocabulary Development. Seven out of eleven peers positively evaluated the effects of STalk2 on vocabulary development in Q2. Other peers used STalk2 within conventional communication contexts. In such cases, they simply attached visual aids to words that were already expressed in verbal messages. They commented that the children understood words more precisely but the vocabulary size did not increase.

Attentional Focus. In Q3, six peers positively evaluated the effects of both STalk2 and the tablet computer on the attentional focus of children. STalk2 and the tablet computer attracted the attention of the children. Some children focused on the messages presented by the device and achieved successful communication. However, other children were sometimes too excited to operate the device.

Problem Behavior. Improved communication skills occurred in conjunction with decreased problem behaviors [5, 6]. Increasing the frequency of presenting visual aids may improve such behaviors. In Q4, seven peers positively evaluated the effects on STalk2 on reducing problem behaviors. Five of them were peers of P1, P3, and P5–children who exhibited problem behaviors in the recordings.

P1 often engaged in self-injury behavior, such as hitting the head and lower limbs, as well as biting. P1 practiced self-selection using a set of printed cards prior to using STalk2. The interval between incidents of self-injury ranged from 10 to 18 s. Once the self-injury behavior started, P1 could not stop at will. The teacher often had to snuggle with P1 until P1 calmed down. After using STalk2, P1 could select desired items, events, and places to go more successfully. The self-injury behavior persisted but P1 sometimes selected items and places by himself in order to keep calm.

P3 often exhibited high levels of anxiety when unexpected events occur; for example, when P3’s brother caught the flu, when the dog barked at night, or when P3 had no classes because it was the Foundation Day of the school. P3 often brought the tablet STalk2 installed to the parent and requested an explanation with visual aids. P3 tried to understand the events that happened and kept calm by repeatedly touching the visual aids and hearing synthesized speech.

P5 frequently stands up, spins around, and slaps his head when he is unable to understand class instructions and the rescheduling of daily activities. P5 concentrated on using STalk2 and teachers provided these information on STalk2; for example, the cancellation of some activities was represented by the slash symbol. The frequency of these problem behaviors was decreased by presenting information via STalk2 instead of printed materials.

Enjoyment. In Q5, all peers except one said that the child found communication much more enjoyable using STalk2. The opinions relating to enjoyment include the following:

  • When peers presented visual aids more frequently, the child had better understanding of the messages. Consequently, the child’s anxiety and confusion were alleviated.

  • The peers could quickly and easily add new visual aids and/or edit them to suit different communication contexts. The children enjoyed opportunities to express their thoughts, including self-selection, by using these visual aids in combination with synthesized speech.

  • Two of the children were very eager to use STalk2 and the tablet computer. Their enthusiasm was sometimes too great, which made communications difficult, as discussed in Q3.

5.3 Peers’ Impressions About the Use of STalk2

Q6–Q10 examined peers’ impressions about the use of STalk2 from in terms of reducing the burden of communication, discovering new sides of children, continuous use, recommendation to others, and overall satisfaction.

Reducing the Burden of Communication. Nine peers answered that STalk2 reduced the burden of communicating with the child in Q6. The implementation of the reduction principle ensured that the visual aids were simple to search, create, edit, and present. However, a few peers reported that STalk2 slowed down in cases of poor network connectivity. As STalk2 uses cloud services, a caching mechanism should be implemented for the updated version.

Discovering the New Side of the Child. Ten peers answered that STalk2 was useful in finding the new side of the child in Q7. P1’s peers felt that P1’s requests, which they previously could not understand, were revealed in the study. The peers of P2 understood P2’s story better; for example, how school was that day. The peers of P3 and P5 observed their flexible actions while accommodating anxiety and changes. The peers of P4 realized that they could communicate with P4 in detail.

Continuous Use, Recommendation to Others and Overall Satisfaction. Ten peers answered that they will continue to use STalk2 in Q8. Ten peers also answered that they want others to use STalk2 in Q9. One peer answered “3” in Q8 and another also answered in Q9. They commented that literacy skills to use tablet computers were the key determinants of acceptability, especially if the child could interact by using nonaided communication on some level. Finally, all except for these peers answered that STalk2 satisfied their needs.

5.4 Implications

In this study, we examined the effects of applying persuasive technology to the mobile AAC app, STalk2. The frequency of presenting visual aids with verbal messages was increased as intended. The simple steps for creating, searching, editing, and presenting visual aids reduced the burden of peers. Additionally, the problem behaviors of a few children were decreased. The reduction of such behaviors made the abilities required for AAC easier. STalk2 also motivated peers by helping them discover new sides of the child; the children also enjoyed using the app. These results suggest that behaviors and attitudes in AAC may be modeled by the dimensions of abilities and motivation as Fogg proposed [8]. Future AAC apps should take into account the principles of persuasive technologies facilitating communication through better design.

5.5 Limitations

This study was conducted on eleven communication peers and five children with CCN. Although it is generally acceptable to have 5–10 participants in research focusing on individuals with disabilities [16], our participants used STalk2 within a limited range of communication needs and settings. Thus, the results may not be generalized to other cases. Future research should replicate this study; for example, with a larger number of participants and/or over an extended period, to cover diverse needs and settings.

We have implemented functions to support the self-monitoring principle. However, these functions were rarely used in this study. Evaluations should be conducted to examine the effects of these functions.

6 Conclusion

The results suggest that persuasive technology in AAC systems may be effective in improving communication behaviors and attitudes. Our future work will be to conduct in-depth observations and/or distributed research [16] to provide more substantial evidence for our results or findings.