Keywords

1 Introduction and Background

Children with autism spectrum disorders (ASD) usually have relatively stronger visual and memory skill but weaker verbal ability; hence, visual cues are especially important in their language learning [14]. Previous researches have shown that children’s familiarity with therapist/examiner and task settings has been crucial in reducing their anxiety [3, 16, 18], which is typically less outwardly visible even to parents and special education teachers [13]. However, very little previous empirical evidence directly examine this issue according to a recent survey by Virnes et al. [20], who argued that more ‘flexible and individually adaptable learning environments’ for children with ASD are highly desired ([20], p. 22).

Those said, we posit that using familiar image (photos) as learning material would ease the children’s learning process in a highly customizable mobile learning environment. In addition, previous studies have proven that parent’s response (verbal comments) to the child’s focus of attention will positively impact the child’s language skill development, especially for children with ASD [10]. However, many parents of autistic children in China may not have much time to be with their child due to economic pressure and expensive educational cost for children with ASD, a phenomenon we also observed during our numerous testing sessions in the past year. In such cases, most children are under care of their grandparents who may not speak the official Chinese language (there are more than 100 vigorous dialects used in China [6]), while it is the mandatory language used at school. Therefore, most children are expected to learn both Chinese and their local dialect at the same time, which increases the burden for those with language-learning impairment.

For the above reasons, we proposed a lightweight mobile learning application that support customization of learning material based on the children’s needs. The material could be customized by parents following instructions from special education teachers, and used anytime by the children for both reinforcement and entertainment purposes, which, hence, extending the intervention beyond regular schooling time. For the sake of flexibility, the language used to compile the material could be either Chinese or local dialect (or other languages).

Although our target users are children with ASD, those with dyslexia or typically developing ones could also use our application. However, Chinese children with dyslexia may exhibit visuospatial deficit [17]; hence, may not take full advantage of our application because it relies heavily on audio-visual contents. Indeed, this application may be used for dyslexic English (or other non-logographic language) learners or other typically developing children.

In the next section we will discuss our design approach. Then, we will describe our prototype in Sect. 3 and discuss our (future) work in Sect. 4. Finally, Sect. 5 concludes our work.

2 Application Design

For young children (toddlers or preschoolers), picture-word cards (flashcards) are commonly used to improve their vocabulary and object recognition, for examples, animals, plants, foods, etc. For older children, it could be expanded for more complicated language learning, for example using a picture word inductive model (PWIM) [2]. In our current design, we focus on object recognition, because it is a common training task for very young children with ASD.

2.1 Design Rationale

Some well-known characteristics of children with ASD include repetitive behavior and difficulty to generalize learned skills. Allen et al. [1] have shown that the medium of presentation, whether using iPad or picture books, does not contribute differently to the children’s word-learning outcomes. Other studies have also reported insignificant benefit of using iPad to the learning outcome, for instance [8] reports that children did not gain significant joint-attention skills after training, and [5] shows that teacher’s intervention is more effective than using iPad. However, the results do not undermine the potential benefit of technology-based intervention (TBI), because most authors agree that it will motivate children’s engagement and some positive results have been reported in the learning of other skills. Hence, our design goal is to exploit technology features such as to personalize the learning contents to ease children’s learning process, because using familiar context is highly desirable [20].

Although many similar language-learning applications have been developed for children with ASD [7], TBI at Chinese home is rare, because some cultural, developmental and environmental settings unique to this population may alter the effectiveness of interventions [19]. Given the scarcity of such applications, involving parents/teachers to co-develop such application could be the best temporary solution. However, the application cannot be a pure intervention (educational) application; instead, it would be blended with entertainment element so that both parents and children may use the application without pressure.

In addition to its usefulness, the proposed application must be affordable and user friendly. Hence, the following four properties highlight the contribution of our work:

  • Minimizing anxiety: the application must use familiar objects and voices and it should be entertaining enough for children; hence, should be in a form of mobile learning game. To further reduce anxiety, it should be able to introduce the least unfamiliar items to children, for example, items commonly used by the children at home or school, or items used by their parents or seen daily.

  • Affordable: the application must be affordable with no or little additional cost for most parents. Since most Chinese families have mobile phone, no additional cost is needed.

  • Flexible and highly customizable: just like many other mobile apps, our application can be played anywhere anytime, which increases the children learning opportunity. Although the application is designed as a media for parent-child interaction, it could also be used by teachers to reinforce children’s learning. Parents/teachers could customize the learning contents according to children’s learning progress. Therefore, the application should be easily altered for word learning, stimulus generalization, or even social-skill learning.

  • Effective: Since visual cues have been proven effective for language learning among children with ASD, the proposed application, which digitizes visual cues from a printed paper to a mobile phone screen, shall also be effective.

2.2 Related Applications

With respect to its customizable feature, there are some commercial applications similar to ours, for examples “AlphaBaby” by Little Potato Software, “Game Factory HD” by Bacciz, and “Kids Games – Photo Touch Food” by Grasshopper Apps. In these apps, users are allowed to add or replace some pictures and sounds used in the apps. For other flashcard applications, most of them are less customizable; hence we will only compare our application with customizable ones. Table 1 shows the comparison results.

Table 1. A comparison between our proposed application and similar ones
Full size table

The main differences between our application and others are in the arrangement of learning content and topics. In our application, each learning page consists of a photo of one or more items; for example, a photo of living room with sofa, desk lamp, and TV set, or a photo of table with apples, oranges, jugs and cups. In this case, our application will preserve the items’ spatial context. Conversely, in other applications, each learning page consists of multiple photos or images in which each of them represents an item, hence, losing their spatial information.

With respect to the learning topics, our application is intended to support teachers and parents to design their own learning topics without any predefined ones. In contrast, many commercial applications are designed for learning specific topics; hence, less customizable.

2.3 Design Approach

User-centered and participatory design with proxy (teachers and parents) is adopted here. Although the main stakeholders are children, teachers will play the most important role to decide the learning contents (tasks), and parents will be the content creators. Therefore, both teachers’ and parents’ feedback is solicited during the design process.

3 Items Recognition for Children with Autism (IRCA)

An exemplary usage scenario of our application is as follow: first, a teacher assigns a new task to her student’s parent to capture an image, for example “apples and oranges”. Then, the parent captures the photo of an apple and an orange, types in the word “apple”, highlights its area in the photo, and records his voice for “apple”, and repeat it for the orange. Then, he will pass the game to his child who will play the game by tapping either the apple or the orange. If s/he tapped the apple, the word “apple” will be shown and his/her parent’s voice for “apple” will be played. Only after both “apple” and “orange” are tapped, a reward will be provided.

One may easily reduce the scenario above by replacing parent(s) with teacher(s), in which all learning materials will be made by the teachers without involving the parents. However, since our application is designed to promote parent-child activities, we will consider the involvement of parent throughout our design.

3.1 Functionalities

The proposed application comes with two user modes: parent mode and child mode, both using the same device. Figure 1 illustrates the necessary activities in the parent mode, while Fig. 2 illustrates those of child mode. When an object is tapped in the child mode, its name will be shown along with the recorded voice. Children are allowed to tap the same item repeatedly. In order to encourage children to use the application, some rewards may be provided after all items were tapped for certain times.

Fig. 1.
figure 1

The procedure to prepare learning material

Full size image
Fig. 2.
figure 2

The gameplay procedure

Full size image

3.2 Architecture

Figure 3 illustrates some basic modules used in this application with extended part being in dashes rectangles.

Fig. 3.
figure 3

Some basic modules of the prototype

Full size image

In current prototype, the reward system has not been decided yet. It could be a simple praise, animated objects, or something more appealing for each individual learner. In the future, we plan to connect the database to the cloud; hence, increasing the amount of memory capacity to store learning contents.

3.3 Prototype

The current prototype is developed using Swift 3.0 for iOS (iPhone and iPad) with support for both English and Chinese. The functionalities in both iPhone and iPad versions are similar at this moment. Figure 4 shows screenshots during content creation in iPhone. The left figure shows the image after being captured and the right figure shows the image after the cup was shaded (colored).

Fig. 4.
figure 4

The prototype showing the original (left) and shaded (right) photos during content creation process (Color figure online)

Full size image

A short demo is available at: https://www.youtube.com/watch?v=ocbzTdIGI-Q.

3.4 Pilot Testing and Results

A short pilot testing is conducted at a private education center for children with both with high-functioning and low-functioning ASD (see Fig. 5 for the testing moment). The primary purpose of the testing is to obtain feedback from teachers.

Fig. 5.
figure 5

The testing moment using iPhone (left) and iPad (right)

Full size image

According to the principal teacher, all children in the Center have used mobile devices (Android or iOS devices) in the Center or home before. Most of them have also played various mobile games and learning apps, although they are not particularly designed for children with ASD. Given the nature of common games/apps in China, most parents would consider them as toys, not learning tools.

Some feedback/comments provided by the teachers after trying the application are as follow:

  • The proposed application is easy to use;

  • Using parents’ voice is good, especially when they want to teach their child some local dialect;

  • The application worked well to record most of indoor scenes in the Center;

  • For older children, it would be better if the app can be used to record video in addition to still photographs;

  • There is a concern that some (grand-) parents may not be able to type in the item names correctly; hence, a dictionary list must be provided by the teacher beforehand.

4 Discussions and Future Work

Based on our initial pilot testing we believe the main difficulties of promoting our application lie on parents’ skepticism toward TBI, especially when it is in the form of a mobile game. Teachers, on the other hand, are very optimistic toward the use of it. Therefore, some modifications should be made by expanding the functionalities of the game to emphasize its educational purposes.

First, the application must allow teachers to set different game rules. For a pure word-learning game, the recorded voices would simply be the pronunciation of each word [15]. In this case, children may be allowed to select (tap) each object multiple times. Objects will be presented according to predefined categories, for example, items on the dining table, furniture in living room, etc. For the game purpose, not all objects in a photo may be associated with an audio source (spelling the word of the object).

Second, for the training of self-initiated speech, a gradual time-delay shall be introduced between the object selection and voice play [12]. In this case, the child must be allowed to select the same object repeatedly but the voice acts as a prompt which will be played after a certain time delay; for example, it will be played immediately for the first-time selection, but played after a second delay for the second-time selection, and two seconds delay for the third-time, and so on. A speech-recognition module may be needed to record the child’s voice for further analysis.

Third, for strengthening future language comprehension and production, the parent’s voice must be verbal comments to the child’s choice (focus of attention) [10]. For example, if the child taps a “cookie”, the voice could be “Hi sweetie, that is the cookie we usually eat in the morning, right?”

Finally, for stimulus generalization, various photos of the same concept must be presented. For example, to teach the concept “apple”, various varieties of apples such as Gala, Fuji, Granny Smith, Red Delicious, etc. must be shown either in a photo or multiple photos. For more complicated concepts, similar items must be shown in multiple photos in random order.

5 Concluding Remarks

The level of awareness toward autism in China is lower than that in developed countries due to some social and cultural barriers [9, 11]. For example, the first systematic prevalence study on autism in China was conducted in 2013 [4, 21]. Hence, affordable and portable technological solutions could offer great helps to families living in ASD, which motivates our study.

It is noted that based on a survey of the technology-based research articles from 2000 to 2010 on ASD [20], only 7% targeted language and conversation skills, compared with 31% on social and socio-emotional skills. Our application, the first of this kind in mainland China, could offer initial yet valuable insights into the research and development of such portable, customizable and affordable application along this path.