Keywords

1 Introduction

Children with ASD commonly have difficulties with social interaction and communication; hallmarks of impaired social skills (American Psychiatric Association 2013; World Health Organization 2018). Furthermore, a lack both imagination and abstract thinking, coupled with repeated behaviors, constitute the core disability of children with ASD (Boelte and Hallmayer 2013). Typical social deficiencies consist of failures to properly respond to greetings, incapability of understanding non-verbal social cues, and inability to gaze into other people’s eyes for emotional communication (Lee et al. 2018). Such social deficiencies make it difficult for children with ASD to socialize with others and integrate themselves into social activities involving their peers (Fodstad et al. 2009). Bauminger and Kasari (2000) suggested that compared with TD children, children with ASD are less able to socialize with others and involve themselves in group activities; nevertheless, these children still desire interactions with others. However, because of their innate social deficiencies, children with ASD are prone to fear and anxiety in the face of social interactions in life (White et al. 2009). Consequently, children with ASD are also more likely to suffer from social isolation and loneliness (White and Roberson-Nay 2009). However, even as children with ASD grow older, such deficiencies in their social skills fail to improve. This fact brought them setbacks in social interactions, and then may allow children with ASD to be afraid or reluctant to come into contact with others (White et al. 2009). Or even the fact may cause their avoiding social interaction with others (Bellini et al. 2007). Finally, such influences may give rise to some inappropriate behaviors, such as impolite behaviors due to their failure in correctly judge others’ social body movements (Lee et al. 2018). As such, enhancing social skills is one of the most important improvement goals for children with ASD (DiGennaro Reed et al. 2011).

1.1 Current Training Methods for Teaching Social Reciprocity Skills

Recent studies have found that if children with ASD did not train their social skills at an early age, it would be more challenging for them to socialize with others in the face of more complex social situations in the future (Mundy and Crowson 1997). Or, these children might not be able to establish profound and positive social contacts and friendships with others (Williams White et al. 2007). Therefore, one direct way of helping children with ASD improve their social skills is to develop strategies of social skills training which offers more frequent practice and presents authentic events and situations (Jarrold et al. 2013). Such training will contribute to increasing opportunities for children with ASD to more extensively interact with others in different social situations. In addition, the training can allow children with ASD to make judgments and make appropriate social responses when encountering different social situations. Therefore, in order to improve social reciprocal behaviors of children with ASD, therapists or special education experts tend to use the social story method, or adopt teaching strategies such as role-playing so that children with ASD can imitate and understand social interactions in different situations (Parsons and Mitchell 2002).

1.2 Primary Problem with Current Training

However, previous studies have proposed different views. Bellini et al. (2007) argued that the traditional role-playing strategy difficult to interest children with ASD, which meant that these children tended to feel bored and lose patience. Compared with the traditional role playing method, training in a real-world environment with interactive games would enhance the learning motivation of children with ASD and generate a superior training effect. Therefore, related studies have suggested that educators should conduct more extensive and diversified social skills intervention training through games and other interactive modes in various authentic environments (Jarrold et al. 2013). This recommendation is important for children with ASD, because such children may have severe difficulty in transferring their application of social skills from one situation to another one. When confronting a new situation and new social objects, children with ASD often feel overwhelmed. Therefore, social training for children with ASD must be conducted in a more natural and straight state (Jarrold et al. 2013), which was more effective than treatments in statically defined and rigid classroom environments (Bellini et al. 2007). As a result, this study is aimed at decomposing complex social behaviors into visual clues, specific rules of games, and steps of interaction conducted under visual, tangible and structured social interaction scenarios. Thus, it is indispensable to translate abstract social concepts into practical social operational tasks and social interaction processes (Krasny et al. 2003). Meanwhile, this study also intends to employ a social training environment produced by MR and enable children with ASD to experience many different contextual interactions in such a training environment.

2 Related Work

2.1 Benefits of AR and VR

With the advancement of technology, more researchers have begun to apply technology to the social skills training of children with ASD in terms of social interaction and emotional perception. According to a study by Parsons and Mitchell (2002), the application of assistive technology to the social training of children with ASD can boost their learning interest and provide opportunities of repeated social exercises. In contrast to traditional single role-playing and static graphic storytelling book, the rigid and inflexible classroom learning method has been unable to interest children with ASD. In this case, the application of assistive technology such as augmented reality (AR) or virtual reality (VR) is quite beneficial for children with ASD in social interaction learning (Mesa-Gresa et al. 2018), because applications of technology media can provide repeated exercises and visual pictures with a better sound-light effect, which is also more attractive for children with ASD.

Previous studies (Lee et al. 2018; Syahputra et al. 2018) have also pointed out that AR can enhance the concentration of children with ASD, and provide visual clues to focus their attention on observing non-verbal social cues. Thus, AR can facilitate the enhancement of their ability to recognize other people’s emotions and social body movements, thereby increasing their chances of successfully interacting with people and responding properly to greetings (Lee et al. 2018). Therefore, more and more studies have begun to suggest that AR can provide different social training options for children with ASD.

Compared to AR, other research has also put forward a positive view on the application of technology such as VR. The research pointed out that VR has realized social interaction and on-the-spot experience that cannot be achieved by traditional Video Modeling (VM) and the role-playing method (Parsons and Mitchell 2002). Compared with AR, VR has an advantage in assisting role-playing and in understanding abstract social concepts, because it can provide social training, training for emotional skills and daily life, and communication skills and attention training of children with ASD (Didehbani et al. 2016; Herrera et al. 2008; Ip et al. 2018; Kandalaft et al. 2013). VR is most frequently used in social skills training, because VR can not only simulate situations in daily life, but can also increase the chances of training in different contextual scripts under the control of therapists as well as in safe simulated environments (Mesa-Gresa et al. 2018). Besides, VR can simulate a variety of situations so as to allow participants to socialize with virtual characters, and can reduce the pressure of participants in case of face-to-face communication with others (Bai et al. 2015), thereby easing the concern of being rejected due to making mistakes or even conflicting with people (Didehbani et al. 2016; Maskey et al. 2014). In addition, participants are more willing to express their real ideas or reveal their emotions and behaviors to virtual characters than to real people. And further, virtual characters have the same social characteristics as real people, and can better convey social cues than pictures or videos. Moreover, VR stimuli can be used to imitate complex social situations, and social interaction with virtual characters can not only possibly influence the social behaviors of children with ASD, but also improve the bottleneck of children with ASD in social greeting training (Lee et al. 2018). In terms of instructing children with ASD how to recognize and understand other people’s social greetings and then make appropriate social responses, the application of VR can create quite intuitive status of social reciprocity which approximates real life. Therefore, related research has also found that VR technology can allow children with ASD to interact directly with characters in virtual scenes as if they were in real world, which is of great assistance for children with ASD who require real social contextual exercises (Didehbani et al. 2016).

2.2 Shortcomings and Deficiencies of AR & VR in Existing Social Training

Although AR offers different forms for children with ASD to interact with others in a more authentic environment in new ways, under the AR training platform, children with ASD can only socialize with interactive objects via a 2D screen and are also limited by the frame of the picture. However, the inability to directly interact with social objects constitutes a serious deficiency for using AR in social training. In addition, and in contrast to AR, although VR can enable users to be immersed in a virtual world and to socialize with virtual characters through a head-mounted display, users can not see the real environment and therefore fail to establish a connection to the real environment. Such a failure has an impact on the perception of children with ASD because people tend to upgrade their understanding and mastery of situations through experiencing the real environment so as to establish emotional connections and perform social interactions.

2.3 Advantages of Mixed Reality Applied to Social Reciprocity Training of Children with ASD

As a result, MR, which combines the advantages of both AR and VR technologies, constitutes a new opportunity for training children with ASD. Accurate computer calculations and image operations of MR allow virtual objects to be accurately superimposed on real objects, enabling 3D virtual characters or objects to be combined with the real environment. In this way, it can create a new MR environment that is generally visually recognized. In addition, according to studies by Cheok et al. (2002); Pan et al. (2006), MR can provide a realistic but controlled learning environment where “virtuality” coexists with reality, and at the same time, MR can produce instant interactions. Kors et al. (2016) adopted MR to allow users to play other roles and feel what such characters can feel from a first-person perspective. Dasgupta et al. (2018) believed that via MR, in which virtual structures are embedded in the physical world and interactions with virtual characters in MR can use real objects to perform social reciprocal interactions, users can feel as if situated in the new environment created by the combination of MR and real environment. The generated feelings facilitate their understanding of situations, and slowly increase their social mastering of and emotional attachment to interactive characters. Therefore, MR might be a powerful technology to help children with ASD since it allows fun and direct interactions between children with ASD and virtual characters. Additionally, it also allows children with ASD to be able to participate in role playing and interact with virtual characters from a first-person perspective, as if they were socializing with real people. Also, it can protect children from the fear of facing real people in real social environments. Such training can reduce the cognitive load on children with ASD, and the training is definitely suitable for the repeated learning of children with ASD.

3 Method

This study developed an MR system for the social-interaction training of children aged 7–9 with ASD, and is aimed at constructing a social interactive situated teaching platform. The purpose of this training platform is for instructing children with ASD to perform proper social reciprocal behaviors while interacting with others. This system design is based on discussions between a designer, therapist, and parent. And further, a therapist, an engineer and two designers were invited to participate in the development of an MR social-training system. The study also included observations and interactions with individual children with ASD in the occupational therapy center to have a better understanding of ASD cases. Based on discussions with the therapist regarding the observations, the designers and therapist of the study jointly determined the optimal system form and structure, which was then applied to develop the platform for the social training of children with ASD.

3.1 Instruments

Development of MR-Based Social Interaction Training System.

The MR-based Social Interaction Training System (MR-SITS) in this study was focused on training children with ASD to identify others’ social body movements that correspond to emotional representations (Fig. 1). In addition, this study also targeted the recognition of related social reciprocal behaviors. In addition to facial expressions, the content for training on the MR-SITS also includes understanding others’ emotional manifestations by means of body movements, and then conducting social reciprocal behaviors in response to those body movements. Therefore, the MR-SITS differs from previous training systems primarily via the use of the MR system. Indeed, MR can better strengthen the relationship between body movements and the environment, and focus on situations such as saying hello to neighbors near home, shaking hands with an old friend in the park, or waving goodbye to friends at school, instead of facial expressions or other non-verbal social cues. The MR-SITS consists of three parts: (1) design of interactive context; (2) social story scripts; and, (3) social task scheduling and social body movement identification. During training, the therapist employed the MR-SITS to conduct social training of the children and scenario simulation. Moreover, the therapist strived to familiarize the children with different social situations through the platform, thereby gradually allowing children with ASD to not only have more chances of interacting with others, but also the ability to perform appropriate social reciprocal behaviors and actions in daily life.

Fig. 1.
figure 1

MR-SITS set up

Full size image

System Development.

This study applied Unity3D for system construction and development of the MR-SITS. Unity3D is an engine supporting game development, the software of which is widely used for the production of interactive games and the construction of MR systems. In addition, the platform can construct contextual scenes and animated characters required by intervention training in interactive games. Developers mainly employ C# programming to achieve interactive presentations. The MR system uses a head-mounted display (HTC VIVE Pro) for spatial positioning and character tracking, which allows the participants to more naturally integrate themselves into the scenes and then socialize with virtual characters (Fig. 1). To create the virtual characters, this study used Iclone7, which can employ the front and lateral views of real people to create characters around training objects, such as students, teachers and unfamiliar social roles. In addition, the social body movements such as handshaking, hugging or handholding were defined based on discussions with the therapist and relevant experts, and then applied to the design of characters who will establish social reciprocal relationships with children with ASD.

Setting.

The MR environments constructed in this study consisted of homes, schools, parks, stations, and other places, which contained related furnishings or furniture. These locations were chosen because such places are often visited by school age children where they would likely need to engage in social behaviors, and so were designed as interactive contextual fields. Using the MR-SITS, children with ASD encounter virtual characters, who establish various social relationships with the children in different situations, for example their families, classmates, teachers, neighbors and grocery-store clerks, or anyone with whom they may encounter often. As for visual manifestation, based on discussions with the therapist and special education experts, the scripts set by this study deleted interference factors not related to social reciprocal behavior, and the manifested content of each event simply included: (1) contextual scenes; (2) virtual characters; (3) text script and, (4) spoken dialogue.

Design of Training Materials.

After discussing the desired scripts and scenarios with the therapist and individual parents, the study used the social-story strategy to create 15 situational story scripts. These scripts were generally inspired by children’s daily life and social experiences or inspired by social situations where children often make judgement mistakes. Then, the scripts were reviewed by two experts with experience in implementing intervening teaching of social story scripts. This study created the design of the social contextual story scripts based on the severity of ASD cases as well as the learning content. The researchers constructed different social story scripts to emphasize different learning goals, such as making new friends, integrating into groups, not interrupting others, caring for others, dealing with others’ anger, and so on. For example, in the story scripts, there may be the following circumstances: after school, my father is waving at me at the school gate; or when I arrive home, my mother is waiting for me with open arms. We simulated current contextual scripts through the MR environment, and asked the children with ASD to fulfill tasks of social judgement while interacting with virtual characters. In order to allow children with ASD to have a better understanding of relationships between emotions and social body movements, this study focused on six emotions proposed by Ekman and Friesen (1971): (1) happy; (2) sad; (3) angry; (4) fear; (5) disgust; and, (6) surprise. Interactions with people in different social relationships were also included. Simultaneously, this study also targeted the training of children with ASD for making appropriate social responses. In terms of social greetings, the study also defined ten different social greetings through discussions, including: (1) waving; (2) nodding and smiling; (3) handshaking; (4) handholding; (5) hugging; (6) head shaking; (7) bowing; (8) patting on shoulder; (9) clapping; and, (10) putting a hand on shoulders. Usually, such greetings are common in daily life in Taiwan.

System Operation and Interactive Program.

Upon using the MR-SITS (Fig. 2), (a) the user initially enters a virtual field, after which the children with ASD can choose to visit different contextual fields (such as a home, school, a store, and a parketc.). The system switches scenes based on the chosen location and displays different contextual scripts and scenarios corresponding to the background environment of different events. (b) Regarding operation, the platform encourages children with ASD to first explore their environment by walking around the permitted space (3 * 3 m2) and then searching the environment to detect clues embedded in the surroundings. (c) When the children with ASD walk into a specific space, they trigger the corresponding social story content and virtual characters to be encountered in that space. (d) These virtual characters then perform social behaviors in response to the predefined situation of the story script, (e) and prompt the children with ASD to respond by body movements and make social judgments (Fig. 3). (f) When the children with ASD perform the correct body movements and social judgments, the system immediately responds with clear visual pictures and auditory feedback, alerting the children with ASD that they gave the appropriate social reciprocity response. The system adopts different text scripts and 3D contextual character animations to provide more non-verbal visual cues for children with ASD. These visual cues are mainly aimed at helping children with ASD observe and understand the environment and situation within which they are located, as well as the causal relationship between social events. In this process, in addition to visual information such as the virtual scenes and the behaviors of the virtual characters, the platform also provides information such as event descriptions, background sounds in different occasions, character dialogues and facial expressions. It should be noted that such information does not suggest answers, but supports children in making judgements. In addition, the therapist or individual parent was asked to help the children understand and use the MR-SITS.

Fig. 2.
figure 2

Therapist teaching children how to use the MR-SITS and how to interact with the 3D role-play avatar

Full size image
Fig. 3.
figure 3

The children with ASD to respond by body movements and make social reciprocity behavior judgments

Full size image

Social Reciprocity Behavior Judgment and Social Response.

In the MR-SITS, there are different social reciprocal tasks and judgements of emotions suggested by body movements and meanings of social behaviors. With regard to tasks, the therapist applied the platform to train children with ASD to better understand others’ social body movements and emotional representations behind the body movements. Each contextual story script involved two phases of tasks. The first phase task required answers to test questions: (1) body emotions, and (2) responding to social greetings (social reciprocal behaviors). There are six options for each question, and these options were randomly selected from the aforementioned six emotions and ten social greeting behaviors defined by the researchers. The children were required to answer test questions in accordance with the system instructions before they could start the second phase, namely, role playing. The system asked the children with ASD to perform the social actions and actual social reciprocal behaviors to fulfill the role-playing tasks. As mentioned, these training situations and socially interactive behaviors were embedded within the contextual scripting of the system so as to help children with ASD improve their social interaction skills. Therefore, in each test question, the children were required to recognize emotions suggested in body movements and respond to social actions in terms of specific events. Test questions appeared during the process of social reciprocal actions and are given repeatedly, allowing the children to have numerous opportunities to make judgments and receive feedback. While socializing with others, the children must first recognize the emotions of others according to facial expressions or body movements and other emotional representations. Then, the children need to observe the body movements of others to determine what the other person desires to do, and finally, make social responses to the received information.

Field Observations and Assessments.

In the study, the items the researchers were watching for included: (1) the social body movements of the children; (2) the children’s judgments of emotions suggested in the social reciprocal behaviors of others and responses to such behaviors; and, (3) the children’s ability to understand social situations. Additionally, the observed results and status could act as a reference for more in-depth contextual design and system development.

4 Results

In order to evaluate the impact of the MR-SITS on the physical and emotional state and social reciprocal behaviors of the children with ASD, we invited researchers engaged in special education to observe and evaluate the system. Also, we had in-depth interviews with three therapists and seven special education experts. Interviews were held after the expert observations and evaluations, in which they stated that the system appeared attractive and was promising as a tool for the social training of children with ASD. They further proposed that the teaching strategies associated with MR could enhance the children’s receptivity to social training, which may support their ability to handle more complicated social situations and interactions with others. Moreover, the MR-SITS allowed children with ASD to receive training more often as well as have more opportunities to interact with different characters under different contextual scripts. And, the process of playing games on the MR-SITS platform along with the performance of children with ASD in contextual training could serve as a reference for therapists when making decisions on strengthening the training of children with ASD and determining the social interaction disorders of such children under certain circumstances. This system can provide appropriate social training courses for children with ASD at the ages of 7 to 9, since script content, personas and virtual situations were created to be of interest to them. Furthermore, with respect to the implementation of the MR-SITS, the special education experts reported that the operation process of the platform was smooth and intuitive. This platform integrates the real environment and virtual characters, thus assisting children with ASD in forming partial connections with certain circumstances and then reducing their cognitive load due to their lack of imagination.

In addition, it is noteworthy that according to the interviews and field observations, the therapist stated that the operating system design was remarkably intuitive. And she also reported that the system had the function of quickly switching the scenes and options, the functionality of which was aimed at helping the therapist control the training process and reduces the system failures caused by improper operation by the children. In addition, seven special education experts agreed that the MR-SITS was interesting and interactive. Feeling a similar interest, many children with ASD were willing to take the initiative to interact with 3D virtual characters; some children would approach and observe the virtual characters, and even reached out and tried to touch them. Accordingly, they also believed that such a system could indeed attract children’s attention and boost their enthusiasm to learn. However, two experts mentioned that if a head-mounted display were worn for a long period of time, it may cause dizziness and discomfort for the children. Further, the therapist proposed that the manual controllers could be improved because the existing Vive controllers could not be used for performing certain gestures, such as waving hands, clapping out and patting on the back. Indeed, such gestures were not intuitive with the current operators. In addition, the special education experts reported that children with ASD are generally sensitive to light and sound, and therefore suggested adding a few optional buttons to control the brightness, hue, and audio volume so that the training conditions could be quickly adjusted to better accommodate the needs of each child with ASD. The special education experts also advised that the images of the virtual characters should conform to the background of the event, because the repeated training would make the children associate the story script with the corresponding role and retain such associations in mind. Put another way, such associations may affect the child’s first impression of anyone they meet in the future. Such factors must be considered carefully in future research to avoid negative impacts.

5 Discussion

The MR-SITS is a tool developed for assisting therapists in conducting social training of children with ASD, not replacing therapy. In addition, the MR-SITS is expected to enhance the ability of children with ASD to understand the emotions reflected in others’ social body movements, recognize social interaction behaviors, and improve social skills. The advantages of MR in social training are listed as follows.

5.1 Realistic Situations and Flexible Teaching Strategies

The semi-immersive MR-SITS reinterprets previous socializing teaching strategies in certain innovative ways, such as social stories, role playing while reading story books, and VM. It allows the training of children with ASD to occur in any space, and the sensors, head-mounted display and handle can be used to easily simulate and reproduce any situation and character so as to enable children with ASD to interact with others. The MR environment can connect the virtual environment with the real environment, allowing children with ASD feel immersed and in direct interaction with virtual characters. Differing from previous rigid learning methods and monotonous training, with such rewards and game feedback, the MR-SITS may become a visual media and training platform that attracts children with ASD to participate in the training.

5.2 Assisting the Children with ASD in Dealing with Diversified Social Situations and Making Immediate Responses

The MR-SITS focuses on social relationships and social interactive events, and the children are prompted to complete different social tasks within different social relationships. In the MR environment, the children were encouraged to repeatedly interact with a wide variety of virtual characters created for the MR-SITS and establish simple social relationships with them. When a child wears the head-mounted display, he or she sees virtual situations, virtual objects, virtual characters, and text of the situation scripts. What he or she sees can jointly simulate authentic situations, so that the child can observe body movements and facial expressions of virtual characters from different perspectives. Also, the MR-SITS gives children an opportunity to correct their mistakes. Moreover, a child need not worry about being hurt in a virtual simulation; as such, he or she will likely be more willing to approach that which they are reluctant to touch and have reduced fear of socializing with people.

5.3 MR Can Quickly and Easily Restore Social Contextual Events and Selectively Strengthen the Social Training Required in Each Case

The MR-SITS permits quick and easy changes of scenes and characters, and such a training environment can satisfy various contextual needs of therapists in social training. Simultaneously, children with ASD can see the actual therapist in the MR, which allows the therapist to assist the children in training for social skills. This enhances the sense of safety of children with ASD, and also allows therapists to provide assistance and instruction any time. In addition, the easy implementation of the system can make it easy for special education experts, therapist, and individual parents to conduct such training for children with ASD to simply and intuitively perform physical and emotional representations, recognitions of non-verbal clues and other social skills. The MR-SITS identifies completed levels and tracks scoring, which can be provided to the therapist and individual families as evaluation data. In addition, the scores could be used to better understand what needs strengthening in each case, and then for the intensification of social training of weak parts.

6 Conclusions

This study concerned only the preliminary design of the MR-SITS, which is set to be more focused on social skills training. The system created virtual characters with which children with ASD can interact. Such a design, namely that the MR system integrates virtual characters into different situations, is an innovative approach used in the social training of children with ASD. In the future, we will exploit the visual advantages of MR and combine these advantages with multi-sensory experiences to create more complete and attractive training content. Finally, we hoped that this study can inspire other studies and bring a new research area and opportunity to studies concerning ASD.