Keywords

1 Introduction

Autism Spectrum Disorders (ASD) affect 1 in 68 children in the United States [1]. ASD consists of a range of developmental disorders characterized by deficits in social skills and communications [2]. Many studies have been investigating social skills and communications of children with ASD [3]. However, access to trained autism clinicians is limited and cost associated with traditional therapies is significant. The use of technology, such as virtual environments, may provide an alternative approach to ASD intervention that is more scalable and accessible [4]. Researchers are increasingly exploring virtual environments as potential intervention platforms for children with ASD in their social skills, including facial expression recognition [5] and communications in social tasks [6]. To date, however, preprogrammed and/or confederate avatars have mostly been used in such environments with only one user, which are unrealistic and inflexible [7].

Another paradigm with potential use in ASD intervention can be Collaborative Virtual Environments (CVE). CVEs are distributed, multiplayer involved virtual environments [8], and are more realistic and flexible than traditional virtual environment by allowing interactions and communications among players within the controlled environments. Controlled tasks could be designed within such environments that may ultimately enhance interaction and communication skills of children with ASD without relying on preprogrammed or facilitated interaction. Previous studies using CVEs for autism intervention were PC based environments with limitations in data analysis [9] and lacked simultaneous interaction [10]. The existing multiplayer video games aiming at entertainment and limited in feedback were not useful for educational or skill training purposes for children with ASD. Therefore, we designed a CVE on the Android platform for ASD intervention with the goal to understand and ultimately promote the collaborative interactions and communications of children with ASD.

Because the use of mobile devices is growing exponentially, mobile applications have the potential to increasingly engage children with ASD [11] by creating ubiquitous learning environments [12]. The studies using mobile applications for ASD intervention include emotion recognition [13], social interactions [14], and vocabulary learning [15]. However, these mobile applications are limited in interactions and communications between real people. Our mobile CVE (Fig. 1)—supporting multiple players’ interactions and communications in the shared collaborative environment—investigated the collaborative interactions and communications of children with ASD.

Fig. 1.
figure 1

Two players using the mobile CVE

Full size image

The goals of the current research were to: (1) evaluate the usability of the mobile CVE; and (2) investigate the collaborative interactions and the communications of children with ASD when playing puzzle games with their TD peers. This novel environment supports following functionalities:

  • Interaction between two geographically distributed players via internet

  • Audio and video communication

  • Automatic performance and audio recording

2 Method

The mobile CVE was designed with Unity3D (www.unity3d.com). The environment, which can be accessed by players using Android mobile devices, allows two players to interact and communicate with each other remotely. A variety of puzzle games were developed that compelled interactions between two players, such as turn-taking, information sharing, and enforced collaboration. The environment supported video and audio communication between the two players. The performance status and dialogue of players were recorded by the CVE system for offline analysis. In order to support these functionalities, a software framework (Fig. 2) was developed with four modules: a game controller module, a network connection module, a communication module, and a data recording module. The game controller module implemented the application logic of the puzzle games. The audio/video chat between two geographically distributed players was supported through the communication module. The network connection module was responsible for transferring the data from game controller and communication modules via the internet. The data recording module recorded locally information related to how players played the game and how they communicated with each other. The functionality and the usability of this mobile CVE has been evaluated by a small user study involving five pairs of players, each consisting of one child with ASD and one TD child.

Fig. 2.
figure 2

The framework of our mobile CVE

Full size image

2.1 System Design

Game Controller.

The logic of the mobile CVE was implemented based on a hierarchical and concurrent Finite State Machine (FSM) model, shown in Fig. 3. The element of concurrency in the design made it possible for players to reside in different sub-states while still maintaining application synchronization. Take the ‘Introduction’ state for instance: player1 could stay in the ‘Wait_P1’ state after he/she finished reading the introduction, while player2 could still read the introduction in the ‘Introduction_P2’ state. After both players finished reading the introduction, their game states would be rejoined upon exiting the ‘Introduction’ state. In Fig. 3, M is the total number of puzzle games to be played in a session, which is customizable for different requirements (will be discussed in Sect. 2.2). N is the total steps of one game, which can vary from game to game.

Fig. 3.
figure 3

The finite state machine model of the mobile CVE

Full size image

The hierarchical ‘Game_i’ state defined the logic of each puzzle game in the CVE. In order to study the different collaborative interactions of children with ASD, three types of puzzle games were designed in the CVE: turn-taking games, information sharing games, and enforced collaboration games. In the turn-taking games, players had alternating control of blocks movement. Information sharing games asked players to share color information in order to move the correct blocks. The enforced collaboration games aimed to impose simultaneous interactions of two players. Both players had to move the same block in the same direction at the same time in the enforced collaboration games. A considerable amount of communication between the players was required in the enforced collaboration games.

Each game was composed of multiple steps. The steps were generalized and modeled using the ‘Step_j’ state in Fig. 3. The implementation of the steps were different depending on the configuration of the following parameters: (1) color visibility (i.e., which players can see the color); (2) block controllability (i.e., which players can move the block); and (3) block rotatability (i.e., which players can rotate the block). The values of these parameters determined the types of game interaction, shown in Table 1. Game 1 was a turn-taking game, in which two players, P1 and P2, were both able to see the color of blocks. At the first step, player P1 had control of all the blocks. At the next step, P2 had control of all the blocks. They would take turns dragging the block in game 1. The number of steps in each game was seven in all but the last game, which had ten steps.

Table 1. Game configuration parameters
Full size table

Network Connection.

A client-server network architecture was used for the mobile CVE involving two players. The device of one player acted as a server, while the other player’s device acted as a client. All the computationally-intensive tasks were implemented on the server side. This network architecture has been widely used for two-player mobile games and it is simple and sufficient [16]. The network connection was created via Unity Master Server (UMS) (http://docs.unity3d.com/Manual/net-MasterServer.html), which allowed players to find each other at any time and at any location. The function of the UMS for network connection is shown in Fig. 4. Any player can initiate a game as a server, for example the ‘Server A’ or the ‘Server B’ in Fig. 4, by clicking the ‘start game’ button in the game. The device information (IP address and port number) were registered with the UMS. Other players can connect to one of the active servers, thus becoming a client. The client can send connection requests by clicking the ‘connect’ button in the game. After receiving this request, the UMS then returns a list of active servers to the client. The returned list contains all pertinent information required for the client to connect to the server. The client then selects one server from the list to create a connection with the selected server.

Fig. 4.
figure 4

Unity master server (UMS) role

Full size image

Communication.

The communication module was designed to support the audio and video communication between the players. The video chat functionality allowed two players to see each other in real time, which was implemented following the procedure in Fig. 5. The image was captured by the mobile camera in ARGB32 format. The captured image was then encoded into a JPG file, which was amenable to network transfer because it had a small data size and was in serialized format. The Unity remote procedure call (http://docs.unity3d.com/Manual/net-RPCDetails.html) was used for the image data transferring. When the receiving mobile device obtained the transferred data, it was decoded into the RGB24 format and displayed using the Texture2D component (http://docs.unity3d.com/ScriptReference/Texture2D.html). The video was updated with a fixed frequency 12 Hz.

Fig. 5.
figure 5

The video chat procedure

Full size image

The audio chat feature was used for players to talk with each other in real time. This feature was implemented with a procedure similar to the video chat. With both a speaker and microphone on the device, there was an audio feedback problem during the audio chat. Head phones were worn by all players to solve this audio feedback problem.

Data Recording.

The data recording module records the performance and dialogue data of players in the CVE for the offline analysis. The recorded performance metrics included the success frequency (how many times they succeed per game), game duration, and collaborative movement duration, which were written to a file in real time. The audio data of each player were recorded locally with a frequency of 44.1 KHz. The recorded audio data were written to a file at the very end of all games, and were later transcribed and labeled with some predefined utterance types manually. The utterance types were defined referring to previous literature to evaluate the communication [17, 18], including the words spoken per minute, the frequency of question-asking, the frequency of response, the frequency of spontaneous information sharing, and the frequency social oriented utterance.

2.2 Experiment Setup

Five age- and gender-matched pairs of subjects were recruited for a preliminary evaluation of the mobile CVE. Each pair was composed of one child with ASD and one TD child. All the children with ASD had a clinical diagnosis of ASD from a licensed clinical psychologist. The Social Responsiveness Scale, second edition (SRS-2) [19] and Social Communication Questionnaire Lifetime Total Score (SCQ) [20] were completed by for a parent of each child with ASD. The experiments were approved by the Vanderbilt University Institutional Review Board (IRB). The information of all the subjects are summarized in the Table 2.

Table 2. Subject characteristics
Full size table

During the experiment, subjects in a pair sat separately in two different rooms. The layout of the experiment rooms is shown in Fig. 6 (left). One tablet, Nexus 9, and one set of headphones were provided for each subject for the experiment. A video camera recorded the subject and the device during the experiment. Each experiment lasted about 40 min. The experiment procedure is shown in Fig. 6 (right). At the beginning, the devices were given to each subject. Then, subjects completed the pre-test, with one turn-taking game (game 8), and two enforced collaboration games (game 6 and game 7). During the core task, seven puzzle games, from game 1 to game 7, were presented in order. The post-test included the same games as the pre-test. After the post-test, subjects completed a survey regarding their experience with the system and with their partners.

Fig. 6.
figure 6

The experiment room layout (left) and the experiment procedure (right).

Full size image

3 Results

3.1 The System Usability

Five pairs completed the preliminary study. One pair had difficulty to finish the experiment and their data were excluded from analysis. The other four pairs completed the experiment. The network connection was lost during one pre-test. The automatically recorded performance and audio data of game 7 in this pre-test were also lost. However, we recovered the audio data by extracting the audio from the recorded video. The turn-taking game (game 8) and the enforced collaboration games (game 6 and game 7) were analyzed separately since different games required different interactions. Four pairs’ data were used for the turn-taking game, the survey analysis, and the communication analysis of the enforced collaboration games. Only three pairs’ data were used for the performance analysis in the enforced collaboration games. Even though the video chat was implemented in our mobile CVE, it was disabled during the experiment because it caused data loss and lag. Instead, audio communication was used during the experiment.

3.2 Data Analysis

All pairs showed an improved performance during the post-test compared to the pre-test. This meant all pairs succeeded more in both the turn-taking game (game 8), and the enforced collaboration games (game 6 and game 7). In addition, all pairs required less time to finish both turn-taking game, and enforced collaboration games in the post-test. All pairs also demonstrated an increased collaborative movement duration, which was defined as the duration of time players moved blocks together. The increased collaborative duration may indicate improved collaboration between the players. The mean and standard deviation (Std) of all these changes are listed in Table 3. The first row shows that the success frequency increased by 3.25 s on average in game 8, which allowed a maximum of 10 successes, with a standard deviation 1.7.

Table 3. Performance changes from pre-test to post-test
Full size table

For the communication data, we found some differences between the children with ASD and their TD peers during the pre-test in the enforced collaboration games. In the pre-test, all the children with ASD spoke fewer words than their TD partners in the enforced collaboration games. They also asked fewer questions, but gave more responses, compared to their TD partners in the pre-test of the enforced collaboration games. The mean and standard deviation of the absolute value of these differences are listed in Table 4. However, these differences were not observed during the post-test. The changes from the pre-test to the post-test in terms of the communication variables were not found.

Table 4. Magnitude change in communication from pre-test to post-test
Full size table

The survey from the subjects reflected positive opinions for the environment. From the survey, all subjects enjoyed playing the games. All subjects perceived an improved individual performance in playing the games and increased ease in talking with their partners at the end of the experiments. Each question in the survey was scored on a 5-Likert scale. In terms of enjoying the game, number 1 indicated no enjoyment at all, while number 5 meant very much enjoyment. In terms of their performance in the game and the communication with their partners, number 1 indicated performance/communication became much worse at the end of the experiment, while number 5 meant performance/communication became much better at the end of the experiment. The mean and standard deviation of the survey questions are shown in Table 5.

Table 5. Enjoyment in games, improved individual performance and Increased ease of communication with partner
Full size table

4 Conclusions and Future Works

This paper discusses the design of a CVE on the Android platform for ASD intervention with the goal to improve the collaborative interaction and communications of children with ASD. The environment facilitates the interaction and communication of two players from different locations by playing multiple block games.

Five ASD/TD pairs participated in the preliminary study. The usability of the environment and its functionalities, including the two players’ interactions, audio communications, and data recording, have been validated by the preliminary study. The result of the experiments may support the potential of the environment in improving the collaborative interactions and communications of children with ASD.

There were some limitations in the system design and user study in this paper. Only a small sample size were included in our current study. More subjects will be involved for the experiments in the future. The current CVE was evaluated in a small local network, which will be extended to the global network in order for players from any location in the world to gain access. Future work will consist of a closed-loop system using targeted feedback based on the player’s communication and collaborative performance.