Keywords

1 Background

1.1 Children with ASD

Children with ASD tend to have difficulties in the areas of language and communication, and it is therefore only appropriate that these characteristics are critical elements for diagnosing ASD [1]. Delays in language and communication of children with ASD occur in the areas of speech articulation, word use, syntax, comprehension, and pragmatics [2]. Depending on the source consulted, between 20% and about 60% of the population of individuals with ASD are estimated to fail to develop functional speech skills [3, 4].

The expressive language difficulties of children with ASD have been met with considerable intervention efforts [5,6,7,8,9]. Receptive language issues, on the other hand, have received relatively less attention despite considerable difficulties [10]. This includes the understanding prepositions. Its becomes evident when observing how frequently this skill is needed on a daily basis [11]. For example, a teacher might (a) refer to the location of an object (e.g., your coat is next to the green one), (b) state where an object should be placed (e.g., put your lunch box on the shelf), and (c) instruct the child where to position herself (e.g., stand behind Oliver). Some children with ASD also have difficulties with transitioning between activities and following step-by-step procedures toward task completion. Hence, they may benefit from visual schedules and task instructions [12]. In general, it has been recognized that children with ASD often prefer and greatly benefit from instruction in the visual modality [13].

1.2 Augmented Reality

Advances in technology have resulted in AR as perhaps the latest and most exciting new approach to instruction, including the supplementing of spoken instruction to children with ASD [14]. Unlike virtual reality, AR is directly related to reality and mixes aspects of reality (e.g., objects and containers on the table top that need to be put in relation to one another based on a prepositional phrase; e.g., “Put the ball in the cup”) with computer-generated information (a visual illustration of the action that can be called up by holding the iPad over the object and container). As a result, some argue that AR does “not require as much capacity for abstraction as VR and people with autism who do not have abstraction capacity could benefit from their use” [15]. AR application have been developed and studied in children with ASD in several ways [16,17,18,19], but none have focused on receptive language skills which is targeted here.

2 Development Process

2.1 Definition of Strategy and Discovery

The AR experience developed by PTC with Boston Children’s Hospital (BCH) was a collaborative effort to provide a working tool for use in studies regarding the effectiveness of Augmented Reality as a learning aid for children with ASD.

The development process used was the Design Thinking Process. This process began with the definition of the strategy as stated previously. This was followed by a discovery research workstream in which a variety of stakeholders were interviewed including the Boston Children’s Hospital clinicians from the Autism Language Program, 3 parents of children with ASD, 3 special education teachers and 1 speech-language pathologist. These interviews were supplemented with online research into the effectiveness of techniques for teaching and learning with autistic children, such as the use of video training and other interactive methods. In conclusion of the discovery, the key insights gained were that:

  • There is a wide range of ASD that is not in a linear scale, instead there are many dimensions

  • Age is defined as chronological and development age. This program would focus on development ages of 3–5 years old

  • Six common problems are encountered for children with ASD:

    • Learning to do basic things in their daily life

    • Transferring a skill to a new context

    • Attention

    • Learning language & expressing themselves

    • Social interactions

    • Changes in routine/knowing what to expect

  • Additional insights:

    • Multi-sensory learning and limitations

    • Need for motivation

    • Move quickly from one thing to another

    • May need longer time to process

    • Take things literally (piece of cake)

    • Parents are the heroes

2.2 Design Workshop

Design thinking principles were next used to conduct a Define Workshop to identify the best set of user stories that would provide the best outcome. These stories were then reviewed and validated with several groups including BCH clinicians, parents of children with ASD and technology specialist at PTC. Upon determination of the specific user story, a script was built to illustrate the process through which the clinician and child would interact with physical objects in combination with the digital representation in the AR experience.

Key design elements:

  • Must be usable by parents in a non-clinical setting

  • Parents have limited time

  • Limited transferable content from one individual to another

  • Treatment plan is not always executed correctly

  • Parents do not know what will work for their child

As a result, three high level concepts were developed (see Fig. 1).

Fig. 1.
figure 1

The AR Buddy, a Visual schedule and tasks instructions, and a Basic terms app.

Full size image

For each of these concepts a visual story board was produced describing both the interaction as well as the actions within the app to map that action (see Fig. 2).

Fig. 2.
figure 2

Visual story boards for each of the three concepts.

Full size image

Each of these concepts was then validated with the clinical team at the BCH based on three key identified measures of success:

  1. 1.

    The solution holds the child’s attention

  2. 2.

    The child can perform the task with assistance of AR

  3. 3.

    The child can eventually perform the task without the aid of AR

This early validation process saved several iterations and thus development costs. The basic terms app was chosen as the right AR experience to develop for phase 1.

2.3 Visualization Process

Continuing with Design Thinking principles a visualization process was initiated based on the decision to model key prepositions. Both physical and digital mockups were used in the evaluation and creation of the user sequences and for the modeling of the User Interface.

Key design considerations were required to ensure a successful AR experience, such as the ability to do comparisons of objects between the digital experience and the physical object. The closer these are in both scale and likeness, the more effective that experience is in teaching the concept by eliminating distractions resulting for differences in objects. Additionally, it was critical that a simple progression of tasks and activities take place to ensure the learning of the preposition was clear for the child (see Fig. 3). As an example, in describing the preposition of “in”, we modeled a spoon being placed in a bowl. Well, the first step is to make sure that the bowl and the spoon are in the right place in the set up to ensure that the experience will work. To get the bowl and the spoon in the right place, we can decide if we show the child how to do that or do we have the clinician/parent do that task first. In either case, we had to decide the progression of activities to ensure an effective experience.

Fig. 3.
figure 3

First draft of a story board to illustrate the directive of placing an object in a container.

Full size image

2.4 Build Process

The build process began in earnest upon completion of the UX design based on the preliminary visualization story boards created. In each step of the build and design process, the staff at the BCH were providing feedback on the approach and the experience. This was done thru email exchanges, and live meetings in which demonstrations were provided.

The application was built leveraging technology from PTC. The primary product used was Vuforia Studio, which is built on top of the Vuforia Engine (the most common AR development platform). The primary reason for the use of Vuforia studio is the ease of implementation and rapid prototyping. Studio allows for importing of 3D models and placing these models into a 3D space without having to deal with the underlying code.

In addition, Vuforia Studio supports ThingMark technology which is marker-based tracking. The ThingMark technology has an advantage over QR codes due to allowing the association of multiple URLs to a single ThingMark. This allows one single ThingMark to store multiple versions as the application was being developed. The animation of the movements of the objects within the application was built using Java script since it integrates well with Vuforia Studio. Vuforia Studio is hosted in the cloud, but the experiences can also be downloaded to the application and run in an offline mode. To create some of the 3D models, Creo 5.0 was used.

Upon successful completion of Phase 1, a Phase 2 project has begun using the same Design Thinking principles to expand the scope of the experience to use more prepositions and the introduction of several verbs with a child’s toy farm set to make the experience a bit more relatable for the children. Initial data evaluating the use of the application by children with ASD are still being collected.