Keywords

1 Introduction

Advances in wearable, mobile and tactile computation are defining a new generation of interactive systems where the quality of the user experience and the creation of applications immersed in the day-to-day activities is transforming the life of many people. In particular, the use of those interactive systems to support physiologic, physical, or mental disorders such Autism Spectrum Disorder (ASD), has awakened the interest of the Human-Computer Interaction (HCI) research community and drives efforts to provide novel and affordable solutions as technology evolves. Moreover, the publishing of the newest edition of the Statistical Manual of Mental Disorders (DSM-5) [1] has changed the way we see autism and thus, we must design supporting technologies that respond properly to that development.

Acknowledging the importance of social interactions and building upon the work developed in MOSOCO [2], that showed that an interactive system in a smartphone can be a tool to enable social interactions, our research seeks to create an assistive application to aid students with autism improve their social interaction. Following a user-centered design approach, we developed a solution that generates low sensory stimuli and an interactive method that integrates the therapy used with the participants of the study. Our approach explores the opportunity of supporting social interactions in a more natural way by working on a smartwatch that is a less intrusive and more personal device compared with a laptop, a tablet or a phone.

The application, called TEA Band (Trastorno del Espectro Autista is Autism Spectrum Disorder in SpanishFootnote 1) is conceptualized as a support tool to learn, not a substitute of therapy nor a permanent assistive technology.

2 Background

2.1 Autism Spectrum Disorder

The Autism Spectrum Disorder is described in the Statistical Manual of Mental Disorders (DSM-5) as a mental disorder characterized by persistent deficits in social communication and social interactions and restricted, repetitive patterns of behavior, interest, or activities [1]. The DSM-5 defines three levels of autism, which indicate the level of support necessary for a person with autism with diagnostic criteria in the areas of social communications and atypical behavior.

The prevalence of autism about 1%–2% in the world [3] with 5 more boys with ASD than girls. In Mexico, although it does not exist any statistics or epidemiological data, a study to generate an estimate of the prevalence of ASD in the country, surveying children in Leon, Guanajuato, indicated that the numbers are consistent with the world prevalence rate [4]. Only in USA and England the cost of maintaining an individual with ASD during his life span is estimated in 1.4 million dollars if the person does not have an intellectual disability [5]. This sum is the result of medical, scholar and therapy expenses, plus the loss of productivity of the parents and the residential and support personal needed when the person reaches adultness; besides, 80% of the adults with autism are unemployed [6].

Typically, individuals with ASD display certain social difficulties such as: difficulties to communicate, with trouble to interpret non-verbal interactions and following conversation rules, difficulties to create friendships appropriate for their age, low sustained attention, lack of visual contact, and stereotyped behavior patterns like motor movements, mutism and echolalia. Furthermore, persons with ASD are also extremely dependent to routines and sensible to changes in their environment.

The fifth edition of the DSM-5, published in 2013, changed the way we understand autism when two of the basic triad problems were joined into one: social interaction and social communication impairments are now described as one conjoined problem. Almost 20 years of separate therapy for language and social interaction have now been deemed unsuitable and new strategies to design assistive technologies are required.

2.2 Inclusive Education

The 26th article in the Universal Declaration of Human Rights declares that “everyone has the right to education […] and it shall be directed to the full development of the human personality and to the strengthening of respect for human rights and fundamental freedoms” [7], this article and the Salamanca Statement, a UNESCO statement which calls on all governments to give priority to inclusive school, support the necessity of education for everybody in an integrated environment that celebrates the differences and support learning by responding to the different needs of every individual [8]. However, the probability of a child with a disability attending school is much lower than a child without a disability [9]. In Mexico, 23.6% of people with disabilities between 15 and 24 years of age are analphabets and children with a disability have 17% to 25% less possibilities to have access to education in comparison of children without a disability [10].

In 2011, the Mexican government created the general law for the inclusion of the people with disability in Mexico [11]. This law defines inclusive education as education that facilitates integration of people with disabilities into regular basic education through the application of specific methods, techniques and materials; however, this definition only talks about integration and there have been barriers to include children with ASD, such as: lack of resources, inflexibility to adapt the curriculum, and teachers that are not sufficiently qualified to support children with autism [12].

Due in part to the difficulties in the social communication, many children with autism in inclusive schools around the world suffer from bullying by their peers. Just in the USA and England, 63% [13, 14] of students with autism have suffered from bullying from their peers. In Mexico, there is not sufficient data but there is evidence that children with autism suffer from bullying from their classmates and rejection from the teachers [15, 16].

Although schools provide an excellent opportunity for the students to interact, the difficulties in social interactions is one of the elements that define the Autism Spectrum Disorder; the lack of successful interactions impacts their concept of identity and belonging, does not allows them to learn what others expect from them, or see appropriate behaviors, and it does not permit incrementing the flexibility in thinking as the consequence of multiple opinions in an interaction. These difficulties to have successful interactions persist and impact all their life and has motivated multiple efforts to increase and improve social interactions in children with autism [17,18,19].

2.3 Technology for Supporting Children with ASD

In certain neurodevelopmental disorders, particularly, in ASD, the use of an interactive system has the potential to be a tool that allows focusing the attention of the individual, avoiding the stress that provokes their environment and, at the same time, to be a support tool to interact with said environment. Taking this in consideration, diverse investigations in the Human-Computer Interaction (HCI) area are focused on the creation of technologies that allow persons with autism, particularly children, to create or improve their social interactions.

Tentori and Hayes created the concept of interaction immediacy: “a set of guidelines help children maintain appropriate spatial boundaries, reply to conversation initiators, disengage appropriately at the end of an interaction, and identify potential communication partners” [20] and established design principles to apply those elements in assistive technologies. Escobedo used that information to create MOSOCO [2], a mobile application that allows the students to remember the lessons of the Social Compass Curriculum [21], a curriculum that helps train social abilities, outside the classroom in real time situations. The study was developed in seven weeks with three students with autism and nine neurotypical between the ages of 8 and 11 years old.

Hourcade et al. [22] conducted a research to analyze the impact of tactile applications in tablets and their impact in social interactions by allowing collaboration, creativity, compromise of interests and a better understanding of emotions. The study analyzes three of the candidates in thirteen sessions of two hours. Similarly, McEwen [23] demonstrated that the use of the iPod Touch in the classroom can motivate the start of a social interaction between classmates in a longer study with 12 students in a 6 months’ period.

The use of smartwatches or smart bands and their sensors in TEA are a relatively new field that opens multiple research opportunities. Multiple sensors in controlled environments have been studied [24,25,26] but the plausibility to use them in a real-world situation is inoperative. Within the framework of affective technology, Picard developed Empatica: a watch with sensors that allow the detection of emotions using sensors. She and her team at the MIT Media Lab are also the creators of Affectiva Q Sensor, a wearable with sensors that was initially created to measure emotions like stress, excitation or calm in children with autism [27]. Similarly, Reveal, developed by Palmer, can help predict crisis alerting the parents or even preventing them since it gives time to react appropriately [28].

3 Designing TEA Band

3.1 Selection of the Smartwatch

To select an appropriate hardware, we considered various aspects to prompt an easy adoption of the device and characteristics that allowed us to exploit all the capabilities of the smartwatch such as weight, fragility, battery life, sensors in the smartwatch and the possibility to develop an application on the device that worked in real time, the outcome of an analysis of 8 smartwatches supported the decision to choose the Microsoft Band 2Footnote 2.

The Microsoft Band 2 fulfilled all the requirements with developer tools to design applications and obtain data of the sensors; the selected smartwatch has Gorilla Glass 3 in the screen, a highly resistant material, the band is made with medical level steel and Thermal plastic elastomer silicone vulcanite, a resistant and comfortable material. The Microsoft Band 2 allowed personalization of colors, icons, vibrations, reading speed and messages. Although we choose this device for the study, the software can be developed for any other device that has the appropriate sensors and developer tools, but the adoption of another smartwatch might be different.

3.2 Interface and Behavior

Our senses enable us to perceive and capture the world, but many people with autism have a problem modulating all the sensory input they receive, resulting in stereotyped behavior and abstraction of the individual from the outside world [29]. The DSM-5 includes again the hyper- or hypo-reactivity to sensory input (this was not included in the DSM-IV) as criteria under the “Restricted, repetitive patterns of behavior, interests or activities”. When we analyzed which elements must be considered in the design, we recognize this sensory sensibility and we seek to develop an assistive technology with low sensory stimuli in two senses: touch and sight.

The color of the application was a very important sensory element, a 2005 study reported that 85% of the people with autism see colors with much more intensity [30]; a simple design with neutral colors that can be modified for each user’s needs, was key. The iconography is another element key for the adoption, the National Autistic Society in England mentions that many people with ASD are visual learners [31], and the icons are an instinctive way to understand TEA Band.

We seek to understand the difficulties the students have communicating and the strategies used by the psychologist and parents to deliver messages on the device in a clear and comprehensive way. The guidelines for language, instructions, and feedback methods are derived from the methodology used in DomusFootnote 3.

Domus institute uses the Relationship Development Intervention (RDI), as their principal behavioral treatment; this program is based on families and searches to remedy autism attacking the primary symptom: the social interaction. RDI is based in the belief that the development of a dynamic intelligence, using neural connectivity, is the key to improve the life quality of a person with autism. The program causes parent to return to their role as guides and, therefore, children become learners again - a role that is lost when parents adjust to the child and their static thinking - and proposes to give information in a multimodal way: body language, verbal, sounds, facial expressions and gestures with the hands, prioritizing the social relation over an activity and eliminating the static process of giving instructions and orders as part of a therapy [32].

We used declarative communication, a language that allows sharing of experiences, ideas, perspectives and thoughts [31]. This type of communication allows a self-regulation of thoughts and actions with cortically integrated and dynamic answers, in contrast of an imperative communication that only has predetermined answers. Also, following the RDI approach, the therapy used delivered immediate feedback to reassure the student and reduce the stress in the moment while the attention of the user is still in the activity.

To deliver the messages we decided to use the haptic sensor to send a small vibration in the wrist, like most smartwatches, but with the possibility of modulating the intensity of it, to control the sensory stimuli created to draw the attention of the user. All the messages were preloaded and unique for each user. The messages were defined with feedback of the users and validated via proxy with the psychologist and the inclusive education monitors. We followed all the recommendations, maintaining a declarative language and the briefness of the messages as the only immovable elements.

The interface color, the background image on the front tile, the haptic sensor, the reading speed and the position of the Microsoft Band 2 (in which wrist and inside or outside of the wrist) were chosen by each participant to involve them in the design of TEA Band and motivate a sense of appropriation.

3.3 Application Design

Following the guidelines of design and experience of the Microsoft Band 2 [33] we created a band application with three tiles, alongside with an iPhone application for setting control and data visualization in real time. We also deleted all the other tiles the device had, leaving only the start screen -named Me tile- that indicates the hour, steps walked, stairs climbed, calories and heart rate, and the Configuration tile, since they could not be deleted.

The application, called TEA Band, has three tiles as shown in Fig. 1.: Stress, Hints and Topics, each one of them has a different purpose and interact different with the user, as a response of the sensors, predefined, and through a user request, respectively.

Fig. 1.
figure 1

TEA band interface: Stress, Hints and Topics tiles

Full size image

The Stress tile’s objective is tranquilizing the student sending a message when there is an increment in the stress level of the user. To measure the stress level, we used the formula developed by Garcia and Gonzalez [34] that detects stress using the values of the heart rate, galvanic skin response and temperature sensors. The formula was initially developed to detect stress posteriori and with data that had one-minute granularity, so we worked with Jesus Garcia to modify it so it would work in real time and with data granularity of one second.

The Hints tile helps the students to improve their social interactions with short messages that remind them appropriate social behaviors such as saying hello and goodbye and keeping a proper distance and voice volume. This tile was initially going to be automatic, sending the messages with time constraints or as a response of the sensors, but since the Bluetooth of the device had a very limited range, sometimes the Microsoft Band 2 would disconnect so we opted to send each message manually, but making the user think they were automatic.

Finally, the Topics tile gives the student various conversation topics to improve the quality and length of an interaction. This tile responds to a touch of the user; it does not send messages unless the user asks for them.

4 Study Description

We worked with Domus, an autism institute, because of their trajectory and the opportunities they offer to do research with them. Since 1994, Domus has had a program of inclusive education, with students in all education levels up to high school. The idea of this project was presented to the director of the institute and later, to the whole team for its approval. We followed their ethical guidelines for the project and both the Domus team and the ITAM’s University Board approved the study.

The access to the students with autism was a long and complex process due to the impact on them and their therapy that implied clear changes such as involving meeting a new person and conducting an experiment with their help. Also, since autism is a spectrum, finding suitable candidates willing to perform the experiment implies an additional limitation. Finally, the authorization to enter the schools and perform the study had other limitations and lengthy bureaucratic protocols that were managed by the Domus team; we made the request to do the study to Domus on September 2015 and seven months later we received permission to access two different inclusive schools.

Domus selected four participants with autism between the ages of 11 and 18 years old. The requirements we made were that the candidates knew how to read, were verbal, had difficulties to interact and, preferably, assisted to an inclusive school; although we did not have any preference with male or female candidates, all the candidates were male; this is consistent with reports indicating that autism is more prevalent in boys than girls. Due to some agenda complications, the first candidate had to retire from the study and we proceeded with the other three students. The study was made with three male candidates (Tadeo, Sergio and AdrianFootnote 4) with ASD, ages 17, 15 and 18, each one with different challenges and abilities.

Tadeo is a student with autism who had his 17th birthday while we developed the study and was in the 1st grade of high school in an inclusive school. At the school breaks he does not interact because he does not have too many common interests with his classmates and he uses this time to switch off for a while. He likes to interact verbally presenting few stereotyped movements, and his social abilities are adequate with his mentor and professors who are patient with him but he struggles to find the same sympathetic behavior from his classmates; his social errors are usually that he forgets to say “excuse me” “please” and “thank you”.

Sergio was in 3rd grade of middle school and had 15 years at the moment of the study. He is very shy and has a lot of difficulties to interact. During his breaks, he does not interact unless someone approaches him, and even then, sometimes he just ignores the interaction attempts. He does not move at all from the place he chooses to eat his lunch. Sergio presents few stereotyped movements and low social abilities, in part because of his shyness.

Adrian is a young man with autism that had 18 years when we developed the study and he was not in an inclusive school anymore. Adrian enjoys verbal interactions but he has very restrained interests and not always responds to another person’s intentions to start an interaction. He has very poor social abilities, with difficulties to respect personal space and modulate his voice volume, and presents echolalia and multiple stereotyped behaviors. The echolalia affects all his interactions since this language perturbation does not allows his interactions to flow naturally and keeps him in a loop for quite a long time.

We would like to remark that although we are describing three candidates with autism spectrum disorder with similar ages and characteristics (i.e. ASD level 1, difficulty to interact, verbal, assisted at some point to an inclusive school) we could observe different challenges and abilities in each one of them. These are expected differences because autism is a spectrum and they present an excellent opportunity to test all the capabilities TEA band has to impact in different scenarios.

4.1 Development of the Study

We programmed fifteen sessions of twenty minutes for each participant, giving priority to the school breaks or activities where we could observe the candidate interact in uncontrolled activities and ensure that these sessions were continuous for up to three weeks. The number of sessions and the duration were suggested considering other studies involving students with autism and the use of technology [2, 35,36,37] and, more important, the availability of the schools we visited. The time used for the student to adjust to the device and the observer as well as the time necessary for a closing with the candidates, were not considered as sessions.

To avoid stressing the participants, the fact that the device was going to measure their stress was hidden and we emphasized that they were helping us to understand how they interacted with their peers. Also, the neurotypical students were not made aware of the purpose of this study to avoid a bias in their behavior.

The functionality of the device was introduced gradually. The first five sessions we observed the individual using the device without TEA Band interacting with them, while we measured their stress, and observed their social interactions. Over the next five sessions, we gradually introduced the tiles and, from the eighth session, we motivated the student to start social interactions using TEA Band. Finally, in the last five sessions the student had all the functionalities of TEA Band working and used them to interact with their classmates while we sent messages and observed the interactions.

4.2 Data Recollection

Using the criteria elements in the DSM-5 and the ADOS-G, the Autism Diagnostic Observation Schedule-Generic [38], as well as the First-Grade Unstructured Peer Interaction Observation System [39] an observation instrument developed by the ORCE based in the Classroom Observation system: COS-1 we constructed an instrument with key elements to observe during the interactions. This instrument included data about the interaction, the device, the conversation and the social errors. We also sent messages when the individual was stressed using the stress formula [34] and the information from three sensors: galvanic skin response, heart rate and skin temperature.

5 Results

5.1 Device Adoption

The device was easily adopted; none of the students expressed any concerning about using it or its functionality. The Microsoft Band 2 resisted perfectly to the use and although it was not thrown, it was bended, beaten and scratched (although not on purpose) without suffering any damage.

An important activity to make the students feel comfortable with the observer and the device was showing them that you could change the screen color and the image for the MeTile and allowing them to choose those; also, as another way to make them participate in the process, they chose how to use the device: in which hand and position, the reading speed and the haptic level for the alerts allowing them to feel comfortable while using the Microsoft Band 2. The interaction with TEA Band was very intuitive and did not require an extra training.

5.2 Interaction with the Device

TEA Band had the objective of being a very discreet support device during the interactions, and we paid particular attention to every time when the users saw or touched the device during an interaction. The messages suggesting them to start an interaction or congratulating them after a successful interaction, were not considered in the observation since they were not intruding an interaction.

Before sending the messages, we prepared the candidate, explaining that they were going to receive messages and, with Tadeo and Adrian (since both like superhero movies) we compared it with the fictional A.I. J.A.R.V.I.S. writing messages to help them instead of talking to them. We introduced the messages gradually during the second phase with positive reactions from the users (“It say Hi!”, “My favorite part was that it sent me messages”, Reading the message “Good job. I did a good job!”) and their inclusive educator monitors (“He gets very stressed [trying to talk with his classmates] because he did not know if he is doing a good job”).

The adoption of the messages was diverse. Tadeo read them all, following the recommendations and commenting them with the observer, while Sergio ignored them the first day and then started following the guidance, and Adrian needed extra reinforcement with the psychologist to start reading the messages on his own. It is important to mention that this behavior was not an intuition problem, he simply did not have any motivation to read the messages.

The Topic tile required a request from the student but it was mostly used before an interaction, trying to find a conversation starter. While Adrian read the ideas sometimes but he decided not to use them and Sergio used it as a relaxation method just reading them, Tadeo actually suggested this functionality in the first phase when the device did not have the TEA Band tiles: (“[the Microsoft Band 2] could remind me that I can talk with my classmates about this movie I just saw”) and he was the one that used it the most during the second and third phase.

5.3 Impact on the Interactions

The analysis of the results and observations allowed us to conclude that the device impacted positively in the quantity of the interactions for two of the participants, Tadeo and Sergio as we can observe in Fig. 2; in this figure, we only report the number of interactions that got an initial response from the other party.

Fig. 2.
figure 2

Number of interactions in each phase.

Full size image

Both Tadeo (1 st phase: 5 interactions; 2 nd phase: 4 interactions; 3 rd phase: 10 interactions) and Sergio (1 st phase: 0 interactions; 2 nd phase: 5 interactions; 3 rd phase: 8 interactions) incremented their interactions from the first phase to the third phase. Even when Tadeo was in exams week during the second phase, a stressful moment for him, and some of the observation spaces changed to more uncontrolled scenarios in the last two phases, from theater class in a classroom to theater class in the schoolyard, he incremented his interactions and the change of scenario only impacted slightly the average length of every conversation (1 st phase: 2 min; 2 nd phase: 1.2 min; 3 rd phase: 1.2 min). Sergio did not talk with anybody during the first phase. The device encouraged him to speak with his classmates and allowed to gradually increment the average length of his conversations when he started feeling more comfortable (1 st phase: 0 min; 2 nd phase: 1.3 min; 3 rd phase: 1.6 min). Since Adrian presented constant echolalia and neither the psychologist nor the messages on TEA Band, were able to withdraw him from a constant loop of repetition, he did not use the device during the interactions, only between them. The changes in his interactions and the length of them (1 st phase: 5.2 min; 2 nd phase: 1.9 min; 3 rd phase: 1.9 min). cannot be considered an impact of TEA Band.

TEA Band motivated two of the participants, Tadeo and Sergio, not only to interact more, it also gave them the confidence to be the ones that initiated the interactions as we can observe in Fig. 3. Tadeo did not respond on time to an intent to start a conversation on the 1st phase, later, he was ignored once both on the 2nd and 3rd phase by another student when he tried to start a conversation. Sergio ignored two intents to start a conversation, one on the 2nd phase and one on the 3rd phase, however his classmates continued trying to talk with him in other occasions.

Fig. 3.
figure 3

Interactions initiators in each phase.

Full size image

The use of TEA Band allowed the students to feel comfortable talking with more than one person at the same time and helped to increment the exchanges (commentaries made during the conversation) and questions (questions or commentaries that require feedback for the other part) during the interactions. Tadeo needed more time to gradually increment his participation during the interactions (1 st phase: 6 exchanges, 2 questions; 2 nd phase: 9 exchanges, 5 questions; 3 rd phase: 14 exchanges, 17 questions), while Sergio was very involved during the second and third phase (1 st phase: 0 exchanges, 0 questions; 2 nd phase: 11 exchanges, 12 questions; 3 rd phase: 10 exchanges, 10 questions). Once again, the echolalia loops make the results not conclusive for Adrian.

Since Adrian only read the messages after the interactions, the quantity and quality of the interactions was not impacted by TEA Band; however, his social errors were impacted because he read all the messages at the end of the interaction and remembered them, using the corrections for his next interaction; consequently, he reduced his social errors considerably specially lowering his voice volume and respecting the other party personal space (1 st phase: 27 errors; 2 nd phase: 16 errors; 3 rd phase: 20 errors). TEA Band also helped Tadeo to reduce his social errors to zero but since Sergio was starting to have interactions (he did not interact at all in the first phase), his social errors augmented slightly, with his voice volume too low and walking out of the conversation without disengaging himself properly (1 st phase: 0 errors; 2 nd phase: 2 errors; 3 rd phase: 4 errors).

Observing the results, we can note that the use of TEA Band impacted positively the three candidates, even if the impact to Adrian was marginal compared to the impact on the other two candidates. It is notable the increment in the social interactions that had both Tadeo and Sergio, specially the effort they made to be them the ones who started a conversation and the increment in the exchanges and questions they made in the interactions, augmenting their participation in them. Additionally, it is important to remark that TEA Band did not impacted negatively in the therapy of any of the students.

6 Conclusion

This study allowed us to design and analyze the use of a technology to aid students with autism improve their social interactions with a very low intrusive device, a design that considers the sensory sensibility of the users and different functionalities to support an interaction. From the very beginning, we considered the profile of the students – without forgetting to create an inclusive technology for everybody- and we developed an app that was easy to adopt and intuitive in its use, keeping in mind the requirements and needs of our users with autism. The adoption of the device and its intuitive use show how relevant was the design, with clear iconography, adequate messages written specifically for each user with the aid of the psychologists at Domus and the participation of the candidate designing his own experience with the interface and the device.

The three tiles worked as expected, although the Topic tile, that required a request of the user was used before and after an interaction, not during it. This indicates a possible improvement for this tile; the users decided not to request a conversation topic during an interaction because it implied a distraction of several seconds: moving the wrist, pressing a button, searching for the Tile and request a Topic (again by pressing a button) and waiting 1–3 s. The use of the formula developed by Garcia and Gonzalez to detect stress worked as expected, sending messages to the students to reassure them when needed, indicating that the sensors worked well and introduced very little noise to the data. However, it is important to remember that the Hints tile worked manually and not automatically, which means that we cannot depend completely on the device to detect or suggest appropriate social behaviors automatically, but we can use it as a non-intrusive support method, controlled by a parent, psychologist or inclusive education monitor to aid students learn appropriate social behaviors while giving them confidence and reassurance to interact with their peers.

The data collected during the interactions with the aid of the observation scheme provided enough qualitative data to back up the analysis made of the use and adoption of TEA Band. It is also important to mention that the use of the device did not impact the interactions nor the behavior of the neurotypical students during them. The Microsoft Band 2 was a very discrete device and the messages could be read fast to continue with the interaction, but this same fact also made that the students did not always made a conscious effort to interact, ignoring sometimes the attempts to start a conversation of a student with TEA.

Finally, we would like to point out, once again, that each student participating in the study had different challenges and thus, had different results as well. Although the impossibility to withdraw Adrian from the echolalia with TEA Band provides a new limitation for our candidates’ profile and a new challenge to create an application that can distract an individual during an echolalia episode or a crisis, we saw improvements in our three candidates: Sergio and Tadeo incremented the number of interactions they had, feeling confident to start a conversation and started to participate more in them, making commentaries and asking questions while Adrian reduced his social errors.