Using Facial Expressions Depicting Emotions in a Human-Computer Interface Intended for People with Autism

Abstract

Autism is a disorder altering verbal and non-verbal communication. People with high functioning autism, who have normal or above normal IQ scores, tend to have difficulties with facial expressions during social interactions. Researchers in multidisciplinary fields provide experimental evidence of the usefulness of computer education for autism. Yet, few software based on facial expressions have been experimentally evaluated. Our multidisciplinary research team aims at defining design guidelines for software dedicated to autism. In an experimental protocol, we have compared learning in two domains: dialogue understanding and spatial planning, an area in which people with autism are expected to be more skilful. Subjects’ performances were assessed in each learning domain during two evaluation sessions, which occurred before and after a training period. Eight teenagers with high functioning autism attended a workshop once a week, during 13 weeks. The training exercise in dialogue understanding displayed a written dialogue, along with three assertions about the dialogue, only one of which was correct. Three training sessions were dedicated to testing a modality where every reply in the dialogue was bound with a 3-D image of the character’s facial expression. Each reply of the dialogue was successively displayed and pronounced by a synthetic voice, while the corresponding facial expression was displayed. The user could then click on any reply to see the associated facial expression. Facial expressions depicted six possible emotions: happiness, sadness, fear, surprise, laughter, anger. Dialogues would contain pragmatic subtleties. We wanted to test the impact of emotional facial expressions on dialogue disambiguation, by comparing subjects’ performances with and without facial expressions. The influence of pictorial style was also tested by comparing realistic with cartoon like facial expressions. Three dependent variables were computed for performance analysis: the number of incorrect trials, the number of correct scenarios and the mean duration of correct scenarios. First results indicate a significant progression in the social dialogue understanding domain after the overall training period. The influence of the emotional facial expressions modality is still being examined at the time of writing.