Keywords

1 Introduction

The language and communication constitute one of the cornerstones of cognitive development of children, since they provide a set of skills that any individual can use to interact with his/her environment, or express his/her needs, ideas, thoughts, and feelings. However, currently an important number of people in the world (children and adults) suffer from different kind of disabilities that can affect their language and communication skills.

Some of the latest world estimates present a complex overview, with 15 % of the world’s population living with some form of disability, whereas 60 million people live with disabling hearing loss and 15 million suffer from stutter. In the same line, the number of existing SLP confirms a lack of personnel to provide adequate healthcare and rehabilitation services [1]: in sub-Saharan Africa there is 1 SLP for every 2–4 million people, while in the US, UK, Austria, and Canada there is 1 SLP for every 2500–4700 people.

Commonly, the People with Communication Disorders (PWCD) can be affected by a disability. The range of the existent disabilities is large, and a disability can appear in any stage of the person’s life. In the early life stages, a delay in meeting developmental milestones may be secondary to perinatal events, involving complicated interactions between mother and fetus during delivery. Maternal factors including weight, diet, and morbidities can affect neonatal adaptation and later development. Prematurity, low birth weight, and previous intrauterine insults as well as complications during delivery of a previously normal fetus increase the risk for perinatal stress [2]. Some of the most common speech and language disorders are the following [3]:

  • Speech disorder is present when a person is not able to produce speech sounds correctly or fluently, or has problems with his/her voice.

  • Language disorder appears when a person cannot understand in a proper way other people (receptive language) or cannot share thoughts, ideas or feelings (expressive language).

  • Swallowing disorder can occur during some of the three phases of swallowing process (oral, pharyngeal or esophageal).

  • Hearing disorder appears when damage exists in the auditory system. This disorder can be one of these three types: conductive hearing loss, sensorineural hearing loss, or mixed hearing loss.

These disorders can be related or not with several types of disabilities/diseases like cerebral palsy, Alzheimer’s disease, Parkinson’s disease, bacterial meningitis, intellectual disabilities, autism spectrum disorder, etc. [3]. On those grounds, in this paper we present a comprehensive approach to support the different activities conducted by SLP, with the aim of providing better healthcare and rehabilitation services for PWCD. Our approach uses an integrative ICT layer of robotic, mobile and web tools that relies on formal knowledge model and standardized vocabularies. In the next sections we will describe in detail the most important features of the Robotic Assistant for Speech and Language Therapy (SPELTRA).

This paper is organized as follows. In Sect. 2 are presented some relevant researches related to ICT tools and SLT. A general overview of the proposed approach and the details related with the robotic assistant and the mobile environment are described in Sect. 3. The pilot experiments conducted in order to validate and analyze the patient’s response to SPELTRA are shown in Sect. 4. Finally, the Sect. 5 presents conclusions and some lines of future work.

2 Related Work

During the last decade several researches have been conducted with the aim of improving different activities conducted during the SLT sessions. Some robots have been developed to assist in the rehabilitation of patients suffering disabilities due strokes. A research example of this area is uBot-5, a humanoid robot capable of providing physical therapy and speech therapy for people suffering from aphasia and hemiparesis [4], allowing to analyze how the intervention in one domain of therapy affects the progress in others domains. Likewise, several elderly people suffering from dementia have shown recovery from depression, reduction of agitation, and recovery from speech disorders during therapies conducted with support of PARO robot (a seal pup) [5].

The interaction between children with special needs and robotic interviewers is another field of study. In this field, a research was conducted in [6] with the aim of analyzing the children behavior during the interviews conducted by a robot compared to interviews performed by a human interviewer. The results reveal very similar outcomes in the children responses for both cases (number of words, filler words and key words). Other approaches propose the use of robots to support therapy sessions and improve the social interaction skill in children with autism spectrum disorders [7, 8], motivate children with communication disorders to learn the sign language [9] or provide support for Mexican patients suffering from motor speech disorders (dysarthria) [10].

The aforementioned proposals do not pay attention to important issues as the domain-knowledge modeling as well as the storage, sharing and querying of electronic healthcare records through international standards and vocabularies.

3 Support Model for SLT

Our proposed approach has the aim of providing a complete support for SLT, considering all people involved during the different rehabilitation and assessment stages and activities. The model helps doctors and SLPs with different kind of activities related with therapy and health care like report generation, support during therapy sessions, patients monitoring, and multisensory stimulation. The relatives can use several mobile applications to conduct reinforcement activities in home and help with patient’s monitoring, whereas students (future SLPs) can be trained using real cases and automated tests. Some of the most important elements of SPELTRA are the following (Fig. 1):

  • The knowledge layer is based on seven main areas of speech and language: hearing, voice, swallowing, receptive language, expressive language and articulation, oral structure and function, and linguistic formulation. Each of these areas allows defining the skills, development milestones, knowledge, assessments and tests that must be considered during the rehabilitation process of PWCD. Through the use of OpenEHR archetypes (www.openehr.org), ontologies, and standardized vocabularies is possible to represent, manage and share all information related with patients/subjects (stored in a database). The left hand side of Fig. 2 shows a partial screenshot of the protocol (template) to conduct a Pure Tone Audiometry test (PTA) test. This template is based on an archetype that allows assessing the patient’s response to sound stimulation and voice commands, and determine whether he/she is able to localize sound sources without visual stimuli. The right side of Fig. 2 shows a partial view of the archetype that captures the information related with speech fluency patient’s evaluation.

    Fig. 1.
    figure 1

    The SPELTRA robot and its relation to a knowledge model of SLT, the support software and the expert system.

    Full size image
    Fig. 2.
    figure 2

    (a) Partial view of the protocol (template) required to conduct a Pure Tone Audiometry Test (developed with OpenGENEHR: https://code.google.com/p/open-ehr-gen-framework). (b) Partial view of the archetype modeled to store the information of fluency evaluation on patients (developed with OpenEHR Archetype Editor).

    Full size image

  • SPELTRA incorporates a set of functionalities (described in the next section) that allow building a wide range of software tools to support different activities of SLT. Some of most relevant examples are the following: an application to assess and help patients suffering from disabilities related with speech production, a tool to present exercises to develop/rehabilitate gross and fine motor skills, a set of exercises to stimulate visual and auditory memory, and a special application to control the robot remotely.

  • The model integrates an expert system to automatically generate therapy plans for long term periods (6 months). The expert system uses an approach based on multi-level clusters (see more details in [11]), and performs an analysis the patient’s profile (medical condition, initial speech-language screening, skills affected, disorders/disabilities suffered by patient, etc.).

3.1 Speltra

The robot is an intelligent environment that uses mobile devices to provide support during the SLT process. Basically, it relies in a set of mobile applications and on a robotic assistant consisting in two elements: a central processor (an Android-based smartphone or tablet) and a displacement platform (electronic device). The robot can perform several activities and provide the following services to SLP:

  • Provide exercises and activities to assist in therapy activities like visual stimulation (colors, geometric forms, etc.) and motor skills acquisition (painting, virtual puzzles, etc.).

  • Recognize four different hand gestures: fist, open hand, closed hand, and semi-closed hand.

  • Detect and recognize faces of several system users.

  • Tell stories, play songs, execute commands like searching words or sentences in Internet, and read emails.

  • Hold objects using a clip.

The most complex tasks performed by the robot are related to facial recognition and gesture recognition. For the first one we use the Local Binary Patterns (LBP) approach, due it is a less expensive technique for mobile devices [12]. In order to recognize the hand gestures, we require the patient to wear a glove of a uniform color (typically, red, blue or green) and apply two techniques that had obtained good results convex hull with polygonal approximation descriptors [13, 14]. With this combination we have achieved around 90 % of precision in gesture recognition.

The SPELTRA control software can work independently of the displacement platform, and be installed on any version of Android from 2.3 to 4.4. The requirements on the devices include 256 MB of RAM memory, a Qualcomm processor and around 10 MB of storage space. It can be controlled remotely through a graphical user interface to allow the SLTs to have complete control of the therapies. The power consumption is low, and an energy supply system is included for 8 h of uninterrupted operation.

In order to recognize the hand gestures, the patient must to wear a glove of a uniform color (typically, red, blue or green). Once the image is captured, we extract and combine two shape descriptors to perform the recognition: convex hull and polygonal approximation. With this combination we have achieved around 90 % of precision in gesture recognition. Figure 4 shows a patient that suffers from cerebral palsy interacting with the robot.

In the other hand, the mobile complementary applications allow to conduct several kinds of exercises related with articulation (phonemic awareness, sentence construction, phonation …). Likewise, these applications provide support during the patient’s articulation assessments (repetitive and spontaneous). Figure 3 depicts the robotic assistant, his displacement platform and two screen captures of the mobile application to provide support during breathing exercises.

Fig. 3.
figure 3

The assembled robot (a), the design of the displacement platform (b), a screen capture that shows the main menu of the breath exercises (blowing candles, inflating balloons, etc.) (c) and a screen capture of the visual menu to help in utterance productions (phoneme/m/) (d).

Full size image
Table 1. Groups of children that participated in the experiment (the column “disability” includes the standard code provided by [3]).
Full size table

4 Experiments and Results

In order to assess the functionalities and the advantages of SPELTRA, we have conducted two preliminary experiments in the “Jesus for Children” foundation of Cuenca, Ecuador. This foundation is a public institution that provides different services as counseling, rehabilitation (SLT, physical therapy, occupational therapy, and psychological support), and provides an environment where children with disabilities can be integrated. The participants of this experiment were 26 patients (23 children between 5 and 17 years, and 3 adults with an average age of 24 years) suffering from different kind of disabilities (see Table 1).

4.1 Relaxation Activities to Prepare Children for Therapy

This experiment was conducted with the aim to measure the required time to perform some relaxation activities and exercises as previous stage of SLT. During this process we have worked with the aid of two clinical psychologists, two speech-language pathologists, one occupational therapist and one educational psychologist. The main goals of this experiment were measure response time of children to activities proposed/conducted by robot, and verify the robot’s integration with the database of profiles and therapy plans.

Likewise, some of the most relevant activities carried out with the guidance of robot were related to auditory stimuli production (listening stories, maintaining a basic conversation, etc.), performing actions when the robot shows a specific color in the screen (jumping, raising the hands, etc.), executing simple commands (playing “Simon says”), among others.

All of the experiments thus far have provided very positive and encouraging results. Among other achievements, the time needed to conduct the relaxation activities and initial therapy has shrunk from 40 min to 25, all the specialists have agreed on the usefulness of SPELTRA as a supporting tool for their work, and children have shown high levels of motivation during their interactions with the robot.

4.2 Level of Interest

In this experiment, we asked the team of experts to provide a subjective rating of the level of interest shown by each group of patients during the activities described above. To achieve this task, we have used a Likert scale in order to characterize each level: very low = 1, low = 2, medium = 3, high = 4, very high = 5.

Table 2 shows the results of subjective evaluation, according to each group of patients aforementioned. It is important to mention that lowest values of interest are present in groups of patients where the interaction with the robot is a more complex task, due to nature of the disability (cerebral palsy, autistic disorder and severe intellectual disability). Conversely, the patients suffering from intellectual disabilities (mild and moderate), attention-deficit and Down syndrome present high levels of interest.

Table 2. Level of interest according to each group of patients
Full size table

Figure 4 shows some examples of stimulation activities to develop memory, motor skills and creativity. On the left hand side of the figure, when the subject closes his hand, the “virtual” hand grabs a geometric figure and allows him/her drag it across the screen. The other pictures show a child using a painting application.

Fig. 4.
figure 4

Some pictures taken during a therapy session with the SPELTRA robot

Full size image

5 Conclusions and Future Work

The results obtained with this research show that is possible to automate several activities related with SLT, with the aim to provide a better service to patients suffering from several kinds of disabilities. In the same way, mobile applications constitute important backing tools, allowing SLPs to perform their activities anywhere and in a more comfortable way.

In general, the use of robotic assistant to guide several activities related with SLT can achieve an important motivation in specific groups of patients suffering from Down syndrome, intellectual disabilities and attention-deficit disorder. It is important preparing special therapy sessions for patients suffering from autistic disorder, since they require establish bounds of confidence with the robot.

As lines of future work, we intend to develop more specific activities and routines for treatment, considering a spectrum of the most common types of disabilities.