Keywords

1 Introduction

Under the guidance of therapists and trainers, many individuals have successfully acquired and reacquired motor skills in rehabilitation and training programs worldwide. A critical component in that success has been the introduction of at-home components to rehabilitation and training, the benefits of which have been well-noted [1, 2]. However, since trainers and therapists often cannot be present for the at-home self-practiced segment of an individual’s training, individuals often fail to perform the recommended amount of at-home exercise in the long term [3]. Given that the intensiveness of therapy has been known to correspond with health outcomes [4], this is a critical issue for motor recovery in general.

To remedy this issue, telerehabilitation and outpatient rehabilitation programs and other at-home services are seeking an optimal mechanism for therapist-prescribed, self-managed exercise in the home environment. One of the key elements in this process which lacks evidence in research is the design of a feedback environment and interface to allow individuals to complete computer-based home therapy for upper limb motor acquisition and recovery [5].

To explore this issue, we are developing a model and toolkit for the delivery of customized exercise programs by therapists and trainers in the home. The toolkit, entitled “The Autonomous Training Assistant” (ATA), is a computer-based system which utilizes multi-modal concurrent feedback and the guidance of a virtual avatar to represent the trainer’s presence in at-home training.

The system consists of three main components:

  1. 1.

    An interactive exercise interface in which a virtual avatar acts as an individual’s trainer at home

  2. 2.

    Authoring software which allows to assign customized exercise routines

  3. 3.

    A rod-shaped training device, the “Intelligent Stick”, through which individuals interact with the virtual trainer.

This system motivates and empowers individuals to take control of their home exercise using a model of action observation which provides fine-grained feedback and knowledge of performance in real-time. An overview of related work in Sect. 2 indicates the need for an effective feedback environment which we explore in the implementation and evaluation of the model and ATA toolkit. In Sect. 3, we describe the model for motor-learning which motivates our toolkit’s design. In Sect. 4 we describe the details of our implementation and initial design of the avatar software, Intelligent Stick prototype, and authoring interface. In Sect. 5 we evaluate our initial prototype of the Intelligent Stick prototype for both usability and accessibility, as well as the effectiveness of our training model in representing a training program, in a case study involving a child with Cerebral Palsy and his martial arts trainer. We conclude in Sect. 6 with directions for future work including a longitudinal study of the effectiveness of the ATA in the home environment.

2 Related Work

2.1 Action Observation Model

A primary weakness in many solutions for at-home motor learning is the lack of a cognitive model for learning to inform the design and implementation of the system. An early attempt at outlining such a model for stroke rehabilitation emphasized the need for frequent repetition of the specific motor tasks pertaining to an individual’s goals, as well as the importance of visual information in the environment in directing an individual’s posture and form during motor training [6]. These foundational principles have resulted in a wide variety of strategies to induce motor learning in individuals undergoing rehabilitation [7]. While these interventions serve well within specific clinical contexts of motor learning, a more flexible model for motor skill acquisition is required for a minimally-invasive solution within the home environment, in the physical absence of a therapist.

Recently, the action observation model has been touted in research as a viable solution to this problem [8]. This model conveys motor learning as an interaction between the subject and a demonstrator, where the subject first observes as the demonstrator performs the required motion and degree for each exercise, then attempts to replicate the demonstrator’s motion [9]. Neuroscience points to the activation of “mirror neurons” as the catalysts of learning through action observation and motor imagery for both the upper and lower extremities, in the presence or absence of a visible effector limb [10–13]. The application of the action observation model to the design of games and gaming environments for at-home motor learning has yet to be explored. We utilize the action observation model in the development of the Autonomous Training Assistant.

2.2 Feedback Environments

While the action observation model provides insights into the method by which individuals interact with therapists and trainers to acquire motor skills, it is an incomplete model of the motor learning process because it does not fully explain how these skills are reinforced through repetition, and does not fully grasp the role of the trainer or therapist in this process. The other component which completes this explanation is the feedback model. The elements of this model include the frequency and nature of the feedback, and the sensory channels through which this feedback is communicated to the subject. Earlier work by Hartveld and Hegarty describes the most critical aspects of feedback from therapists in a physiotherapy context [14]: it is frequent, flexible and qualitative, and can be conveyed both verbally and nonverbally. More intriguingly, this work also points out that the equipment used in therapy can also play a critical role in providing feedback that is quantitative, objective, immediate and accurate, which can complement a trainer’s feedback and provide the information that trainer needs to assign new goals for the subject on a regular basis.

These findings are the basis for our inclusion of the Intelligent Stick device within the Autonomous Training Assistant not only as a game controller, but also as equipment which provides feedback to the user. More recent studies on feedback within rehabilitation and training have revealed further details on how it can be most effectively administered. One such study by Parker et al. suggests that the given feedback be customizable [15]. Several works emphasize the benefits of multi-modal feedback but caution that the effectiveness of multimodality vary by individual [16]. The virtual presence or representation of a therapist or trainer is described as an effective strategy for delivering this feedback in the absence of a real therapist [17]. We explore this feedback in the Autonomous Training Assistant through both the Intelligent Stick and the virtual trainer.

3 Proposed Model and Approach

Based on the above principles, we propose the following model for the acquisition of motor skills in rehabilitation and training, consisting of the subject (the observer) and the trainer (the demonstrator):

  • Action Observation: The observer watches and listens as the demonstrator performs and explains an exercise using a piece of training equipment at the expected degree of motion.

  • Motor Imagery: As the observer watches and listens, he or she imagines completing the motion with his or her body, creating a mental mapping of the motion.

  • Action Replication: The observer then attempts to replicate the demonstrator’s motion using the training equipment. During these attempts, the user receives feedback from both the demonstrator and the equipment which comprises knowledge of performance (pacing, posture, progression toward the targeted degree of motion) and knowledge of results (number of successful completions of the required motion, improvement since previous session). This feedback is frequent, accurate, verbal and non-verbal, multi-modal, and occurs in real-time.

  • Evaluation: During each attempt, the observer uses the information provided through feedback to evaluate his or her performance, and attempts to improve performance on the next attempt.

  • Assessment: The demonstrator assesses the observer’s performance and, based on this information, forms a new set of goals (pacing, degree of motion, number of repetitions) to assign in the next session.

This version of the action observation model ensures that the principles for successful rehabilitation and training [6] are met while the components of observational learning [9] are preserved. We use this model to design a solution for at-home exercise by introducing a key middle-agent into the process: the virtual trainer. This trainer, embodied as a virtual avatar within the home environment, serves as the facilitator of indirect at-home interaction between the real trainer (we will use “virtual” and “real” to distinguish between these two entities in this paper) and trainee. The virtual trainer is programmed with a real trainer’s exercise program and training protocol on a per-trainee basis (the interface, motions and goals are different for each individual using the software). The virtual trainer uses this information to conduct and oversee an individual’s at-home training, including the following roles:

  • Demonstrate visually and provide a description for each motion.

  • Measure and record an individual’s performance in real-time using the parameters given by the real trainer.

  • Provide feedback to an individual in real-time on each attempt of an exercise using the protocol provided by the real trainer (knowledge of performance).

  • Provide overall data to an individual on his or her performance after a completed session for each exercise, based on the parameters given by the real trainer (knowledge of results).

  • Provide data to a real trainer on an individual’s performance per exercise, per session, on all of the parameters listed by the real trainer. This information should be usable by the real trainer to assess the trainee, view progress over time, and provide new goals for each parameter to the virtual trainer.

This framework of interaction, based upon the action observation model given above, is summarized in Fig. 1.

Fig. 1.
figure 1

Interactions between virtual trainer and trainer/trainee

Full size image

4 Implementation

To implement the framework for interaction shown above, we designed the Autonomous Training Assistant as a proof-of-concept prototype to determine how well the framework can facilitate at-home training in a real-world scenario. We focus the scope of our implementation to upper extremity motor rehabilitation for individuals with mild to moderate hemiparesis (motor impairment in one arm with full function in the opposite arm). While this is a very limited scope due to its specificity, it serves as an initial testing point for our proof of concept. We reserve the generalization of this approach for future work in which we will explore lower extremity function and applications toward other populations.

4.1 Intelligent Stick Prototype

The Intelligent Stick is a rod-shaped training stick which is held and swung by the trainee to complete at-home motion training. Every exercise designed with the ATA system uses the Intelligent Stick as training equipment. We chose to include the Intelligent Stick prototype due to the critical role of training equipment in obtaining objective measures of performance and providing physical feedback to the user [14]. A design sketch of the prototype is shown in Fig. 2. The design consists of a hollow tube molded with plastic resin durable enough to protect the inner hardware in case of drops or collisions, and interior modules including a vibration motor, a power supply, an accelerometer with high-frequency sampling, a gyroscope, and Bluetooth interface for communication with the software.

Fig. 2.
figure 2

Intelligent Stick prototype design sketch

Full size image

4.2 Motion Authoring

We include Motion Authoring software in the ATA system to initialize and update the virtual trainer with the training tasks, feedback cues, and goals of the trainer. The same software provides data on performance to the trainer to allow them to assess and monitor and individual’s performance and progression through the training, and to assign new goals for exercises as well as entirely new exercises.

The primary challenge in developing this software is to create a definition of a motion task that is flexible enough to cover as wide a range of different motions used by as many different trainers/therapists as possible. To help us achieve this task, we focus specifically on upper-extremity motion exercises with a single degree-of-freedom. This limits the software and the system to motion tasks in which the major joints of the arm (wrist, elbow, shoulder) are rotated along a single plane. While this presents a major limitation to our current implementation (limited domain of exercises captured in the software), it enables us to simply and accurately represent a motion within the software and simplifies the interface for trainers, allowing them to adjust goals with numeric entry.

Using these restrictions, we define a motion task via the following properties:

  1. (a)

    Name of the motion

  2. (b)

    Text description of the motion

  3. (c)

    Primary limb (elbow, shoulder, wrist) involved in the motion

  4. (d)

    Unimanual (left or right arm) or Bimanual

  5. (e)

    Axis of rotation (x, y, or z)

  6. (f)

    Starting position

  7. (g)

    Degree of motion

  8. (h)

    Expected average speed of motion (pace)

  9. (i)

    Body posture

  10. (j)

    Time limit to complete the exercise

We represent the trainer’s feedback on these motions as a series of feedback cues. Each feedback cue is defined with the following attributes:

  1. (a)

    Parameter of feedback (progression, pacing, posture.)

  2. (b)

    Threshold of feedback (ex. “pace drops below ½ of expected speed”.)

  3. (c)

    Feedback modality (Visual, Audio, or Physical)

  4. (d)

    Description of feedback

A training protocol then simply becomes a set of feedback cues with the attributes above.

4.3 Virtual Training Software

To administer the at-home training component of the ATA using the information obtained from the Motion Authoring system, we developed a software interface which communicates with the Intelligent Stick prototype above to allow a trainee to complete the exercises assigned by a trainer. This software is developed in the Unity platform and the current prototype has been deployed on PC. The software consists of a Heads-Up Display (HUD) which displays visual information on an individual’s progress and a 3D embodiment of the virtual trainer which demonstrates the motion and mirrors (imitates) the trainee’s motion as he or she attempts the exercise. The software may be automated (screens are timed and progress automatically) or controlled manually, depending on the individual’s comfort with keyboard usage. Each exercise contains the following screens:

  • Exercise Title: Name of the exercise is shown.

  • Demo: Virtual trainer demonstrates the intended motion task with a text description on the screen.

  • Main Exercise Screen: Virtual trainer mimics the user’s motion with the Intelligent Stick while feedback is given as selected by the user’s real trainer.

  • Report: Shows individual’s performance in terms of number of reps completed, previous number of reps completed, and highest number of reps completed by that individual for that exercise.

A screenshot showing an example of the software interface is shown on the left in Fig. 3. This interface can vary between individuals, as the actual information shown on screen to each individual is determined by that individual’s real trainer via the Motion Authoring interface. This allows trainers to adapt not only the exercises and their complexity, but the complexity of the interface itself, based on each individual’s progress.

Fig. 3.
figure 3

Virtual Trainer software interface (left) and Kinect body motion capture (right)

Full size image

Because the sensors on the Intelligent Stick are insufficient for detecting and recording information such as body posture, an additional sensing mechanism is needed. In our implementation, we include the Kinect motion sensor as a simple, non-intrusive method by which to collect information on body posture. This sensing is shown in the screenshot on the right of Fig. 3.

5 Case Study

In a previous user study, we evaluated the usability of the Intelligent Stick prototype and motion authoring software with users who have no motor impairment [18]. Through this study we eliminated the most basic issues with the usability and accessibility of our device. In order to determine the usability and accessibility of the system in a rehabilitative setting, and to test the ability of our model to completely capture an interaction between trainer and trainee, we conducted a case study of the ATA system between a trainer and a trainee who meets the current user restrictions of our prototype (hemiparesis, upper extremity motor impairment, mild to moderate degree of impairment).

5.1 Procedure

The subject of the case study was a 12-year-old child with Cerebral Palsy who is hemiparetic as a result of the condition. The individual is undergoing martial arts defense training as a form of motor rehabilitation; consequently, the individual’s martial arts instructor became the trainer in the study. The study was conducted in the trainer’s martial arts training facility, and since the training already involved the use of stick equipment for martial arts defense techniques, we focused on basic stick training exercises as a testing platform for the Intelligent Stick device. The study was split into three sessions: regular training with the trainer’s stick equipment, training using the Intelligent Stick without vibrotactile feedback, and training using the Intelligent Stick with vibrotactile feedback.

For each session, we observed the interactions between the trainer and trainee on a set of four basic exercises: elbow flexion/extension, wrist flexion/extension, bimanual steering, and wrist ulnar deviation in the paretic arm. The first three of these exercises involve bimanual motion while the last exercise is unimanual. The exercises themselves are single-degree-of-freedom exercises selected by the trainer as a part of his training program, and each meet the limitations and restrictions of our model. For each exercise, the trainer began by demonstrating a motion task to the trainee for one minute and then guided the trainee through five minutes of exercise with that motion task. We recorded each session, observing the subject’s response to the trainer’s feedback and to the prototype. In the third session (Intelligent Stick device with vibrotactile feedback) vibrational cues from the Intelligent Stick were used to replace the trainer’s feedback in indicating when the trainee had reached the targeted degree-of-motion for each exercise. Observational data was used to compare the response of the trainee to this feedback relative to the trainee’s response to the trainer’s feedback. Furthermore we collected feedback from the trainer and trainee on the usability and accessibility of the device, including weight, strength of vibro-tactile cueing, and grip comfort.

5.2 Results

We began by capturing the four required motions using numeric values defined by the trainer under the definition of motion tasks within our model (speed shown in degrees/sec and time shown in minutes):

From the three sessions, we were able to observe the following categories of feedback from the trainer to the trainee during each exercise, all of which serve as quantifiable parameters within our model:

The feedback from the trainer was consistent across all three sessions, and was formatted to match the definition of a feedback cue within our framework. An important attribute of the audio modality was that, as most of the feedback was delivered verbally within this modality, it was given sequentially rather than in parallel. For example, if the subject’s posture and pacing both required correction, the trainer would correct one before correcting the other, with priority given toward posture over pacing. This feature of the feedback (priority assigned to one category over another when overlaps occur) is currently not present in our model and will be incorporated in our next iteration of the design.

For the session using vibrotactile feedback from the Intelligent Stick prototype, the trainer selected progression as a parameter and chose the endpoints of the motion (the rest point and the targeted degree-of-motion point) as the critical points which would trigger half-second vibrations from the device. The Intelligent Stick behaved as follows for the third session: for each motion, the stick would vibrate at the starting position and ending position (the targeted angle of motion) of the motion. The instructor chose the range-of-motion for each exercise based on his current training program. These values are shown in Table 1. The stick sent a vibrational signal at the start and end points listed above for each exercise in the third session. The subject was informed of the purpose of these vibrations before beginning the session, and responded to the vibrations as though they represented the stick equipment touching the palm of the trainer to represent a fully-completed motion.

Table 1. Case study motion tasks
Full size table

Feedback on usability was generally positive across all exercises. The trainer and trainee were both satisfied with the weight of the device and the responsiveness and amplitude of the vibrotactile feedback. However, there was a major concern with accessibility: as the subject had a weak grip strength in the paretic hand, it was difficult to secure a grasp on the device, which would hamper the subject’s ability to use the equipment at home. To remedy this, we incorporated the same solution which the trainer used with his regular equipment: we added a strap mechanism to secure the subject’s grip on the device which consists of a simple band that wraps around the wrist and is secured with a velcro strip. With the added grip mechanism in place, the subject was able to proceed through the training by using the Intelligent Stick prototype device as if it were regular training equipment.

6 Conclusion and Future Work

While the results of this initial study provided limited information regarding the efficacy of the virtual trainer as a solution deployed in the home setting, it has indicated that our model of action observation can be successfully incorporated within an existing training program by capturing the motion tasks and training protocol of the trainer. To address the effectiveness of the virtual avatar, we have begun a longitudinal case study wherein the same subject will use the device with regular monitoring and updates from the trainer in the home setting, using the training parameters shown in Tables 1 and 2, for a period of 6 months. The results of this deployed solution in long-term at-home training will provide further insights into how this model for motor learning can constitute an effective, optimal interface and training environment in the absence of a real trainer.

Table 2. Case study training protocol
Full size table

Upon demonstrating this to be a successful case for this subject, we will then take steps in future work toward generalization of the model and toolkit by addressing issues with respect to accessibility and usability across other populations, including the stroke population, and issues with respect to model accuracy and capacity for representation across multiple trainers and training programs. As the domain of training programs and assessment strategies by trainers captured within the toolkit continues to grow, so should the definition of motion tasks and feedback cues to accommodate the variation in these protocols. We hope that this model for training will serve as a foundation for a new generation of exergaming interfaces which can directly attribute their design choices upon the training strategies of expert therapists and trainers.