Keywords

1 Introduction

Skin and muscle receptors in the leg and foot provide able-bodied humans with crucial sensory information for balance and movement control. In lower-limb amputees however, this information is either missing or incomplete. In order to compensate for the partial loss of somatosensory function, people with lower-limb prostheses rely on haptic feedback from the interface between the socket and the residual limb to recognize gait-phase events and monitor balance [1]. This is problematic because the areas of the residual limb that directly interface with the socket are prone to painful skin irritation, which greatly degrades the haptic feedback users receive [2].

Without somatosensory input from the prosthetic limb, persons with lower-limb prostheses experience difficulty monitoring and correcting their center of gravity [3]. This manifests in measurable gait deficiencies such as impaired balance [1], asymmetrical walking patterns [4], larger stride-width than able-bodied humans [4], increased risk of falling [5], and decreased balance confidence [6]. These problems not only contribute to the difficulty of using a prosthesis in everyday life, but can result in the development of musculoskeletal diseases over time (e.g. osteoarthritis in the unamputated leg) [7].

Augmented sensory feedback systems have been established as useful tools to non-invasively replace missing somatosensory information in persons with lower-limb amputations. These systems generally fall into two categories: therapy aids, and assistive technologies. Therapy aids are used during rehabilitation sessions, and work by informing a patient when a specific biomechanical variable falls outside of a preset range [8,9,10]. Assistive technologies encode a specific biomechanical variable into stimulus patterns. A hypothetical example of an assistive technology for persons with lower-limb protheses would be a system that communicates ground-reaction forces through patterns of electrical stimulation on the thigh. Assistive feedback systems are intended to enable users to incorporate the feedback into their body control schemes, with the end goal of promoting more physiologically sound gait. These systems function through sensory substitution, a process in which the brain processes information from an alternative receptor in place of information that is normally transmitted through an intact sensory organ [11].

Several studies have found promising results from sensory feedback systems intended for daily use. Crea et al. [12] developed a system based on piezoresistive force sensors worn on the plantar surface of the foot, and vibrotactile actuators worn on the thigh. This system provided time-discrete vibrotactile stimuli to communicate gait-phase transitions. The efficacy of this system was verified through testing with able-bodied subjects. Fan et al. [13] proposed a similar system that communicated plantar ground-reaction forces through four pneumatically controlled balloon actuators worn around the thigh. Again, the efficacy of this sensory feedback scheme was verified through testing with able-bodied subjects. Lastly, Sabolich et al. [3] developed a device that encoded ground-reaction forces into continuous electrical stimuli displayed on the thigh via electrodes worn within the socket of the prosthesis. Results of testing showed significant improvements in weight distribution, balance time, and step length symmetry in both trans-tibial and trans-femoral amputees.

This study presents a novel haptic feedback system designed for daily use. The device consists of a custom force-sensing insole, a belt-worn microcontroller, and a linear array of vibrotactile motors worn on the thigh. The system communicates movement of the anteroposterior center of pressure (COP) to the user in real-time through time-discrete haptic feedback on the medial thigh.

To the best of our knowledge, this device is the first sensory feedback system to communicate movement of the anteroposterior COP through time-discrete vibrotactile stimuli. Time-discrete stimuli were used for two primary reasons: to increase the intelligibility of the stimuli, and to avoid some of the disadvantages of time-continuous stimulation. Previous studies supported that a time-discrete approach helped subjects perceive feedback patterns as rhythms [12, 14]. Additionally, time-discrete stimulus patterns have been shown to aid users in integrating the information into their body control scheme [15,16,17]. Furthermore, time-continuous stimulation can be perceived as aggravating by the user, and is likely to lead to habituation [12, 18]. A time-discrete approach was implemented to avoid these drawbacks.

This paper presents the design of the haptic feedback system, and verification of the stimulus approach through perceptual testing with 13 able-bodied subjects.

2 System Architecture

In this section, we detail the modules of the proposed haptic feedback system. In particular, we describe the hardware portion (sensorized insole, microcontroller, and haptic sleeve) and the feedback algorithm.

2.1 Hardware Design

The haptic feedback system is composed of a sensorized insole, a microcontroller in a belt-mounted enclosure, and a linear array of vibrotactile pancake motors, as detailed in Fig. 1.

Fig. 1.
figure 1

(a) Haptic feedback system as worn by an able-bodied subject. The system consists of a sensorized insole, wearable processing unit, and thigh-mounted one-dimensional vibrotactile array. (b) Detailed view of the vibrotactile motors mounted within the neoprene thigh wrap. (c) Diagram of the sensorized insole with FSR placement indicated by red dots. Quadrants 1–4 are as labeled. (Color figure online)

Full size image

The sensorized insole design was inspired by previous work by Rana [19], Al-Baghdadi et al. [20], Howell et al. [21], and Ferenczl et al. [22]. A flat foam insole was fitted with five piezoresistive force sensing resistors (FSRs; Tekscan A201) in positions corresponding to critical contact points on the foot: hallux, first and fifth metatarsal heads, and heel. Each FSR was calibrated to characterize the voltage-force relationship individually. An x, y coordinate system was imposed on the insole, and sensors were assigned coordinates corresponding to their individual positions. In this coordinate system, the x plane corresponded with the coronal plane, and the y plane corresponded with the sagittal plane. Analog voltage readings from the five FSRs were taken as inputs to the microcontroller, and used to calculate the COP.

A one-dimensional array of vibrotactile eccentric rotating mass pancake motors (tactors) was used to apply feedback. The tactors were 10 mm in diameter, and had a height of 3 mm. The tactors were soldered to a WS2801 flexible LED strip, and 5 V was applied to all tactors. The tactor array was driven by the microcontroller. Individual tactors were placed 5.1 cm apart, a greater distance than the 4.3 cm two-point discrimination threshold for touch on the thigh [23]. The vibrotactile array was mounted in a neoprene thigh wrap, and worn so that the array ran proximal to distal on the medial thigh. The medial thigh was chosen to display the stimuli based on previous research, which indicated that it is more sensitive to vibrotactile stimulation than the lateral or anterior thigh [24].

The current system was designed for laboratory testing with able-bodied subjects. The final version of this system will be fully integrated into prosthetic limbs, with FSRs mounted directly on the prosthetic foot and vibrotactile motors embedded in the socket liner. The processing unit may be incorporated into the prosthesis, or worn on the waist. Integrating the system into prostheses will ensure that users’ gait will be unencumbered by the device, thereby improving the usability of the system.

2.2 COP Calculation and Feedback Strategy

COP was calculated in the sagittal plane using the following weighted average equation,

$$ COP_{y} = \frac{{\mathop \sum \nolimits_{i = 1}^{5} F_{i} y_{i} }}{{\mathop \sum \nolimits_{i = 1}^{5} F_{i} }} $$
(1)

where Fi was the force registered by the ith sensor, and yi was the y position of the ith sensor, and COPy was the COP in the anteroposterior plane. Feedback control was handled by the microcontroller. The plantar surface of the insole was divided into four quadrants, as shown in Fig. 1. Each quadrant directly corresponded to one of the four tactors. Movement of the COP during the stance phase was communicated to the user though geographically mapped haptic feedback on the thigh. The feedback scheme during a normal step is as follows: When the COP is in the heel region during the heel-strike, the most proximal tactor fires. As the COP continues to progress through the quadrants from the heel to toe region, corresponding tactors fire. Finally, when the COP enters the final quadrant during toe-off the most distal tactor is fired.

Two strategies were employed to ensure that tactor events only occurred during the stance phase of the gait cycle, and stimulus patterns remained time-discrete. First, individual tactors would fire for 100 ms. This time interval was chosen so that the individual tactor events were long enough to be recognized, but not overlap during use. Second, a force threshold was implemented based on user weight and the magnitude of ground reaction forces during individual users’ gait. Tactors were only fired when the total ground reaction forces were above this threshold value. While the total ground reactive forces were continuously above the force threshold during the stance phase of gait, subsequent tactor events were triggered by the COP entering a new quadrant.

The proposed system was designed to communicates changes in the anteroposterior COP alone. Providing the user with only essential information for gait control and balance was intended to avoid sensory overload [25]. The anteroposterior COP was chosen in this system due to the relationship between the heel-to-toe COP movement and gait phase transitions [26, 27].

3 Experimental Methods

The experiments presented here tested the haptic feedback system’s efficacy in communicating COP information to the user. Perceptual testing was conducted prior to any integration into prostheses or application to people with lower-limb prostheses. This was done as a preliminary step to assess if users can interpret meaningful information from the haptic patterns.

All tests were conducted with able-bodied subjects who were seated wearing the haptic sleeve. Step patterns were generated based on hypothetical patterns that an end-user would receive during gait. These patterns were applied to the subjects by the experimenter via a PC serial connection with the microcontroller. Response times (RTs) were obtained using a timer integrated into the GUI used by the experimenter. The timer was started when a stimulus pattern was administered and stopped by the experimenter when the subject spoke their response. All subjects participated in both Experiments 1 and 2 in order.

3.1 Experiment 1: Classification of Step Pattern Speed

Subjects.

For this experiment, 13 able-bodied subjects (7 male and 6 female) between ages 20 and 32 were recruited to participate. The subjects had no known sensory impairments.

Experimental Setup.

The haptic sleeve described in Sect. 2.1 was used. Subjects were seated for the duration of the experiment. The vibrotactile motors in the sleeve fired sequentially and administered 100 ms pulses.

The stimulus patterns in this experiment focused on classification of step pattern speed. Stimulus patterns were developed to represent slow, medium, and fast steps. Stance times were selected to be 850 ms for the Slow pattern, 700 ms for Medium, and 550 ms for Fast based on laboratory testing with one subject. Tactors were activated sequentially at evenly spaced time intervals through each pattern.

Procedure.

Subjects were instructed to wear shorts or pants, and the haptic sleeve was worn on top of the clothing. The sleeve was positioned on the subjects so that the vibrotactile motors ran along the medial thigh, and the most distal tactor (tactor #4) was approximately 4 cm above the knee. Subjects were instructed to rest their forearms on the arms of the chair to prevent the perception of additional sensory information from the haptic sleeve through the forearms.

Once situated in the apparatus, subjects were familiarized with the three step patterns (Slow, Medium, and Fast). The experimenter spoke the name of each pattern, then applied each stimulus in order with approximately 5 s between stimuli. This procedure was performed four times for each subject. The training phase consisted of 18 trials, where each pattern was administered 6 times in random order. Subjects were instructed to guess what each pattern was, and feedback was given to correct or confirm their responses. Like the training phase, the testing phase consisted of 18 trials, however no feedback was provided. Correct and incorrect responses were recorded, along with sensorimotor reaction time (time between application of the stimulus and subject response).

3.2 Experiment 2: Classification of Full or Partial Step Patterns

Subjects.

The 13 subjects that participated in Experiment 1 also participated in Experiment 2.

Experimental Setup.

The apparatus in Experiment 2 was identical to that of Experiment 1, however the stimulus patterns were different. The stimulus patterns in Experiment 2 were as follows: Full, Partial-123, and Partial-234. In each of these patterns, tactors were active for 100 ms periods, and there were 100 ms pauses between each tactor firing. The Full pattern consisted of tactors #1–4. Partial-123 recruited tactors #1–3, and Partial-234 recruited tactors #2–4.

Full and partial step patterns were related to changes in the movement of the COP in normal and abnormal gait. The ability to recognize these differences is the basis of understanding changes in gait phase through the haptic feedback system. For example, if a person were to take a physiologically-sound step on flat ground, the COP would progress from the heel to the toe, resulting in a Full pattern. If a person were to take a step and not complete the toe-off gait phase, this would result in a partial pattern similar to Partial-123. Similarly, if a person were to take a step and land flat footed, omitting the heel strike phase, it would result in a pattern similar to Partial-234.

Procedure.

In Experiment 2, subjects were familiarized with the stimulus patterns that were tested. The experimenter spoke aloud each of the stimulus names, then applied each pattern sequentially. This was repeated four times for each subject. The subjects then went through a training phase, which consisted of 18 trials with each stimulus presented 6 times in random order. Feedback was given by the experimenter to confirm or correct subjects’ responses. Testing was performed after the training phase, also with 18 stimuli presented. The procedure for testing was identical to the training procedure, however no feedback was given by the experimenter. Correct and incorrect responses, and sensorimotor reaction times were recorded.

4 Results

4.1 Classification of Step Pattern Speeds

The classification accuracy from Experiment 1 was averaged across all subjects and summarized in Fig. 2. The overall classification accuracy for Experiment 1 was (92.3 ± 2.6)%. A one-way ANOVA was performed on the classification accuracy data for all three patterns, and the results indicated that there were no significant differences in accuracy between any of the stimulus patterns (F(2) = 1.80, p = 0.18). The distribution of subject responses is shown in the confusion matrix in Table 1. Out of all incorrect responses, only one was not from a directly adjacent speed pattern.

Fig. 2.
figure 2

Classification accuracy for Experiment 1. Error bars represent 95% confidence intervals.

Full size image
Table 1. Confusion matrix for Experiment 1 showing classification accuracy. Rows represent the stimulus provided, and columns represent subject responses.
Full size table

The response time averaged across all stimulus patterns was 1.25 ± 0.04 s. A one-way ANOVA was performed on the data from the three stimulus groups, which indicated that there were significant differences between the RTs of at least 2 of the groups. Post hoc paired t-tests with Bonferroni Correction were performed between each of the three stimulus groups. Significant differences (F(2) = 13.7, p = 3.80E-05) were found between RTs associated with Fast (1.0 ± 0.03 s) and Medium (1.41 ± 0.08 s) patterns, and Fast and Slow (1.32 ± 0.05 s) patterns.

4.2 Recognition of Full and Partial Patterns

The classification accuracy in Experiment 2 was averaged across all subjects, as summarized in Fig. 3. The overall classification accuracy was (94.9 ± 2.1)%. There were no significant differences in classification accuracy across the three stimulus patterns, as indicated by the results of a one-way ANOVA (F(2) = 1.97, p = 0.155). The subjects’ performance in Experiment 2 is detailed in the confusion matrix in Table 2.

Fig. 3.
figure 3

Classification accuracy for Experiment 2. Error bars represent 95% confidence intervals.

Full size image
Table 2. Confusion matrix for Full and Partial Pattern testing. Rows represent the stimulus given, and columns represent subject answers.
Full size table

RTs averaged across all stimulus patterns were found to be 1.32 ± 0.03 s. A one-way ANOVA was performed on the RT data from all three stimulus patterns which supported that there were no significant differences in RT between any of the patterns (F(2) = 0.208, p = 0.81).

5 Discussion

Artificial sensory feedback systems have great potential for improving the mobility of people with lower-limb prostheses by compensating for lost somatosensory function. For an artificial sensory feedback system to be useful in dynamic situations such as gait, it is essential that the feedback patterns can be classified accurately and rapidly by the user [28]. Data from the two perceptual tests indicated that potential feedback patterns from the proposed system can be classified accurately by the subjects. Furthermore, results showed that RTs were relatively quick and consistent. Low RTs suggest that classification was not cognitively demanding, but further work must be done to verify this.

Classification accuracy was very high in both experiments, despite relatively brief training periods. Subjects classified speed patterns with an accuracy of (92.3 ± 2.6)%, and full and partial patterns with an accuracy of (94.9 ± 2.1)%. Results from one-way ANOVA tests indicated that there were no significant differences between classification accuracies of any of the stimulus patterns within each experiment.

The classification accuracy values in our study were similar to those found by Fan et al. [13] despite a large discrepancy in the time each tactile unit was fired and the full pattern times. Their system consisted of four tactile units worn on the thigh, and patterns were made up of three sequential tactile stimuli, all 1 s in duration for a total pattern duration of 3 s. In our system, vibrotactile motors were fired for 100 ms, and pattern durations ranged between 550 ms and 850 ms. Achieving similar accuracy with shorter active motor times indicated that this system can respond to more rapid COP changes during gait without a significant drop-off in classification accuracy.

In addition to high classification values, subject misclassifications were generally close to the correct responses. In Experiment 1, only 1 out of 17 total misclassifications was from a non-adjacent speed pattern. In Experiment 2, all misclassifications came when Partial patterns were administered, and only 1 of the 12 total misclassifications was incorrectly identified as the Full pattern. This indicated that when the subjects failed to correctly classify the stimulus patterns, their responses tended to be close to the correct responses. When the complete system is implemented, users will encounter stimulus patterns with much more subtle differences than the ones tested here. The ability to identify exactly how the COP is progressing in all cases is ideal, but being able to roughly interpret the movement of the COP and understand how it compares to stimulus patterns from other steps may be sufficient. Future perceptual testing will examine the minimum perceivable difference between stimulus patterns to find how similar patterns can be while still remaining consistently classifiable.

Subject response times were relatively low across all experiments. The overall RTs were 1.25 ± 0.04 s for Experiment 1, and 1.32 ± 0.03 s for Experiment 2. Within Experiment 1, there were significant differences between the average RTs to Fast and Medium, and Fast and Slow patterns. In both cases, the Fast RT was significantly lower than the other RT. Factors that could have lead to this discrepancy were pattern length, and the method of recording RT values. Developing a system in which the subject responded and stopped the timer by pressing a button may have decreased the observed RTs, as response modality has been demonstrated to have a significant effect on observed RT in previous studies [28].

This initial study featured a small subject pool, and varied tactor contact due to differences in subjects’ clothing and thigh length. Despite these factors, the system was shown to perform well, as evidenced by the high classification accuracies and low RTs observed across all subjects in both experiments. These initial results supported that the proposed system was able to sufficiently communicate data about step speed and full or partial steps to the user, and that this feedback strategy may be effective in dynamic conditions such as gait.

6 Conclusion

In this paper, we have presented a novel haptic feedback system to deliver anteroposterior COP information to persons with lower-limb prostheses in the goal of substituting for missing somatosensory information useful for gait. In addition to the description of the hardware and stimulus algorithm, perceptual experiments were performed with able-bodied subjects to determine the efficacy of the proposed feedback system in communicating COP information to the user. From the results of these experiments, it was ascertained that the subjects could accurately recognize differences in the speed of step patterns, and between full and partial step patterns. Future studies will proceed in two primary directions: testing the efficacy of the haptic feedback system in communicating COP data based on challenging conditions for gait (uneven terrain, sloped surfaces, stairs), and testing subjects’ ability to classify step patterns during gait.