A Soft Exoskeleton Jacket with Pneumatic Gel Muscles for Human Motion Interaction

Abstract

This work proposes to use an assistive device to augment the perception of human motion force applied from one subject to another through an avatar. This experiment presents an avatar augmentation that can provide the feeling of motion with two degrees of freedom at the elbow and shoulder acquired with algorithmic detection of input angles. To generate the operation of the appropriate motion, the interface is comprised of a depth sensor, an ESP32 embedded board, electric valves, and pneumatic gel muscles. An evaluation is presented which confirms the performance of the suit by measuring the latency of the system. The experimental results demonstrate that the developed suit can convey the motion of one user to another with a delay of 670 ms.

1 Introduction

Elderly society is generally afforded very little free time [1], and in most cases it may be difficult to go to a hospital when an injury occurs. Furthermore, hospitals generally include a limited number of therapists relative to the demand for rehabilitative therapy, which necessitates the implementation of alternative means of supervision and training.

Due to technological advances in modern science, this challenge is being addressed through externally worn assistive force output devices or “exoskeletons”. These devices are capable of providing external support to augment human motion, and in many cases, to improve performance in a variety of tasks. Scientists have explored the application of this technology to other fields of study beyond force-based human augmentation; for example, exoskeletons have also been applied in the areas of rehabilitation and tele-operation. Typically, these exoskeletal assistive devices have consisted of bulky, rigid and heavy electric machinery; recently, however, “soft” lightweight alternatives to these interfaces have been developed. This new generation of exoskeletons, called “soft exoskeletons”, utilize lighter, more flexible materials to generate externally-delivered force along the body. Unlike their counterparts, soft exoskeletons lack an external rigid frame, which serves as both an advantage and a disadvantage [2]. The primary drawback of this design is that these wearable devices have difficulty in transferring power from any area of the body to the ground. In addition, motors and sensors are more difficult to mount. Furthermore, torque and force generated by actuators enter the user’s body. Imagine for example an elderly individual who uses an exoskeleton to go the store and back, in which case the exoskeleton must assist with balance, usage of stairs, entering and exiting a vehicle (currently no product on the market has this capability although technology is rapidly overcoming this limitation). A soft exoskeleton can provide the additional force to supplement a user’s muscles [3], but most if not all of that energy will take a toll on that individual’s body. The elderly and disabled populations already have weakened bones and muscles, so adding more strain to their bodies may be counterproductive.

With the relative advantages provided by soft exoskeletons, it is possible to provide a device to a patient to assist with the performance of rehabilitative exercises at home. One challenge with this approach is that without the physical presence of the therapist, the force-guidance protocol must be as intuitive as possible to avoid misguidance or injury. To address this, the proposed system is an integration of a soft exoskeleton suit with a natural user interface including as Kinect sensor, and is designed to prove an interaction that is direct and consistent with our ‘natural’ behavior, without requiring additional sensor attachments on the learner. The advantage of Natural User Interfaces (NUIs) is that the user interaction feels fun, easy and natural because the user can utilize a broader range of basic skills compared to more traditional graphical user interface interaction. In this case a camera-based motion tracking method can be used by the therapist to supervise a patient’s posture during exercise, process and analyze this information, and activate the appropriate valves of Pneumatic Gel Muscles (PGMs) attached in specific locations along the exoskeleton to generate corrective gestures and guide this patient.

In addition, Internet of Things (IoT) boards, such as Raspberry Pi or ESP32, can be used as embedded systems to add their many capabilities in the provision of services such as tele-rehabilitation [4, 5]. This suggests that if the output user (the user receiving the assistance) wears a soft exoskeleton, he or she can receive tele-rehabilitative coaching to correct posture during exercises, receive direct and active routine guidance by physiotherapists [6], and participate in direct communication with the hospital from within the user’s home. In addition, if a set of output users assigned to a single therapist each wear different jackets, the therapist can potentially provide rehabilitative sessions to several of these users at the same time, assuming that these patients are assigned the same routine. This process is entitled “mimic motion”.

Several works have developed soft exoskeletons and telerehabilitation systems with ingenious strategies to improve the performance of these procedures. For example, several works have documented the use of Kinect and the IoT to acquire remote patient data and perform therapy studies without the need for the subject to personally attend the hospital [7, 8], and in assistive suits to support elbow motion with soft materials [9, 10] to provide a light alternative to traditional hard exoskeletons.

One of the objectives of these technological areas is the development of innovative systems and/or applications that offer services that were not available until recently.

For example, the process of introducing new technologies that offer new applications for human augmentation as wearable devices has been explored from the design stage to the prototyping of terminal equipment. Undoubtedly the field of modern human augmentation devices is one of the most dynamic and fastest growing segments in the areas of wearable assistive devices [11]. This paper proposes the use of new technologies for human augmentation as a tele-coaching system generating gestures in a VR environment. It consists of a jacket that equips PGMs developed by Daiya Industry Co. Ltd. [12] in various joints along the upper body, along with their corresponding valves, a microcomputer and an Oculus VR system. An individual wearing the proposed soft exoskeleton jacket can feel the motions of another individual providing guidance from a remote location. Figure 1 depicts the framework of the purposed system.

Fig. 1.

Framework of the presented system.

2 Pneumatic Gel Muscles

In this system, McKibben type artificial muscles (PGMs) [14] were used. The McKibben artificial muscle is the most popular PGM. Its behavior resembles biological muscles. The McKibben artificial muscle is comprised of an inner tube surrounded by braided mesh. The inner tube is made of a stretchable rubber tube and the braided mesh provides protection and controls contraction of the artificial muscle. The pantograph structure of the braided mesh allows the muscle to extend and contract easily. The contraction of the actuator depends on the air pressure and volume and the non-extensibility of the braided mesh shortens and produces tension if the endpoint is attached to a load. Such actuators are favorable for motion assistance because of their similarity to human muscle functions [15].

This artificial muscle contracts and expands according to the air pressure fed to one of its ends. Figure 2 demonstrates the method of operation of the PGM. The PGMs are specially designed low-pressure muscles which can be efficiently operated within a range of 0.1 MPa to 0.3 MPa air pressure. These artificial muscles are light-weight, flexible, and their contraction behavior resembles that of human muscles. Therefore, PGMs are highly reliable for usage in developing soft force-feedback providing systems (Fig. 3).

Fig. 2.

Pneumatic gel muscle actuation [15].

Fig. 3.

Comparison of results between PGM and PM-10RF [15].

3 Design and Implementation

Here the procedures and design in the implementation of the proposed system are described in detail, including the acquisition of movements of the user, processing of this motion data to calculate the angles used for activation of the jacket, transmission of activation data, and the usage of this data by the microcontroller to activate the appropriate PGMs. The challenges and limitations encountered during development and the steps taken to overcome these issues are also explained. Figure 4 presents a block diagram of the design implemented for the system.

Fig. 4.

Block diagram of the system.

3.1 Acquisition of the User Data

The sensing environment relies upon a Kinect V2 sensor, a depth sensor used to acquire the information of the input subject (the individual providing the motion guidance). The acquired information is processed in the PC to estimate the position of the input user joints based on the Kinect SDK algorithm. This information is then displayed in a Unity 3D environment to provide a visual experience to both the input user and output user. The output user can then view the motion of the input user in the first-person perspective by wearing an Oculus helmet that displays a VR environment as shown in Fig. 5.

Fig. 5.

Avatar displayed in the VR environment developed to emulate the tele-existence of the input user.

Angle Calculation.

The primary objective at this stage is to calculate the angles between each limb and its adjacent body parts. The PGM will then be activated to reproduce the angles of the user’s shoulder. The “angle of the shoulder” refers to the angle that the shoulder joint creates between the upper arm and the torso. Figure 6 depicts the angle that to the system must track to emulate gestures involving shoulder motion. The desired values can be derived by calculating the input user’s shoulder angle against the vertical line of the torso and the elbow angle about the orientation of the upper arm. Therefore, before calculating these angles, the vectors corresponding to each of these axes of orientation must be created. Subtracting two vectors produces a third vector representing the orientation between them. The axes of orientation can be created in this way by subtracting two pairs of skeleton joints. For the shoulder joint, a vertical axis that points up along the orientation of the user’s torso is necessary. It is calculated by subtracting the right hip from the right shoulder and left shoulder. Since those joints are approximately vertically aligned on the body, the vector between them captures how the user is leaning away from vertical. Once these orientation axes are formed, the calculation of the angles of the limbs can be completed. The logic of the function that was applied is demonstrated in Fig. 7.

Fig. 6.

Joints and limb required to calculate the angle which generates the gestures in the output user.

Fig. 7.

Demonstration of conversion of the two vectors representing the ends of a limb into a single vector representing that limb in relation to a given orientation.

3.2 Receptor

The receptor is deployed within the exoskeletal jacket. Here, one socket connection is created to generate the bridge between the client and the server. A socket connection provides a near real-time communication, also known as synchronous [16]. Synchronous communication is also necessary for live performance applications that need real-time interaction between elements. A network socket connection is a continuous connection between two programs across a network, each consisting of an IP address, the numeric address of a machine on the network, and a port number, a number used to interact between elements. One example of a simple socket connection is shown in Fig. 8. The project tests were carried out with one embedded system. The ESP32 card, with Wi-Fi capabilities, was programmed as a server whose task is to receive the information of the client and activate the outputs that are connected to the valves to actuate the PGM located in the jacket. This card is stable because the routine is dedicated to generating the server and receiving the information that is continuously sent, and it is not necessary to halt this routine with any delay when the port is receiving information from the client to control the valves. The primary issue in this case is that in order to change the routine of the card, it is necessary to reprogram from the PC. In addition, because this card is smaller, the power source was also embedded within it to connect it directly with the board.

Fig. 8.

Socket connection overview.

3.3 Soft Exoskeleton Jacket

The proposed soft exoskeleton consists of a soft lightweight jacket, pneumatic gel muscles located in the arms, the receptor as previously described, a set of connective tubes, a CO2 tank, and a power bank to energize the system. Figure 9 provides an overview of the elements within the developed suit. In robotics terminology, it can be stated that this jacket has two degrees of freedom (i.e., it can only move along two axes). The jacket holds 3 PGMs of 42 cm and one of 27 cm dedicated to the shoulder’s extension motion. In this approach, the PGMs are placed in a straight alignment. For the elbow’s flexion motion, unlike the previous configuration, 2 PGM’s of 42 cm were twisted at the height of the user’s elbow to provide more torque and a better feeling to the user. Figure 10 indicates the configuration followed for the PGMs in the elbow and shoulder locations.

Fig. 9.

Elements that contain the soft exoskeleton jacket.

Fig. 10.

PGM’s configuration viewed from different perspectives.

The system, once all of the previous steps are completed, can actuate the PGMs in the elbow and shoulder joints so that the motion of elbow flexion is supported, and the extension of the shoulder is supported. This provides 2 degrees of freedom. Min. angle elbow

One significant consideration in this design strategy is to maximize the mobility of the user while wearing the exoskeleton to avoid issues with executing these motions.

4 Evaluation

A preliminary set of experiments was conducted to confirm the delay that exists in the system and to determine the degree of force felt by the user through the force-feedback system. Four male adults and one female adult participated in this evaluation. None of the participants in the studies had any previous knowledge of the system. Before evaluation, the task was to measure the minimum and maximum angle detection of the Kinect sensor for the joints of the elbow and shoulder. It was found that the Kinect could detect a minimum of 17° and a maximum of 177° for the elbow, and a minimum of 5° and a maximum of 178° for the shoulder.

After determining the effective range of the sensor, the PGMs of the suit were activated in both the shoulder and elbow configurations to calculate the maximum angles that could be achieved using the suit. Figure 11 displays the maximum. angle that can be reached in both configurations of the presented system after PGM actuation. The horizontal axis of Fig. 11 shows the conditions and the vertical axis the angles, in degrees. The evaluation found that the presented system can reach 62° for the shoulder extension configuration and 84° approximately for the elbow flexion configuration. This includes the points adjusted by the PGMs. After knowing these maximum angles, various threshold values were formed to emulate gestures of the input user in the output user. Two threshold intervals were derived to emulate pulling up of the elbow of the output user; the first is between 25° and 30° and the second is over 30°. For the second mimic, a threshold of 25° was set to activate the valves to emulate pulling up of the shoulder of the output user by the motion of the input user.

Fig. 11.

Maximum angle of joints after PGM actuation.

The next evaluation measured the delay of the system from the user detection to the valve actuation. This test was performed 30 times with each subject. The delay between the timing of opening the valve and the timing of generating the assist force due to the characteristics of the PGM was not measured in this study.

The third test consisted of measuring the maximum force generated by each PGM with the configurations mentioned in the soft exoskeleton jacket configuration. To this end, a force transducer model Leptrino PFS080YA501U6 was used. The procedure started with the shoulder configuration, wherein the back of the subject’s hand is placed parallel to the sensor, very close to its surface but without touching it, and the subject’s arm was relaxed without opposing force. Under these conditions the activation of the shoulder muscle configuration was performed 15 times per subject. The measurement device was held in a horizontal frame to ensure close contact with the back of the hand as is shown in Fig. 12. Due to variations in the height and length of the arm of each subject, the sensor was adjusted to ensure consistency of measurements.

Fig. 12.

Shoulder extension measurement.

The next experiment focused on measuring the maximum torque generated by the elbow configuration. Each subject was asked to place his or her arm horizontally to the trunk as is shown in Fig. 13. Once this requirement was met, the PGM configuration was actuated, and the palm of the hand was aligned horizontally with the transducer to achieve the measurement.

Fig. 13.

Elbow flexion measurement.

5 Results

The results from the previous evaluations are as follows:

Latency.

Figure 14 shows the latency presented by the system when the calculations for pose estimation, angle derivation and reception of data were executed. The horizontal axis shows the conditions and the vertical axis is timing in milliseconds. It is possible to reach a delay of approximately 602 ms to estimate the joints of the current user, approximately 55.6 ms are required for the angle calculations to control the valves, and approximately 13.43 ms are required for the reception of the data and transmission of the signal to control the valves.

Fig. 14.

Latency presented in the system.

Maximum Assistive Force Measured for the Gestures.

Figure 15 shows the graphical representation of the physical force exerted by the PGM actuation by the shoulder configuration in the shoulder extension measurement. This experiment was illustrated in Fig. 12. The vertical axis represents the maximum force in Newtons and the respective perceived forces of the 5 subjects. We observed that different subjects perceived different levels of forces with the force-feedback system. The physical force induced by the actuation of PGMs was observed as approximately 5.4 N. As discussed in previous research, muscle activity plays a significant role in distinguishing different weights or forces (in this context). Perceived force is greatly dependent on the operating range of the muscles involved [17], and also depends on the length of each user’s arm.

Fig. 15.

Force-feedback experiment results for shoulder.

Figure 16 shows the representation of the torque generated by the PGM actuation by the elbow configuration in the shoulder extension measurement, an experiment that was illustrated in Fig. 13. The vertical axis represents the maximum force in Newtons, and the horizontal axis represents the respective force perceived by the 5 users. As observed in the previous experiment (Fig. 15), in this experiment the subjects perceived different levels of forces with the force-feedback system. However, the physical force induced by the actuation of PGMs for this configuration was differed from the previous by approximately 15.3 N. Muscle torque depends on musculoskeletal geometry [18], and the anatomy of a muscle has a pronounced effect on its force capacity, range of motion, and shortening velocity. Various examples exist demonstrating the characteristics leading to the relative change in force in this experiment compared with the previous one:

Fig. 16.

Force-feedback experiment results for elbow.

• Change in muscle length

• Rate of change in contractile force varies with the ratio of change in sarcomere length.

In addition, movement, or the muscle-controlled rotation of adjacent body segments, means that the capacity of a muscle to contribute also depends on its location relative to the joint that it spans. The rotatory force exerted by a muscle about this joint is referred to as muscle torque.

The moment arm usually changes as a joint rotates though its range of motion; the amount of change depends on where the PGM is attached to the body relative to the skeleton joint.

Restrictions of the System.

The solenoid valves presented a delay of 3/6 mS [19]. Furthermore, the nature of the pneumatic gel muscles presented a hardware delay that cannot be reduce as a restriction of the current actuators.

6 Application

As was mentioned in the introduction of this work, most of the soft exoskeletons that were developed include sensors attached on the user’s body and don’t use completely soft actuators, such as the pneumatic gel muscles that were used in the developed work. Solutions that were developed for telerehabilitation with the Kinect sensor tend to focus on obtaining data during remote therapy sessions without force feedback interaction. Taking into account these previous considerations, the current application that was developed in this project takes advantage of natural user interface systems, such as the Kinect sensor, to directly project the motion of an input user to an output user using a soft exoskeleton jacket through mimic motion.

The system can be used in rehabilitation or telerehabilitation to provide motion force feedback during remote physiotherapy exercise for elbow and shoulder rehabilitation after upper extremity injuries such as fractures, dislocations and tears. In addition, due to the soft and lightweight nature of pneumatic gel muscles capable of providing resistive force, this implementation may be useful in low-intensity resistive training for elderly individuals with cardiac problems as an example [20]. Figure 17 shows an example of mimic motion for elbow flexion wherein the input user generates a gesture in the VR environment to pull up the hand of the output user. Once the threshold angle is reached, the control signal will reach the receptor in order to actuate the valves so that the output user can feel the gesture.

Fig. 17.

Mimic gesture displayed in a VR environment.

7 Discussion

In the evaluation, an effective transmission of motion toward an output user wearing an exoskeletal jacket was observed. However, there are several factors which can be improved to yield more reliable indicators.

• The first such improvement is a reduction in the delay of the system by migrating to another SDK or acquisition method.

• Secondly, in addition to the Wi-Fi via LAN connection, a WAN connection can be implemented. This will allow the output user to experience the motion of an input user from a more remote location.

• Third, as the Kinect sensor can track the joints of the legs, it is also possible to actuate PGMs with the angles of the ankles or legs.

• Fourth, while PGMs can provide sufficient force-feedback to generate the gestures, the length of the user’s arm can directly impact that user’s perception of the force due to the static size of the jacket prototype. To remedy this, various sizes of the jacket can be implemented.

• Finally, a future evaluation can determine how the current system could play a physiological role in the current patient by studying how the perception of force guidance from the jacket could improve a user’s performance in rehabilitative exercise.

8 Conclusion

In this study, a soft exoskeleton jacket was developed with pneumatic artificial muscles for human motion interaction by using low-pressure-driven artificial muscles.

Future work includes the improvement of the algorithm to reduce the delay in the pose estimation process, and the reduction in the size of the control step. We plan to design a rehabilitative exergame with the developed system.

The experimental results allow us to conclude that:

• The pneumatic gel muscles are a good alternative for hard actuators, regarding price, flexibility and naturalness in the adaptation of the human motion.

• The system presents robustness in the acquisition of the motion of an input user to project to an output user.

• The system has flexibility in tracking various motions of an input user, due to the flexibility of the PGM, such that new gestures can be implemented which were previously difficult to deploy.

• The implementation of this system represents a strong relationship between cost and benefit.