Keywords

1 Introduction

While recent virtual reality (VR) devices, such as the HTC Vive or Oculus Rift, provide high resolution images, high fidelity haptic interfaces could ideally be used to provide physical feedback to improve VR immersion.

Hands play a particularly important role in interactions with objects and characters in VR environments. While commonly available VR systems provide a VR controller to input hand motions and output vibrotactile feedback, the vibrotactile feedback lacks high fidelity. However, there are several potential avenues that could be explored in terms of other types of haptic feedback devices.

Different kinds of haptic devices have been the subject of research to provide haptic feedback on the wrist and hand. Some of these have taken the form of glove-type devices [1, 2] and torque-feedback devices [3, 4].

Fig. 1.
figure 1

Haptic device combined with EMS and hanger reflex (left) and Virtual Reality application using the proposed system (center, right).

Full size image

Hanger reflex and electrical muscle stimulation (EMS) have also been explored as methods for providing force sensations in previous research projects on hand-mounted haptic devices.

The hanger reflex is a phenomenon that causes a force sensation and involuntary head rotation when specific pressure is applied to the head with an attached wire hanger. A similar phenomenon has been observed with the wrist as well, and it has been used to control the wrist [5]. Hanger reflex can cause wrist supination and pronation. Supination is a rotational movement where the forearm or palm are rotated outwards. Pronation is a rotational movement where the hand and upper arm rotate inwards. This means that hanger reflex can be used to realize control of the wrist on its rotational axis.

EMS uses electrical impulses to elicit involuntary contraction in the subject’s muscles. This technique has been used for controlling a user’s hand or wrist for different purposes, for example, for playing an instrument [6, 7], for VR feedback [8, 9], and for plotting data [10]. EMS can be used to control the wrist on the pitch axis accurately, since it is effective in stimulating muscles to cause flexion and extension, i.e., the movement of bending the palm down or of raising the back of the hand, respectively.

However, both hanger reflex and EMS have limitations as methods for controlling the wrist. In the case of hanger reflex, it can be used to control the wrist on the roll axis, but it is difficult to control on the pitch axis. On the other hand, EMS is efficient in controlling the wrist on the pitch axis, but not as precise on the roll axis, since the muscles used to cause wrist supination and pronation are too complex to stimulate the inner muscles directly.

To address these problems, we propose a method that combines electrical muscle stimulation and hanger reflex (Fig. 1). Hanger reflex is used to elicit the supination and pronation, while EMS is used to cause the flexion and extension.

A VR application was also designed and implemented where users could have interactions with a VR environment with haptic feedback. A user study was conducted to investigate the effects of using the proposed haptic device along with the VR application. The findings indicate that users can have more enjoyable and realistic VR experiences using the proposed device.

2 Implementation

The proposed system included two technologies: EMS and hanger reflex [11]. In this section, we introduce how we designed and implemented the hardware and software for each of the two technologies. We also explain the design of the VR application (Fig. 1) that allows a user to experience a VR environment with the haptic feedback.

2.1 EMS for Flexion and Extension

The EMS technique was used to induce the flexion and extension. The circuit that generated EMS applied voltage pulses into the muscles of a participant through two pairs of electrodes placed on the arm. Figure 2 shows where electrode pads were attached on the arm for both flexion and extension. The strength of the electrical muscle stimulation from the electrode was controlled independently by a microcontroller (Arduino Mega) changing the impedance of digital potentiometers (MCP4131). Each digital potentiometer was connected to the microcontroller through a serial peripheral interface (SPI). The EMS voltage could be adjusted in the range of 0–50 V, and it was adjusted according to individual differences.

Fig. 2.
figure 2

Electrical muscle stimulation device configuration (left) and the pad positions used to induce flexion and extension (right).

Full size image

The circuit generated an EMS pulse, whose pattern was controlled by a microcontroller (Arduino Uno). The EMS waveform was set to produce a 40 Hz periodic pulse signal with a pulse-width of 200 \(\mu \)s.

2.2 Hanger Reflex for Supination and Pronation

A 3D printed wristband was designed that can be fixed to a wrist for invoking hanger reflex in the wrist. It was reported that the perceived force from hanger reflex is enhanced when a vibration is also used, as presented in previous study [12]. Two vibrators were thus attached (Vibro Transducer Vp210) to the wristband to change the intensity of the reaction computationally. The vibration is a 50 Hz sinusoidal wave, which has been previously reported as the most effective for use in enhancing hanger reflex in the wrist. Two wristbands were attached to the wrist as shown in Fig. 3 (right), so that the wrist could rotate in two different directions: the pronation and supination. The vibrators were activated in one of the two wristbands based on the desired direction of the wrist rotation.

Fig. 3.
figure 3

Hanger reflex device configuration (left) and the two wrist bands that induce pronation and supination (right).

Full size image

2.3 Application Design

Overview. A shooting game was designed as a VR experience (Fig. 1) where the participants played and went through a scenario using a head-mounted display (HMD). In this study, the HTC Vive Pro was used as an HMD and HTC Vive controller. In addition to the headset and controller, the participants wore the hardware devices for EMS and hanger reflex.

Fig. 4.
figure 4

Shooting wooden targets using a gun in a VR environment.

Full size image

In the shooting game, two different events were designed that a player could interact with in the VR environment. For the implementation, Unity (Version 2017.4.1f1) was used, which is a game engine platform used for game creation. Software implemented in Unity communicates with a microcontroller to send signals to activate haptic feedback or to change the intensity of the feedback in response to events within the VR environment.

Shooting Wooden Targets. In one of the events in the shooting game, the players tried to shoot wooden targets with a gun in the VR environment (Fig. 4). When players successfully hit the target, it exploded and could destroy other nearby wooden targets as well. The event was designed so that the player received a sensation of force on the wrist when they shot the gun. EMS was utilized for the haptic feedback for this event to mimic the sensation of shooting a gun. In this event, there were nine wooden targets and the goal for the player was to destroy all the targets as quickly as possible.

Fig. 5.
figure 5

Gun taken away by an enemy in a VR environment.

Full size image

Gun Taken Away by an Enemy. In the other event, an enemy approached a player and tried to take the gun away from the player by pulling the gun from side to side (Fig. 5). The player experienced a sensation of force through the use of the EMS and hanger reflex, which enabled the player’s wrist to move in reaction to the enemy’s action.

3 User Study

To investigate what kind of an affect the combined haptic feedback had on the VR experience, we conducted a within-subjects laboratory experiment using two different conditions of VR experience:

  1. (1)

    VR game with the haptic feedback using our device.

  2. (2)

    VR game without the feedback.

3.1 Participants

We recruited thirteen participants (2 female) aged between 18 and 26 years from the institution. The group’s mean age was 21.23 years (SD = 2.72). Five participants had experienced hanger reflex before and ten had experienced EMS before.

3.2 Procedure

On arriving at the experimental location, the participants were asked to complete a consent form and answer survey questions about their age and any previous experience of EMS and hanger reflex. All the participants experienced the VR application under both of the two conditions. Seven of the participants played the shooting game using the proposed device first and then without using the device. The other six of the participants played without the device first and then using the device. For calibration, for EMS and the vibration for hanger reflex were adjusted according to individual differences. After each condition was experienced, the participants were answer a Likert-scale questionnaire regarding the enjoyment and realism. The participants also completed free-description questions in the questionnaire to explain the reasons for each score. Once they completed both conditions, they were asked which VR experience they preferred.

3.3 Result

Enjoyment. The graph in Fig. 6 on the left side shows the results for the question regarding whether the VR experience was enjoyable for the participant. In the question, the resultant rating for the VR experience with the proposed device (5.9) was higher than the VR experience with no haptic feedback (4.7) on an average.

Fig. 6.
figure 6

Rating the realism, enjoyment, and overall preference after the VR experiences both with the device (Haptic Device) and without the haptic device (Baseline).

Full size image

According to the participants’ free-descriptions, there were positive opinions with respect to the benefits of the force sensation that moved the participants’ wrists unconsciously.

“It was fun my hand was moving unintentionally.” – P1

“I thought it was realistic and I enjoyed the experience because I felt it was stimulating unlike the vibration of a general controller.” – P6

“I enjoyed the feedback more than the normal vibration because the stimulus was directly conveyed in comparison with the vibration of a controller.” – P6

The participant (P6) who had used a VR controller with vibration feedback before, described that it was more enjoyable with the haptic feedback of the proposed system than the vibration that a controller could provide.

Realism. The resultant rating of the VR experience with the haptic device (5.5) was higher than without it (4.2), with respect to the question as to whether the VR experience was realistic, as shown in Fig. 6.

Participants commented on the reasons for their belief that the haptic device improved the realism of the VR experience. Some participants implied that the force sensation synchronized with the events in the VR game helped them experience a greater sense of realism.

“When I had my gun taken away, I felt the realism because my hand was moved as in the video (played in the HMD).” – P4

“I realized the effect of the device especially in the second content (Gun taken away). I got a sense that my arm movement was synchronized with the application.” – P5

“Compared to the experience of playing without the device, the impact and vibration of shooting a gun was reproduced and I felt it was more realistic.” – P8

Preference. Participants were also asked which VR experience they preferred after they had completed both conditions. 84.61% of all the participants (11 out of 13) answered they preferred the proposed condition (see Fig. 6).

In a free-description question, participants were invited to comment on the reasons why they had indicated that they preferred the haptic feedback.

“I felt more immersion with the haptic feedback.” – P2

“I felt as if I had actually shot a gun.” – P8

The participant’s comments implied that the immersion in the VR scene was improved by using the device and that the haptic feedback from the device allowed them to feel as if they had used an actual gun.

On the other hand, there were two participants (P10, P12) who felt afraid because of the electrical stimulation of their muscles while playing the VR game.

“I was afraid of the electrical stimulation. Because I want to enjoy the game, I don’t want to experience fear.” – P10

4 Discussion

4.1 Limitations

Participants commented on the vibration of the transducers employed on the device for hanger reflex. These comments indicated that users may have perceived the vibration rather than the force sensation of the hanger reflex itself. The current device utilized two transducers for inducing wrist rotation, which may apply strong vibration to a wrist. It might be possible to decrease the intensity of the vibration intensity but may cause less of a sensation of force.

The perception of force sensation using the EMS and hanger reflex differs depending on the condition of the individual’s muscle and skin. Using the EMS and hanger reflex technology can generate individual differences in the movement speed and rotation angle of the wrist. In this study, the EMS voltage was changed to balance the differences between individuals. However, there are other parameters that could be adjusted for calibration. Examples include the positions of the electrode pads, the frequency of the signals, and the pulse-width. These might need to be considered in the context of future work.

4.2 Future Work

In the user study conducted as part of this study, EMS and hanger reflex were utilized simultaneously in one of the events in the shooting game. However, they could mutually influence the effects of force sensation. In future work, it will be important to investigate how the EMS and hanger reflex affect each other when both methods are applied to a subject’s wrist at the same time.

It has been reported that the hanger reflex phenomenon is found not only associated with the wrist but also with other body parts such as the head, ankles [13], and even the waist [5, 14]. Moreover, EMS can be applied to the arms [8, 9] and fingers [6]. This study was focused mainly on utilizing the combined method on the wrist. If it was possible to apply the combined method to other body parts as well, there could be other possible interactions within the VR environment and users could experience more immersive VR contents. For example, while EMS is not suitable for giving haptic feedback on the head but is suitable for use on the arms, hanger reflex is suitable for feedback on the head but not on the arms. Installing hanger reflex equipment on the head and EMS equipment on the arms would enable a boxing application to give haptic feedback to both of the head and arms in contrast to a boxing simulation using only EMS [8]. Thus, the combination of the two methods allows future VR applications to expand in a variety of haptic interactions, which will lead to more engaging and realistic experiences in VR.

The combined control method is also applicable to other fields outside of VR such as medical rehabilitation and display of instructions. In the field of medical rehabilitation, a therapist could utilize the wrist control method to reactivate a patient’s wrist that have been affected during surgery and start to trigger the muscle memory. The proposed method could also be applied as a technology for instructing such skills as specific tool use.

5 Conclusion

We propose a novel method that combines EMS and hanger reflex to provide haptic feedback to the wrist. The hanger reflex was used to elicit supination and pronation and the EMS was used to cause flexion and extension. We also implemented a VR application where users could receive haptic feedback synchronized with interactions within the VR environment. A user study was conducted using a VR application to investigate how the combined haptic feedback enhanced the VR experience. The results showed that the proposed haptic device allowed users to feel more realism in, and enjoyment of, the VR experience. They also described the VR experience while using the system as more immersive. In future work, the mutual effects of EMS and hanger reflex need to be investigated in further detail. It would also be interesting to use EMS and hanger reflex in other locations on the body for various other types of haptic interactions. It is expected that the proposed method will be useful and will aid in designing new haptic interactions that enrich the VR experience. Though its utility in VR has been demonstrated, the proposed system may be adapted for other uses in a number of ways.