Keywords

1 Introduction

The sense of touch is our irreplaceable channel for daily communications. It provides not only well-recognized discriminative information to the human brain, but also an instinct and abundant affective input. Different types of social touch like shaking hands among friends, embracing the lover and the mother’s caress convey different emotions. Touch is featured with abundant emotions which can largely affect users’ emotion and experience.

With the development of science and technology, telecommunications have been upgraded from traditional mails to real-time phone calls, and even high-quality video face times. Meanwhile, it is expected to convey more affective information during communications. Therefore, mediated social touch is becoming a promising research field in recent years [1, 2]. Apart from the normal video and voice, the combination of the sense of touch offers a new level of multimodal immersive experience.

Past researches outline the multimodal interactions and a set of affective research guidelines [3], but limited work focuses on affective haptics in a multimodal scenario. In our daily life, most user rely on visual or audio channel as main communication methods, while using haptics to assist. Therefore, we need explore communicating emotion by haptics in a multimodal scenario.

It has been widely demonstrated that a gentle slow stroking signal brings pleasant feeling [4, 5], while a high intensity vibrating signal leads to unpleasant and aroused feeling. These findings, however, are proved only in a haptic only environment. In this paper, the present work explores the above affective haptic findings in a visual-haptic multimodal environment. In addition, we investigate whether there is any low intensity haptic stimulus that causes unpleasant or aroused feeling. In the rest of this paper, we present the background for our approach followed by methods and results from two experiments. More detailed findings are discussed, and we conclude the paper with potentials of haptics in a multimodal scenario.

2 Related Works

2.1 Affect Subjective Evaluation

People has complicated emotional activities. In the field of affective haptics, dimension evaluation is often used. Russell introduced “A Circumplex Modal of Affect” [6] in 1980, which put all human emotions into two dimensional plane rectangular coordinate system by “valance” and “arousal”. As Fig. 1 shows.

Fig. 1.
figure 1

A Circumplex Modal of Affect [6], redraw by authors

Full size image

In this emotional model, X-axis stands for valence and Y-axis stands for arousal. both dimensions are linear. If the value is bigger, it will be farther away from the center point and reach deeper emotion. In contrast, if it is closer to the center point, it will be emotionally lower. The zero-point stands for human normal status. The four points between two coordinate axes in the Fig. 1 (Excitement, Contentment, Depression and Distress) are the common effect of valence-arousal. More detailed 15 emotions mapping see [7]. Thus, human emotion can reflect on a point in the valence-arousal space.

Circumplex model of affect is a concise, direct and effective emotional evaluation method. Compared to diversified classification of emotions, this model uses orthogonality of valence-arousal to establish emotional space. Besides, in terms of this model, some researchers have been trying to extent more dimensions apart from valence and arousal, including the dimension of “dominance” [8]. The experiment, however, showed that the dominance degree shows a certain correlation with the other dimensions, which is not supposed to be a third independent dimension. Therefore, most current emotional evaluation is mainly using valence-arousal method. Based on this evaluation method, any external signals can affect people and map the emotional state to a point in valence-arousal plane so as to establish a connection of physical signals and human emotional feedback.

Followed by this evaluation, the International Affective Picture System (IAPS) was designed and implemented by Lang et al., 2008 [9]. It is a set of emotionally evocative color photographs representing a wide variety of events and objects encountered in human experience that have been rated on dimensions of pleasure, arousal, and dominance by men and women. IAPS stimuli are widely used in studies of human emotion throughout the world [9]. For each emotion dimension, the score from 1 to 9 is defined as: 1-least and 9-most. In our study, we use IAPS as visual stimuli to conduct a visual-haptic multimodal scenario.

2.2 Affective Haptics

Many affective haptic devices and their experiments have demonstrated a strong link between human emotions and the sense of touch. Among those, “pleasant touch” and “aroused touch” are two main findings.

Line S. Loken revealed “pleasant touch sensations” in Nature Neuroscience, 2009, that during soft brush stroking, low-threshold unmyelinated mechanoreceptors(C-tactile) responded most vigorously, which were perceived by subjects as being the most pleasant [4]. It can be inferred that C-tactile afferents show positive correlation with pleasant sensation. Furthermore, C-tactile experiment discussed the relations between caressing speed and the degree of pleasant sensation. When the stroking speed of haptic signal received on the hairy skin is between 1–10 cm/s and the pressure force is low, the pleasant sensation feels strongest.

In contrast to pleasant touch, “aroused touch” aims to draw user’s attention by strong, aroused touch sensations. It is usually created by a high intensity haptic signal, and it brings aroused, and unpleasant emotions.

With these neuroscience findings, many affective haptic devices have been designed and developed. They can create gentle, soft stroking movement on the skin, aiming to bring pleasant touch sensation to the human. In this field, Biamchi et al. explores a fabric-based softness display [10] and a novel fabric-based tactile display [11]. Both displays create “caress-like” haptic signals, make reciprocating movement on the skin and generate the affective sensation. They shared the results in common: the increase of tension led a higher pressure and the subject valence would decrease.

The above researches demonstrated the factors of affective touch in a touch only interaction basis. In this paper, we set up experiments in a visual-haptic scenario, trying to explore the affective haptics in a multimodal interaction basis.

3 Visual-Haptic Affective Study

In this section, two experiments were conducted in which participants experienced haptic vibrotactile stimuli as well as visual affective pictures. In the first experiment, participants experienced 7 vibrotactile signals including random, vibration and stroking sensations. In the second experiment, Half of the stimuli were conducted in a multimodal way in which both a haptic vibrotactile signal and a visual picture were presented simultaneously, the other half was by pictures only. Their subjective feedbacks by valence and arousal reveal that haptic stimuli were able to affect the participants’ emotion.

3.1 Hypotheses

Based on the above literature review, we try to investigate the affective haptics in a multimodal interaction basis. Firstly, we believe haptic stimuli affect participants’ emotion (hypotheses 1). Then we designed three types of haptic stimuli: random signal buzz signal and stroking signal, with several stimuli of different intensity in each type of signal. We infer: random signals convey unpleasant feeling (hypotheses 2), vibration signals convey aroused feeling (hypotheses 3), and stroking signals convey pleasant feeling (hypotheses 4). Last but not least, haptic affective signals affect similar emotion in both the haptic only and visual-haptic conditions (hypotheses 5).

  • H1: Haptic stimuli affect participants’ emotion.

  • H2: Random signals convey unpleasant feeling.

  • H3: Vibration signals convey aroused feeling.

  • H4: Stroking signals convey pleasant feeling.

  • H5: Haptic affective signals affect similar emotion in both the haptic only and visual-haptic multimodal scenarios.

3.2 Apparatus

The experimental apparatus consisted of a 2-by-9 tactors array worn on the dorsal side of the participants’ non-dominant forearm. The 18 tactors form 2 rows, with 9 tactors on each row that were evenly distributed along the direction of elbow to wrist. The center-to-center distance between each tactor was around 2.5 cm. A gauntlet was employed to keep all 18 tactors attached to the dorsal part of the forearm in the appropriate position. A wide-bandwidth tactor (Tectonic Elements, Model TEAX13C02-8/RH, Part #297-214, sourced from Parts Express International, Inc.) was employed as the actuator. It has a flat frequency response in the range 50 Hz to 2 kHz with a resonant peak close to 600 Hz. A MOTU 24Ao audio device (MOTU, Cambridge, MA, USA) was used for delivering 18 channels of audio waveforms to the 18 tactors through audio amplifiers. A Matlab program running on a desktop computer was made to generate the multi-channel waveforms corresponding to the haptic vibrotactile signals.

Meanwhile, the program interface (shown in Fig. 2) could show a picture from IAPS, as well as two scoring bars in terms of valence and arousal.

Fig. 2.
figure 2

Program interface of the multimodal experiments

Full size image

3.3 Haptic and Visual Stimuli

Haptic Stimuli

According to the hypotheses, 7 haptic stimuli were divided into three groups (see Table 1). In group I, unpleasant feeling was sometimes related with “chaos” or “randomness”, thus, we designed stimulus by randomizing location, frequency, duration and intensity levels. The 18 tactors activated one by one in a randomized order. The vibration frequency of each tactor was between 100–300 Hz. The vibration duration of each tactor was between 50–800 ms, and the duration of overall stimulus was 4 s. The vibration intensity of each tactor was between 16–25 dB above Sensation Level, short with dB SL, (Stimulus #1) and 26–35 dB SL (Stimulus #2). By doing this, we tried to simulate the chaotic and disordered feeling by haptics. Group II contains two vibratory signals. All tactors were activated at the same time with a 300 Hz vibration frequency and lasted for 500 ms. The vibration intensity was 20 dB SL (Stimulus #3) and 30 dB SL (Stimulus #4). Group III consisted of the pleasant stroking signals. Three stroking signals were designed using the tactile apparent motion theory [12], which could be applied to control the velocity and vibration intensity. For the 2 rows of tactors, the first column of 2 tactors near the wrist were activated firstly, followed by the second column of 2 tactors from the wrist, and so on. The vibration traveled from the wrist to the elbow, and the signals stopped after the ninth column of 2 tactors vibrated near the elbow. The vibration duration of each tactor was 600 ms and the SOA (Stimulus Onset Asynchrony) was 440 ms. Thus, the overall duration was 440 × 8 + 600 = 4120 ms, and the stroking velocity was around 5 cm/s. The vibration frequency was 100 Hz for the 3 signals in Group III, and the vibration intensity was 20 dB SL (Stimulus #5), 28 dB SL (Stimulus #6) and 36 dB SL (Stimulus #7).

Table 1. Summary of 7 haptic signals
Full size table

Visual Stimuli

Seventy pictures with affective ratings from the IAPS database were selected, forming 35 pairs. Each pair of pictures shared highly similar contents and affective scores in terms of valence and arousal according to the database. For example, 1040.jpg and 1114.jpg are both pictures about snake. Their valence scores are 3.99 and 4.03, and arousal scores are 6.25 and 6.33, which indicated that both pictures were unpleasant and aroused. Therefore, this pair of pictures was selected. Thirty-five pairs of affective pictures covered a large range of score from very unpleasant (1.90) to very pleasant (8.00), and very deactivated (2.67) to very aroused (7.29).

3.4 Participants

A total of 14 participants (7 females; age range 18–35 years old, 23.4 ± 5.2 years old) took part in the studies. All participants were right-handed with no known sensory or motor impairments. Subjects gave informed consent before testing, and the study was approved by a university internal review.

3.5 Procedure

The participant was seated in front of a table on which there was the tactile device, a computer together with a screen and a mouse. The participant wore the gauntlet on the left forearm to experience the tactile simulation. Meanwhile, the participant used the right hand to interact with the graphical interface by the mouse. The participant was instructed to maintain the same arm position and wore a noise-reduction earphone to prevent any sound to bias tactile perception during the whole experiment.

Experiment I

The participant experienced 7 haptic stimuli one by one without visual stimulus. Then based on the haptic only stimulus, participant was asked to use the VAS with 9 points bipolar Likert scale to log the response. The participant could try out the haptic signal as many times as he/she needs and change the VAS scale. After rating all 7 stimuli, the first experiment finished.

Experiment II

After experiencing 7 haptic stimuli, in experiment II, the participant turned to do the affective rating of the visual-haptic multimodal conditions. The experiment interface included two parts: one of the 70 pictures from IAPS, and the VAS with 9 points bipolar Likert scale as well as the SAM picture for the reference.

In terms of the pair between the pictures and haptic signals, the visual-haptic correlation needed decrease to minimum. In the first step, between each pair of pictures sharing similar emotional ratings, one was selected and it came with one of the seven haptic stimuli, and the other one was presented alone. Therefore, thirty-five pictures were selected to present with haptic stimulus, and the other 35 were presented alone. We also needed switch the sequence of each pair of pictures, making the first 35 pictures without haptic stimulus and the other pictures in visual-haptic condition. Thus, there are 2 alternatives for the selection of half of the 70 pictures. In the next step, 35 pairs of pictures were divided into 7 groups, with 5 pairs in each group. Each group was accompanied with one type of haptic stimulus. 7 haptic stimuli allowed for the 7 alternatives for each group to pair any different haptic stimuli. By these 2 steps, there were 2 × 7 = 14 alternatives for the pair of visual and haptic stimulus, and each alternative was corresponded with one participant. Lastly, the sequence these stimuli were randomized before presented to the participant.

After a visual-only stimulus or a multimodal visual-haptic stimulus, the participant was asked to use the VAS with 9 points bipolar Likert scale to log the response about the overall emotional state.

There were 70 stimuli in all. The total duration of the study was approximately 30 min for each participant.

4 Results

4.1 Experiment I: Haptic Stimuli Results

In terms of statistical analysis, One Sample T test was used and valence and arousal scores were compared with 5, meaning neutral state. A p value < 0.05 was also considered statistically significant.

In terms of the haptic only condition, for each haptic signal, the mean and stand deviation of the valence and arousal scores are presented in Fig. 3. One sample T test shows the statistical significance in arousal of Stimulus No. 1, valence of Stimulus No. 6, and valence and arousal of Stimulus No. 2, 4 and 7.

Fig. 3.
figure 3

Mean and stand deviation of the valence and arousal scores in haptic only condition

Full size image

4.2 Visual-Haptic Multimodal Results

During the multimodal experiment, 70 valence and arousal scores were collected. Firstly, they were grouped by 35 pairs of pictures, one with haptic stimulus and one without. Secondly, in order to extract the effect of affective haptics, we calculated the subtraction between the V-A scores with multimodal stimulus and the scores with visual only stimulus. Therefore, the plus number of the subtraction meant that the visual-haptic stimulus feels more pleasant or more aroused, while the negative number meant that the visual-haptic stimulus feels more unpleasant or more deactivated. For example, between a pair of pictures, the multimodal one had scores (6, 3) for valence and arousal, and the visual one had scores (4, 4). Thus, the subtraction (2, −1) meant that the employment of haptic signal tended to be more pleasant (degree of 2) and less aroused (degree of −1). Thirdly, each type of haptic signal was assigned to 5 pairs of stimuli, so there were 5 subtraction results for every haptic stimulus for each participant. Lastly, all the results of 14 participants were grouped by each haptic stimulus, and there were 14 × 5 = 70 subtraction results for every type of haptic signal.

In terms of statistical analysis, the One Sample T test was used. Theoretically, if the employment of haptic signal was not able to affect the participants’ emotion, then the V-A scores of multimodal and visual only stimuli would be identical, so the subtraction results would be (0, 0). Therefore, we compared the 70 subtraction results with the number “0” in terms of valence and arousal for every type of haptic signal, and tried to investigate if the subtraction variance was significantly different with zero. A p value < 0.05 was considered statistically significant. Likewise, in the haptic only experiment, One Sample T test was used and valence and arousal scores were compared with 5. A p value < 0.05 was also considered statistically significant.

Random Signals Results

The subtraction mean and standard deviation ratings of valence and arousal for signal No. 1 and 2 are presented in Fig. 4. One sample T test shows the statistical significance in arousal of stimulus No. 1 and No. 2, It proves hypothesis 1 that the employment of haptic signal No. 1 and No. 2 is able to affect participants’ emotion in arousal. The visual-haptic multimodal stimuli are more aroused than visual only pictures. The valence scores, however, do not have a statistically significant factor. With the increase of signal intensity, the valence scores tend to decrease. Although the valence of stimulus No. 2 is negative, it is the strong intensity that leads to the unpleasant feeling, showing that “disordered” signal pattern is not able to convey unpleasant feeling. Hypothesis 2 is not validated.

Fig. 4.
figure 4

Mean and stand deviation of the valence and arousal scores in multimodal condition, for random signals

Full size image

Vibration Signals Results

The result scores of valence and arousal for stimulus No. 3 and 4 are illustrated in Fig. 5. One sample T test shows the statistical significance in arousal of stimulus No. 4. The valence scores of stimulus No. 4 are also below zero. In terms of stimulus No. 3, the light quick buzz signal does not affect the emotion a lot in both valence and arousal. It demonstrates that only the haptic signal intensity is strong enough that can convey aroused feeling. It is not corresponding with “vibration” pattern of signal. Hypothesis 3 is validated. Also, the strong arousing signal tend to feel unpleasant.

Fig. 5.
figure 5

Mean and stand deviation of the valence and arousal scores in multimodal condition, for vibration signals

Full size image

Stroking Signals Results

The mean and standard deviation of ratings of valence and arousal for signal No. 5, 6 and 7 are shown in Fig. 6. One sample T test shows that the valence and arousal of stimulus No. 6 are both statistically significant, so as the arousal of stimulus No. 7. As the signal intensity increases from No. 5, No. 6 to No. 7, the arousal increases from 0.14, 0.75 to 1.58. Meanwhile, the valence scores show a peak in stimulus No. 6, with the significant mean value of 0.53, while the other two scores are around 0. Stimulus No. 6 is the only signal that is statistically significant in valence among 7 stimuli. In terms of stimulus No. 7, the strong intensity spoils the pleasant stroking and leads to a negative valence. The “stroking” touch is able to convey pleasant feeling, and hypothesis 4 is validated.

Fig. 6.
figure 6

Mean and stand deviation of the valence and arousal scores in multimodal condition, for stroking signals

Full size image

Table 2 compares the mean scores between haptic only and multimodal results. If we assume that 5 is the default affective score, then the subtraction of 5 shows the haptic affective effect (4th column in Table 2). Then we compare with multimodal mean scores, and both experiment results follow the similar trend. Hypothesis 5 is validated. Moreover, some haptic only scores are more intense (further from 5) than multimodal mean scores. It indicates that in haptic only environment, participants’ emotion is more easily affected by haptic signals than that in multimodal scenarios.

Table 2. Comparison of haptic only mean and multimodal mean scores
Full size table

5 Discussions

The above study explored the affective haptics in the visual-haptic multimodal scenario. Quantitative analysis of both the haptic only and visual-haptic conditions are made. There are similarities and differences of the results between these two conditions. Firstly, the sense of touch affects participants’ emotion. In the haptic only condition, the scores of valence and arousal change with different haptic signals. In the multimodal condition, the scores in most visual only and visual-haptic pairs differ in many cases. Therefore hypotheses 1 is validated. Moreover, if we compare the emotion scores in haptic only and visual-haptic condition, haptic affective signals affect similar emotion. Hypotheses 5 is also validated.

In terms of random signal, hypotheses 2 is not validated. It is not easy to convey unpleasant feeling by haptics with low or medium vibration intensity, neither in haptic only nor in multimodal condition. Vibration signal conveys aroused sensation in both conditions, and hypotheses 3 is validated. Lastly, stroking signal conveys pleasant and relatively unaroused in moth conditions, and hypotheses 4 is validated.

If we compare the emotional impacts from visual stimuli and haptic stimuli, it can be observed that visual stimuli largely dominate the emotional state. There is sometimes limited difference of emotional state with same visual picture and different haptic signals, but there are mostly huge changes of emotional state with same haptic signal and different visual pictures. It demonstrates the assistive role of multimodal interactions in terms of haptics.

6 Concluding Remarks

This paper explores the affective haptics and multimodal experiments. By the literature review of human emotional feedback on haptic signal and affective haptic devices, the multimodal experiments were designed and implemented. Three types of haptic signals (random, vibration and stroking stimuli) are designed and implemented. The results demonstrated that haptic affective signals affect similar emotion in both the haptic only and visual-haptic multimodal scenarios. Moreover, visual stimuli largely dominate the emotional state compared with haptic stimuli. Based on the physiological characteristics of human multimodal channels and emotional feedback, designers need to consider involving haptic signals as assistive role, and satisfy the user’s emotional needs into the multimodal interaction scenario. Meanwhile, designers also need to refer to the state of the art of affective haptic devices, focus to the detailed interaction scenarios and real needs of the users to offer better affective experience.