Keywords

1 Introduction

As technology is increasingly a medium with which humans interact, the interface between the person and the machine is an increasingly salient interaction. Thus, it is possible to design technology with the intention of blurring this “boundary” (whether physically or conceptually) to augment user experience. The interface in human computer interaction (HCI) is commonly a computer screen, and in this study specifically featured a person—notable because other human beings and their bodily movements, which can be mapped onto our own bodies, are a distinctive and richly meaningful class of stimuli [1].

Given growing evidence that cognitive representations such as sense of self are not amodal but connected with our bodies, this experiment tested manipulations of the sensorimotor domain within a socially-situated HCI. Such research that advances theories of embodied and situated cognition (in which cognition is seen as taking place not only in the brain, but also in interaction with the world supported by the body) will be central for developments in HCI as well as in related fields such as artificial intelligence [2].

1.1 Literature

Our body representation and sense of self is constantly updated starting from the integration of different sensory inputs [3, 4]. Sense of self begins with the body, but self is also one’s psychological being (e.g., thoughts, feelings, attitudes), which in turn exists within a social matrix [5]. Conceptually, the self can be socially extended to other persons [6]. Relationships with loved ones are often described as a blurring of self-other boundaries; such merging can occur at both the conceptual and bodily level [7]. We use metaphors like “we are one” to convey our subjective experience of self in physical terms [8]. Our conceptual sense of self, then, is bound to our physicality and influenced by the sensorimotor domain, including bodily feedback [8, 9]. While one’s body/self seems distinct from the world, this boundary is artificial [10]–an actuality that can be employed in user interface and interaction design.

Synchronous bodily movement has been used to manipulate conceptual self-boundaries. For example, in their study of synchrony and cooperation, [11] had participants walk in step in the experimental condition and walk normally in the control condition. They then asked the question: “How connected did you feel with the other participants during the walk?” Participants in the synchronous condition reported feeling more connected with their counterparts than those in the asynchronous condition.

Similarly, synchronous bodily sensation has been used to manipulate conceptual self-boundaries. Reference [7] brushed the cheek of study participants while participants watched the brushing applied to the face of a stranger shown in a video, in synchronous versus asynchronous conditions. The degree of self-other merging was determined by measuring participants’ body sensations, their perception of face resemblance, their judgment of the inner state of the other, the closeness felt toward the other, and conformity behavior. The results showed that synchronous multisensory stimulation blurred self-other boundaries; participants exposed to synchronous stimulation showed more merging of self and other than participants exposed to asynchronous stimulation. The study concluded that multisensory integration can affect social perception and create a sense of self-other similarity [7]. Similar experiments have yielded this enfacement effect in the synchronous condition, which creates in the participant the sensations of being in front of a mirror [4].

The current study was based on such evidence that the ways we conceptualize, reason about, or visualize interpersonal experiences are influenced by the sensorimotor domain [8, 12]. The objective of this study was to take the metaphoric “overlap” that may arise between self and other and test it experimentally, and the significance is that the results can be applied to HCI. The evolution of technology toward adaptive systems that are capable of inferring emotion and incorporating multimodal interactions demands such research to provide a deeper understanding of the interplay of body and self, of emotion and social cognition.

1.2 Independent Variables and Hypotheses

The current experiment varied the conditions of the face-brushing experiment [7] by presenting versus not presenting a synchronous haptic experience (i.e., neck massager) between the participant and an actor (rather than synchronous versus asynchronous stimulation). The current study also included verbal content from the actor, which varied between two levels—energy and calm. For example, in the energizing video, the actor said the neck massager acted like coffee in making her energized, and in the calming video she said it acted like herbal tea to make her calm.

Hypothesis 1 predicted that the energy video would have an energizing effect and the calm video would have a calming effect in both the haptic and no haptic conditions. Hypothesis 1 further predicted that in the haptic conditions, these effects would be greater due to the synchronous haptic experience with the actor. In other words, the neck massager was intended to heighten similarity and familiarity, and was expected to aid in blurring the conceptual boundary between participant and actor in the haptic conditions. The synchronous haptic experience was hypothesized to convey “sameness” in both the tactile modality as well as the visual modality (in which it may be conceived as a shared morphology [13]), thus aiding in the transmission of the actor’s mood via an emotional contagion [14].

Likewise, it was anticipated that the presence of the haptic device (the synchronous haptic experience) would result in participants perceiving a closer connection to the actor. Hypothesis 2 then predicted that participants in both the haptic-energy video and haptic-calm video conditions (as compared to those in the two no haptic conditions) would realize a pre-post gain in interpersonal closeness with the actor.

2 Methods

The study was a three-factor, between and within participants experiment. With regard to the two physiological measures, the study was a 2 (Haptic or No Haptic) × 2 (Energy Video or Calm Video) × 3 (Pre, Post 1, and Post 2 repeated measures) factorial design. With regard to the three self-report measures, the study was a 2 (Haptic or No Haptic) × 2 (Energy Video or Calm Video) × 2 (Pre and Post repeated measures) factorial design.

2.1 Dependent Variables

Based on the two hypotheses, two categories of dependent variables were of interest. One was the mood change (i.e., blurring of self-other boundary) from pre-video to post-video, which was expected to be affected by the actor’s mood—more so in the haptic condition. Mood was measured by changes in heart rate and skin conductance, and subscales of [15] ’s short mood scale. Reference [15] developed and evaluated the psychometric properties of this six-item scale based on the Multidimensional Mood Scale [16]. Two of the scale’s three factors, Calmness and Energetic Arousal, were used in this study, as measured by four items. Reference [15] found reliability to be 0.90 for both Calmness and Energetic Arousal.

The second variable of interest was the self-other boundary as manifested in the participant’s perception of interpersonal closeness to the actor. Participants responded to [7] ’s version of [6] ’s Inclusion of the Other in the Self Scale (IOS) scale (originally developed by [17]). The 7-point scale consists of a series of circles representing the self and the other person with different degrees of overlap. Reference [17] ’s reliability check (test-retest) showed correlations between the original and the 2-week retesting as r = 0.83 overall and other measures supported the concurrent validity of the IOS Scale. Also to assess the self-other boundary, participants responded to the questions “How connected did you feel with the actor?”, “How similar is the actor to yourself?”, “How close do you feel to the actor,” and “How much do you like the actor?”, using a 9-point Likert scale. These questions were derived from [11] and [18], for which reliability was not reported. For this study, the interpersonal closeness measure (the sum of these four questions plus the IOS Scale) had a Cronbach’s alpha of 0.87.

2.2 Apparatuses and Actor

The device used for haptic input was the Brookstone Shiatsu Neck and Shoulder Massager. Heart rate and skin conductance were measured with the FlexComp Infiniti Encoder. Video of the actor (an 18-year-old female college freshman) was recorded digitally on a Flip Video Camera and was viewed by participants on a 15-inch MacBook.

2.3 Participants

Healthy females between the ages of 18 and 30 were recruited via advertisements posted on campus and email invitations. Gender and age could be confounding factors, and therefore the study sought a somewhat homogenous sample of participants who resembled the actor in the video. Each participant read and signed consent documentation approved by the Institutional Review Board and received a $10 Caribou Coffee gift card for her participation.

Forty-eight women participated in the study. They ranged in age from 18 to 29, with a mean age of 21.29 (SD = 2.68). All were full-time students. About two-thirds (33) identified their race as White, whereas nine were Asian, two were Black or African American, and four offered multiple choices or left the question blank. About 90 % (44) were not Hispanic or Latino, whereas three were Hispanic or Latino, and one left the question blank.

Each of the 48 participants were randomly assigned to one of the four conditions, so that there were 12 participants in each condition. Related studies have used 10 to 18 participants per condition [4, 7, 11].

2.4 Procedure

A PC laptop contained the software (Biograph Infiniti Version 5.0.3 by Thought Technology Ltd.) used for the recording of physiological data. The accompanying hardware was FlexComp by Thought Technology Ltd., and consisted of the encoder, the SC-Flex/Pro sensor (for skin conductance), and the BVP-Flex/Pro sensor (for heart rate).

The investigator attached the skin conductance and BVP sensors to the non-dominant hand. Once the investigator began recording on the Biograph Infiniti, the participant responded to a brief demographic questionnaire and the Short Mood Scale. The investigator showed a still of the actor wearing the neck massager on the MacBook, and the participant responded to the interpersonal closeness measure. A rest period began, which took a total of 15 min from the start of the Biograph recording. The participant was offered a word search booklet to work on—a neutral activity to pass time.

About one minute prior to the start of the video (the 14-min. mark), the investigator placed the neck massager on the participant if she was assigned to the haptic condition. At the 15-min. mark, the investigator began the video on the MacBook—either calming or energizing, depending on the assignment. If the participant was in the neck massager condition, the investigator turned on the neck massager immediately after starting the video.

At the conclusion of the 3-min. video, the investigator turned off the neck massager (if applicable) and asked the participant to again respond to the Short Mood Scale. After showing the still of the actor wearing the neck massager on the MacBook, the participant again responded to the interpersonal closeness measure. The participant was then offered the word search booklet for the few remaining minutes of physiological data recording.

The period of physiological pre-data was defined as the five minutes prior to the start of the video. Two five-min. post-data periods were defined. Post 1 started one minute after the start of the video and Post 2 started at the conclusion of the video.

3 Results

A 2 × 2 × 3 analysis of variance for the physiological dependent measures and a 2 × 2 × 2 analysis of variance for the self-report dependent measures were computed, with repeated measures on the last factor. There was a significant main effect for time for all measures (Table 1), but no significant main effects for either haptic or video. Additionally, there were no significant effects for either Haptic × Time or Video × Time.

Table 1. Changes of measures over time
Full size table

A three-way interaction effect was found for two measures. First, the calmness measure produced a three-way interaction effect, F(1, 44) = 7.21, p = < .01. A paired samples t-test revealed that one condition showed a significant pre-post increase: haptic-energy video (p = < .05). Second, there was a three-way interaction effect for interpersonal closeness, F(1, 44) = 6.62, p = < .05. A paired samples t-test revealed that two conditions showed a significant pre-post increase in interpersonal closeness: haptic-energy video and no haptic-calm video (p = < .05).

Therefore, Hypothesis 1 was not supported, but Hypothesis 2 was partially supported in that the haptic-energy video condition realized a significant pre-post gain in interpersonal closeness (14.33 (SD = 5.37) to 23.00 (SD = 11.38)) (Fig. 1).

Fig. 1.
figure 1

Time × Haptic × Video interaction for interpersonal closeness measure

Full size image

In summary, the results showed a main effect for time for all dependent variables: heart rate decreased, skin conductance increased, self-reported calmness increased, self-reported energetic arousal decreased, and self-reported interpersonal closeness to the actor increased. A three-way interaction effect was evident for the measures of calmness and interpersonal closeness. With regard to the primary hypotheses, there was a significant pre-post gain in interpersonal closeness for those participants who both shared the haptic experience and saw the video in which the actor conveyed energy. Interestingly, in this condition there was also a significant increase in participants’ reporting a feeling of calmness.

4 Discussion

4.1 Heart Rate and Skin Conductance

Across conditions, there was a significant decrease in heart rate, which is in contrast to the hypothesis that heart rate would increase in the energy video conditions and decrease in the calm video conditions. Heart rate psychophysiology literature helps elucidate why heart rate decreased even in the presence of an actor showing energy.

A slowing of heart rate generally accompanies presentation of a novel stimulus [19]. Heart rate deceleration has been found to involve the mental intake of environmental stimuli or to accompany situations in which subjects “take in” perceptual materials; decreased heart rate has also been recorded during performance of a task as associated with increased sensitivity to stimulation [20].

In general, heart rate slows during intensive attention to stimuli [21]. However, if the external stimulus is a source of psychological stress, anxiety, or fear, heart rate will instead increase [21]. There is good support for the hypothesis that heart rate deceleration is associated with the orienting response and stimulus intake, and heart rate acceleration is associated with the defensive reaction and stimulus rejection [21]. In the present study, this hypothesis also appeared to be supported; neither of the videos, nor the haptic device, would generally be a cause of stress, anxiety, or fear, and therefore the decrease in heart rate can be interpreted as an orienting response and stimulus intake, and perhaps an indication of interest.

Across conditions, there was a significant increase in mean skin conductance, which is in contrast to the hypothesis that skin conductance would only increase in the energy video conditions and only decrease in the calm video conditions. The investigator’s prediction that heart rate and skin conductance would rise and fall together was based on postulates of activation theory (e.g., [22]). However, criticisms of activation theory can be found in the principal of stimulus-response specificity, which states that specific stimulus contexts “…bring about certain patterns of responding, not just an increase or decrease in an unidimensional activation continuum… By definition, a stimulus-response specificity exists if a stimulus brings about a similar pattern of physiological responding among most subjects” [23, pp. 54, 65].

Directional fractionation is a special case of stimulus-response specificity which occurs when response directions are not uniform [24]. Therefore, for example, a person who is attentive to a potential threat will experience a decrease in heart rate and an increase in skin conductance. However, this particular physiological response does not necessarily imply stress, but is in fact an orienting response, which enhances our sensory processing while directing our attention to novel stimuli [23].

Therefore, the physiological response in this experiment was likely an orientation response to the presentation of the video, and this response was not affected by the haptic or video content factors.

4.2 Calmness and Energetic Arousal

Across conditions, there was a significant increase in self-reported calmness and decrease in self-reported energetic arousal. Additionally, there was a three-way interaction effect found for the calmness measure, for which further analysis revealed a significant effect in only one condition, haptic-energy video. These results are in contrast to the hypothesis that the two calm video conditions would increase calmness and that the two energy video conditions would increase energetic arousal.

First, why would calmness increase in the haptic-energy video condition? It is unusual that this effect did not also (or only) present in the haptic-calm video condition, and its absence may be due to insufficient power. Ideally a “control” condition of haptic-no video would have been implemented, but that was not feasible at the time of the study. Certainly, such neck massagers are marketed as relaxing (e.g., [25]), and although there is a dearth of experimental studies regarding their effects, the user generally has an expectation of relaxation or calming.

Second, why would energetic arousal decrease across conditions? One, this effect could be an artifact of the experimental procedure (i.e., the quiet room, generally low levels of stimulation, and the word search task during the measurement of baseline physiological measures). Two, the orienting response (as identified by the increase in skin conductance and decrease in heart rate) could have impacted self-reported mood. Specifically, participants may have been more aware of their heart rate decelerating rather than their skin conductance increasing. Heart rate is to some extent perceptible and breathing—the physiological process that most directly influences heart rate—can be voluntarily controlled; but voluntary control (or even awareness) of skin conductance is less likely [21, 26, 27]. Three, the experiment’s video is similar to watching television, which “… appears to be a most potent means of providing relief from stress” [28, p. 107]. One reason for this is that television viewing likely disrupts “… rehearsal processes that would perpetuate states of elevated arousal associated with negative affective experiences” [28, p. 109]. Four, color that appeared in the video may have been a confounding factor. The actor wore a blue shirt and the background was blue-gray, and people tend to associate certain colors with certain emotional or arousal states. In two studies, [29] concluded that there was a systematic tendency for long-wavelength colors like red and orange to induce feelings of high arousal and for short-wavelength colors like violet and blue to induce feelings of low arousal. Similarly, [30] found that participants associated blue with the word group “calm, peaceful, serene.”

4.3 Interpersonal Closeness

Across conditions there was an increase in interpersonal closeness as well as a three-way interaction effect, which revealed that two conditions showed a significant pre-post increase: haptic-energy video and no haptic-calm video. It was hypothesized that both haptic conditions would result in an increase in interpersonal closeness. Therefore, this hypothesis was partially supported by the increase in the haptic-energy video condition, which showed the largest mean increase. Although the mean increase in the no haptic-calm video condition was smaller, it was still significant. One potential issue is that of confounding variables—in this case, that participants’ responses are likely in part influenced by the verbal content of the videos.

Another finding to examine is the increase in interpersonal closeness that was found in the haptic-energy video condition but not in the haptic-calm video condition. One possible reason for this is that energetic people may be perceived as more likeable, and likeability was part of the interpersonal closeness measure. For example, in a study of the personality traits of liked people, individuals who described themselves as “energetic/active” and “happy/joyful” were more liked in the sample of college freshman living in dormitories [31]. Certainly the actor in the energy video was more energetic and active than in the calm video, and it could also be argued that the energy video conveyed greater happiness and joy than the calm video.

Likewise, it is possible that the energy video engendered a more positive mood state than the calm video, thus resulting in a higher interpersonal closeness score. In a study of first-year college students, [32] hypothesized that positive emotions broaden people’s feelings of self–other overlap in the beginning of a new relationship. They found that with new roommates, positive emotions predicted increased self–other overlap, which was measured with the Inclusion of Other in Self Scale [17] that was incorporated into the interpersonal closeness measure in the current study.

Additionally, various studies have found that mood can have a profound effect on information processing and judgments, as well as mimicry behaviors. For example, a correlation has been observed between mood and the non–conscious mimicry of a person on television; the more positive an individual’s mood state, the more this individual mimicked the behavior of the person on the television [33].

5 Conclusion

The most notable finding of this study was the significant pre-post gain in interpersonal closeness in the haptic-energy video condition. The haptic effect found was as hypothesized, and possible reasons for the energetic condition interaction were suggested above. HCI applications that contain both haptic and interpersonal closeness factors would be wise to consider this effect and subject it to further testing.

With regard to the hypothesized (but unrealized) effect of the haptic factor blurring mood (i.e., increasing emotional contagion) between participant and actor, it is possible that the “experiential match” (the neck massager) between the actor and the participant was too subtle. The relevant literature emphasizes “action” (e.g., [1, 13]) and thus the passive haptic match between participant and actor (without movement) may not have been sufficient to invoke the hypothesized effects. On the other hand, mood was not a factor in the studies on synchronous stimulation and conceptual boundaries (e.g., [4, 7]) and therefore may not be susceptible to the manipulations performed in the present experiment.

This study was important in taking the metaphoric “overlap” between self and other and bringing it into the experimental realm. The study herein was based on theoretical evidence that this metaphor is grounded in the body (i.e., that sensorimotor inference informs concepts and social cognition) [8, 9, 12]. Additionally, these theories have been supported by studies that have demonstrated that simple body-based manipulations (e.g., of facial expression, posture, or movement) can causally influence the processing of emotional information [34] and that the cognitive phenomenon of self-other overlap can arise from a purely sensorial experience [7]. Future work must seek to understand the affective feedback loop between the user and the computer system; to design intelligent, adaptive systems, we need a deep comprehension not only of the computer system, but of the human system as well.

If an interaction designer seeks to increase users’ feelings of interpersonal closeness to an actor, avatar, or virtual agent, synchronous haptic experience may prove useful. Important factors for exploration in future research include levels of immersion and levels of interaction. It may be that even in a low-immersion, low-interaction experience such as television viewing, a shared haptic experience can improve user experience. An example would be to use a football player’s point of view for video while transmitting his heart rate to a user’s smartwatch via haptic output, to more fully immerse the user in the player’s experience.

In a similar example of HCI innovation, [35] described her students’ development of a breathing sensor for users that mirrors the behavior of the user’s character in the game. Synchronizing one’s breath with the avatar’s breath was necessary for game success, and [35] asserted that this type of experience is key to HCI design processes. “Without them we cannot create compelling and meaningful interactions in dialogue with our prospective users. But articulation of these experiences in academic texts is lacking—and sometimes very hard to capture… The dynamic gestalt of the interaction does not reveal itself to you until you experience it” [35, p. 10].

With regard to therapeutic applications, [36] summarized that touch is essential to our well-being, and that medical science is consequently developing therapies to incorporate haptics into the treatment of conditions such as autism spectrum, mood, anxiety and borderline disorders. They have prototyped a number of haptic devices that show promise in such treatments. While they did not explore the possibilities of pairing a haptic device with a computer mediation or video intervention (in a manner similar in the current study), that is certainly a possible avenue of investigation. If an increase in interpersonal closeness (or perhaps empathy) as well as calmness were goals of a given therapy, then synchronous haptic experience may be an appropriate piece of the interaction design of a novel HCI-based therapy.

Empirical findings such as those reported herein, coupled with advances in technology and user experience design, present new directions in blurring the boundary between the user and the interface. Such an embodied approach will likely prove valuable not only for refining human computer interaction, but for understanding the user as well.