Keywords

1 Introduction

Many methods that self-reported the affective states have been developed with questionnaires [1,2,3,4,5]. The methods with the questionnaires, however, have the common limitation that they cannot provide continuous affect rating with a real time resolution. Some continuous affect rating methods have been developed in order to overcome this limitation [6,7,8,9,10]. Some of them rated the affective states only on one particular dimension [6, 7, 10]. Russell [2, 11,12,13] demonstrated that affective states could be mapped in the Cartesian space from two bipolar dimensions of pleasure-displeasure and arousal-sleepiness. It meant that one dimensional rating methods could not rate general affective states but a specific affective state (positive-negative [6, 7], pleasure-displeasure [10]). On the other hand, Shubert [8] proposed the continuous affective rating method in two dimensional space of happiness-sadness and aroused-sleepy with multiple visual feedbacks. Nigel et al. [9] also proposed the continuous affective rating method in two dimensional space of positive-negative and arousing-calming with visual feedback. Both of the methods had the visual feedbacks to the participants about the cursor position when they manipulated the device such as the joystick or the mouse. These feedbacks might make it easy for the participants to manipulate the device to the cursor position they wanted and also keep the participants’ workload for manipulating the device low. These methods were applied to rating the affective states caused by temporally static stimuli or music. However, the visual feedbacks might interfere with watching the stimuli if the dynamic visual and auditory stimuli such as a movie was used. The continuous affect rating method in two dimensional space of pleasure-displeasure and arousal-sleepiness without visual feedback might be valuable for rating the affective states caused by the dynamic visual and auditory stimuli though it might be difficult for the participants to manipulate the device to the cursor position they wanted.

This research proposed the continuous affect rating method that rated the affective states by joystick in the Cartesian space from two dimensions of pleasure-displeasure and arousal-sleepiness without visual feedback and examined the effectiveness of the proposed method for rating the affective states caused by the dynamic visual and auditory stimuli through two experiments.

2 Proposed Continuous Affect Rating Method

In this session, we describe the construction of our proposed continuous affect rating method. The participants are required to manipulate the joystick in order to rate their affective states in the Cartesian space from two dimensions of pleasure-displeasure and arousal-sleepiness [2]. The joystick is placed at the origin of the pleasure-displeasure (x) axis and the arousal-sleepiness (y) axis written on the desk. In order to express the maximum pleasure state, for example, the participants are required to tilt the joystick to the right as possible and in order to express the maximum arousal state the participants are required to tilt the joystick forward as possible. If the participants release the joystick, it returns the neutral position. The original software records the joystick movements within the virtual square ranging from −100 to 100 in increments of 10 on each axis and the sampling rate is 100 ms. For example, the maximum pleasure state is recorded as 100 on x axis and the maximum arousal state is recorded as 100 on y axis. The neutral position of the joystick is recorded as (0, 0). The relationship between the orthogonal projection of the tilt angle of the joystick to each axis and the recorded coordinate of each axis in the virtual square is the linear relationship. When the participants manipulate the joystick, there is not any visual feedback except in the training mode. In the training mode, the coordinates box on the screen provides the numerical feedback about the coordinates (x, y) in the virtual square which the manipulated joystick expresses.

3 Experiment 1

The aim of the first experiment was to examine whether the coordinates obtained by the proposed method could reflect the changes of the affective states caused by the changes of the dynamic visual and auditory stimuli.

3.1 Method

Participants.

Twelve participants (mean age 21.8, SD 0.39, 6 females and 6 males) who were voluntary right-handed undergraduate students and were not provided any payment, took part in Experiment 1. They received a thorough explanation about the method of Experiment 1 and signed the consent form.

Apparatus.

The original software for the proposed continuous affect rating method was written in Visual Basic 2015 and was made operational on the windows 10. The software controlled the presentation of the dynamic visual and auditory stimuli, the transformation from the tilted angle of the joystick (Defender Joystick Cobra R4 USB 12 buttons) to the coordinates in the virtual square, and the records of the joystick movements. In the training mode, the software provided the coordinates box on the screen as a visual feedback.

An edited video was prepared as the dynamic visual and auditory stimuli. The edited video consisted of four video clips, “M1 Grand Prix”, “The Champ”, “Juon”, “Maldives Beach” in this order. “M1 Grand Prix” (“M1”) was the contest for comedy duos and was expected to make Japanese participants delight [14]. The clip was edited to 4 min 14 s. “The Champ” (“Champ”) was the famous American movie in 1979 and expected to make the participants sad [15]. The clip was edited to 2 min 52 s. “Juon” was Japanese horror movie in 2003 and was expected to make the participants afraid [16]. The clip was edited to 1 min 26 s. “Maldives Beach” (“Beach”) was the background video and was expected to make the participants satisfied [15]. The clip was edited to 40 s.

The training video was also prepared for the training mode. The video consisted of 100 sets of the coordinates (x, y) generated randomly between −100 and 100 in increments of 10. The interval between every two sets was 20 s.

Procedure.

At first the participants manipulated the joystick in the training mode in order to be able to manipulate the joystick at will. They were required to manipulate the joystick in order to make the values of the coordinates box correspond to the presented coordinates (x, y) as soon as possible and, after that, also required to make the joystick keep the same position till the next set of the coordinates was presented. After training, the participants were required to manipulate the joystick to correspond to their affective states which were caused by the dynamic visual and auditory stimuli.

3.2 Results and Discussions

Unfortunately, the data of one participant (male, 22 years old) could not be recorded correctly. The data of eleven participants were adopted as the results of this experiment. Figure 1 shows eleven participants’ averaged time series of the coordinates in the virtual square which the manipulated joystick expressed. The sampling rate of these time series was 1 s by obtaining the average data from every 10 data whose sampling rate was 100 ms.

Fig. 1.
figure 1

Averaged time series of the coordinates expressed by the joystick

Full size image

Both of the coordinates of pleasure-displeasure (x) axis and arousal-sleepiness (y) axis showed the positive intermediate value in “M1” and coordinates almost corresponded to the position of “DELIGHTED” in a circumplex model of affect [11, 12]. Both of the coordinates showed the negative intermediate value in “Champ” where coordinates almost corresponded to the position of “SAD”. The coordinate of x axis showed the negative intermediate value and the coordinate of y axis showed the positive intermediate value in “Juon”. The coordinates almost corresponded to the position of “AFRAID”. The coordinate of x axis showed the positive intermediate value and the coordinate of y axis showed the negative intermediate value in “Beach”. The coordinates almost corresponded to the position of “RELAXED”. These results revealed that the coordinates obtained by the proposed method changed according to the changes of the dynamic visual and auditory stimuli, and that the coordinates could reflect the changes of the affective states caused by the dynamic stimuli.

However, these results are only the trend by eleven participants’ average and it cannot be examined whether the obtained coordinates were consistent with the subjective response. Experiment 2 was held in order to examine statistically whether the proposed continuous affect rating method could rate the difference of the affective states between the different dynamic visual and auditory stimuli and whether the obtained coordinates were consistent with the response of the self-reports.

4 Experiment 2

4.1 Method

Participants.

Eleven participants (mean age 21.8, SD 0.40, 6 females and 5 males), who participated in Experiment 1 and whose data of Experiment 1 were adopted, participated in Experiment 2. They received a thorough explanation about the method of Experiment 2 and signed the consent form.

Apparatus.

The original software for the proposed continuous affect rating method, which was the same as for Experiment 1.

Four different four video clips were prepared as the different dynamic visual and auditory stimuli as follows; “M1”, “Champ”, “Juon”, and “Beach”. These video clips were the same as for Experiment 1.

Positive and Negative Affect Schedule (PANAS) Japanese version [17] was prepared as the self-reports of the current mood immediately after watching the clip. The PANAS Japanese version consists of two scales: Positive Affect (PA) and Negative Affect (NA), each with 8 items. The response format provided six alternatives from “1. Not at all” to “6. Extremely”. PA score and NA score were respectively obtained by adding the scores on each 8 items.

Procedure.

Experiment 2 consisted of four sessions composed of four different video clips. In each session, the participants were required to manipulate the joystick to correspond to their affective states which were caused by a video clip. After watching through the clip, the participants were required to describe their current mood with the PANAS Japanese version. The order of the sessions was the same for all the participants as follows: “M1”, “Champ”, “Juon”, and “Beach”.

4.2 Results and Discussions

Affective States Ratings by the PANAS.

The results of one way repeated measures ANOVA revealed that the averaged scores were significantly different between four different clips (PA score F(3,30) = 25.84, p < 0.01, NA score F(3,30) = 55.70, p < 0.01). The results of the multiple comparison by Bonferroni revealed that the averaged scores between any two clips has the significant difference in one at least of the PA and NA scores as shown in Table 1. These results of the responses of the self-reports of the current mood confirmed that the four different clips caused the different affective states immediately after watching the clips each other.

Table 1. The results of multiple comparison by Bonfferoni for the averaged PA and NA scores
Full size table

Affective States Ratings by the Proposed Method.

The results of one way repeated measures ANOVA revealed that the averaged coordinates were significantly different between four different clips (Pleasure-Displeasure F(3,30) = 17.86, p < 0.01, Arousal-Sleepiness F(3,30) = 23.15, p < 0.01). The results of the multiple comparison by Bonferroni revealed that the averaged coordinates between any two clips has the significant difference in one at least of the coordinates as shown in Table 2. From these results the proposed continuous affect rating method could rate the difference of the affective states between the four different clips confirmed by the results of the responses of the self-reports.

Table 2. The results of multiple comparison by Bonfferoni for the averaged coordinates
Full size table

Relationship of Affective States Ratings Between by PANAS and by the Proposed Method.

As mentioned above, the proposed continuous affect rating method could rate the difference of the affective states between the four different clips, which was confirmed by the results of the PANAS scores. However, the relationship between the coordinates obtained by the proposed method and the PANAS scores was not examined directly because the dimensional space, pleasure-displeasure and arousal-sleepiness, for rating by the proposed method was different from the dimensional space, positive affect and negative affect, for the PANAS scores. As the dimensional space labeled positive affect and negative affect could be produced by 45° rotating the dimensional space, pleasure-displeasure and arousal-sleepiness [18], the new averaged coordinates \( ({\text{PA}^{\prime},\text{NA}^{\prime}}) \) were produced by rotating the obtained averaged coordinates (x, y) of each clip and each participant by −45°. And the correlation coefficients between the new averaged coordinates and the PANAS scores of each clip and each participant were calculated.

Table 3 shows the results of the correlation analysis between the new averaged coordinates and PANAS scores. There were the significant correlations between PA and \( \text{PA}^{\prime} \) and between NA and \( \text{NA}^{\prime} \). It suggested that the coordinates obtained by the proposed method were relatively consistent with the response of the self-reports.

Table 3. The correlations between the new (averaged, peak) coordinates and the PANAS scores
Full size table

Though the correlation between PA and \( {\text{PA}^{\prime}} \) was significant, the correlation coefficient was not so high as shown in Table 3. Fredrickson and Kahneman [7] suggested that the responses of the self-reports after watching the clips correlated more highly to the peak of the continuous affect rating than to the average. So the peak coordinates \( ( {\text{x}}_{{\text{peak}}} ,\text{y}_{{\text{peak}}} ) \) of each clip and each participant were obtained and the new peak coordinates \( ( {\text{PA}}_{{\text{peak}}}^{{\prime }} ,\text{NA}_{{\text{peak}}}^{{\prime }} ) \) were produced by −45 ° rotations. The results of the correlation analysis between the new peak coordinates and the PANAS scores were shown in Table 3. There were the significant correlations between PA and \( \text{PA}_{{\text{peak}}}^{{\prime }} \) and between NA and \( \text{NA}_{{\text{peak}}}^{{\prime }} \). It suggested that the coordinates obtained by the proposed method were highly consistent with the response of the self-reports.

5 Conclusion

This research proposed the continuous affect rating method that rated the affective states by joystick in the Cartesian space from two dimensions of pleasure-displeasure and arousal-sleepiness without visual feedback. The effectiveness of the proposed method for rating the affective states caused by the dynamic visual and auditory stimuli was examined through two experiments. The results of Experiment 1 revealed that the coordinates obtained by the proposed method changed according to the changes of the dynamic visual and auditory stimuli, and that the coordinates could reflect the changes of the affective states caused by the dynamic stimuli. The results of Experiment 2 revealed that the proposed method could rate the difference of the affective states between the different dynamic stimuli confirmed by the results of the responses of the self-reports. The results also suggested that the coordinates obtained by the proposed method were highly consistent with the response of the self-reports. From these results of the experiments, it seemed that the proposed continuous affect rating method was effective though the proposed method had no visual feedback.

This research could not confirm sufficiently the validity of the time series variation of the coordinates obtained by the proposed method. In the future research, the coherence between the time series of the coordinates by the proposed method and the time series of some physiological data such as heart rate will be tested [6].