Keywords

1 Introduction

Report has shown that consumer owns an average of four digital devices [1], which means that the number of touchpoints has expanded with the rise of device ownership. Multi-screen services like Netflix, Allrecipes and Fitbit have broken usage boundaries and allowed users to freely shift from one device to another. Multi-screen services involve consumer interacting with multiple devices and is characterized by the different location, duration and purpose of use compared to single screen services. Multi-screen services have formed new kinds of interaction experiences: consistent, continuous and complementary experiences [2]. Thus, the user-device relationship now involves more than one device and has become even more complex. According to CASA (Computers are social actors) paradigm, users tend to interact with computers as if they are human beings and apply social rules (e.g. politeness) to them [3]. This study aims to take it further by investigating the relationship between users and multi-screen interaction experiences, with emphasis on users’ emotional responses to the experiences and personality traits they assign to the interactions.

2 Literature Review

2.1 Interaction Experience

When devices operate within a multi-screen service, the relationship on how devices connect with each other to accommodate different services or to conform to the users’ behavior, have formed new kinds of interaction experiences. How people interact with multi-screen is different from the traditional way of interaction with single device. Levin has revealed three kinds of interaction experiences: consistent, continuous and complementary experiences [2]. Consistent interaction refers to the same experience interacting with similar contents, visual effects and main functions with different kinds of devices. These characteristics decrease people’s learning curve when migrating through devices, and people can intuitively and easily adopt the best available device to be used according to their wishes. Continuous interaction emphasizes on its continuation in accomplishing a single or sequential of activities across various devices; therefore, the coordination between devices is determined through users’ activity flow, duration of the activity, context of the activity and the number of subtasks needed to be accomplished. Activities like cooking, trip planning or reading books on multiple devices can be done in a more seamless and sophisticated way of interaction. Complementary interaction is the most representative interaction of the multi-screen ecosystem. Two or more devices in an activity work together, or complement with each other or control one another to elevate the overall experience. It is the most accessible for users since users can use their own device as a starting point and extend the experience beyond what was already in existence. For instance, using one’s own smartphones as a way of input in a multi-player game so everyone who owns a smartphone has the opportunity to participate in the game and thus promote social connections. Also, devices could serve as second-screen to provide additional information to enrich the main activity such as receiving movie description and viewers’ comment on a tablet that corresponds to the movie that is currently being watched [4].

These three kinds of multi-screen interactions have provided various impacts, and the experiences can only be amplified and accommodated when users’ behaviors have been identified and when the actual needs were fulfilled. Prior research have emphasized on multi-screen user behavior such as energy intake [5] or human activity for system recommendation [6]. There is a deficiency in the understanding of the interplay between multi-screen interaction and user perceptions. Höök et al. stated that the overall experience of interaction could not be explained fully by the knowing of only traditional human factors; however, users’ perceptions such as emotion, understanding, and interpretation are also as equally as important [7]. Therefore, this study aims to identify the emotions and personality traits perceived from multi-screen interaction, which has never been done in this aspect before and to provide more information from the user perception point of view.

2.2 Emotion in Multi-screen Interaction

Emotions play an important role in human-machine interaction as stated by Norman [8], and this has been extensively studied and analyzed in HCI (Human-computer interaction). From a user experience point of view, emotional responses can be influenced and determined by interpreting the iterative interplay between the context and the continuous actions [9]. The emotions identified and analyzed in this study are the results and consequences of interactions with multiple devices, which is different from the concepts of seeing emotion as an antecedent or mediator to product use [10]. Most often, emotional responses are associated with the needs and goals of an individual. In an information-seeking context, if the user interfaces that users interacted with can help to navigate or facilitate in finding desired information, users’ emotions are often positive. That is, users’ emotional experience can be directly impacted by the user interfaces to the extent of conforming to their needs and goals [11]. Therefore during the context of multi-screen services, interaction experiences occurred with the desire to accomplish certain goals, so users’ emotional responses would be determined by whether the interaction could successfully assist users toward those goals. Although it is hard to say what kind of emotions users would experience from these multi-screen experiences, but certainly they would influence users’ emotions [12]. For this reason, this study assumed that different emotions would be elicited from the different interactions in consistent, continuous and complementary experiences. Identifying the emotions from interactions in multi-screen services can help to determine which interaction to assign to a given activity that is consistent with users’ emotional expectations or the demand of the services. The study of the non-instrumental aspects through emotions and personality traits enriches the current field of research with more knowledge on how people perceive this kind of newly emerged interaction experiences.

2.3 Personality in Multi-screen Interaction

Based on different contexts of use in multi-screen services, people tend to personify their devices with certain personalities and roles that go beyond their instrumental power. For example, tablets are interpreted as an on-the-move professional assistant in formal business context [13], and devices are described as companions [14, 15]. Personality traits applied to products are originally used in describing human beings, and users are able to differentiate them among products [16]. Since people interact with devices similarly to how they interact with human beings, being aware of their device personalities can help them better mange tasks successfully, just like how people in social interactions with each other [17]. Most often, a person’s personality can be assessed from his or her facial expressions, personal styles or postures, and a product’s personality can be perceived from its shape [18] or material choices [19]. Since appearance is an important determinant in the perception of personality in products, user interfaces and computers, it is unclear whether the same theory applies to multi-screen interaction where more than one device are involved. Desmet et al. have analyzed the relationship between interaction, appearance and personality in physical interactions, which demonstrated that appearance is strong in interaction personality, but personality is not merely the sum of product appearance and the effect of interaction [20]. Therefore, it would be interesting to probe into the attributes constituting personality in multi-screen interactions. This study aims to understand people’s perception toward multi-screen interaction, comprehend whether the interactions can be described with personality traits, and to explore the relationship between user perception and the three kinds of interaction experiences.

3 Method

In order to get users’ real feedback, the experiment simulated the real world setting, and invited participants to physically interact with the devices. The purpose was to find out users’ perception towards consistent, continuous and complementary interaction experiences through post-interaction questionnaire and a follow-up interview to answer whether personality traits can be discovered in multi-screen interactions and the relationship between user perception and three kinds of multi-screen interaction in respect of emotion and personality.

3.1 Participants

A sample of 16 participants (9 male and 7 female, mean age = 28.63) participated in this experiment. Majority of them had experiences in using two to three devices along or in conjunction with one another in completing a task. On a daily basis, participants owned and used smartphones (100 %), tablets (44 %), laptops (81 %), desktops (81 %), Internet-enabled TV (13 %), smart-watches (6.3 %) and other wearable devices (6.3 %).

3.2 Questionnaire

A paper-based post-interaction questionnaire was used to collect participants’ impressions after completing each task. The questionnaire includes questions regarding emotion and personality descriptions. For personality assessment, a set of 24-item personality traits was adopted from previous studies [20–23]. Participants’ emotional responses were assessed with the positive and negative affect schedule (PANAS) [24]. Questionnaire data were analyzed using Ward’s hierarchical clustering method, and selected by means of the cluster and variation. Therefore, ten items in total were retained to be used in the experiment: cheerful, open, relaxed, easy-going, cute, interesting, dominant, boring, gentle and preference for personality scale; and strong, enthusiastic, inspired, determined, attentive, active, interested, distressed, hostile and irritable for emotional scale.

3.3 Apparatus

The hardware used in this experiment includes an Apple iPhone 4 (iOS version 9.2.1), a 42-inch TV, a 13-inch MacBook Pro and a 13-inch Asus laptop. The user interface images shown in the experiment were designed previously in graphic design software. Since appearance is a major determinant in product personality [18], in this experiment the appearance of user interfaces were controlled, so the design elements in all images remained identical in terms of color, icon and style. The images were then imported to and simulated on smartphones with POP prototyping application (Woomoo Inc.), which enables the simulation of applications participants would interact on their smartphone. For the simulation on laptops and TV, Microsoft PowerPoint was used to simulate webpage browsing and TV watching. One of the essential features of multi-screen interaction is that it is context sensitive. Therefore, the experimental setting divides the room into three distinct areas: living-room area, study room area and an outdoor area (Fig. 1). Participants were informed about the purpose of each area before the experiment took place.

Fig. 1.
figure 1

Experimental setting

Full size image

3.4 Task Procedure

The scenario of the experiment was developed based on the topic: exercise. With the emerging number of people using multiple devices like smartphone, laptop or wearable devices (e.g. Nike fuelband, Fitbit, Apple watch, etc.) to record, keep track and share their physical activities, exercise is a good example to demonstrate the multi-screen service therefore it was chosen for this study. Based on the scenario, three tasks were designed according to three multi-screen interactions of consistent, continuous and complementary design. In order to simulate the interaction and usage between two to three devices together, the Wizard of Oz (WOz) technique was applied. During the experiment, participants were told that the devices they are using were all connected together in the multi-screen ecosystem by a service called Myworkout, and in the following three tasks they were going to interact with different kind of interaction provided by the service (Table 1).

Table 1. Task description
Full size table

As the experiment was a within-subjects design, participants were required to perform all three tasks. The experiment was carried out in an individual basis. Participants were assigned to three different task sequences (e.g., ABC, BCA, CAB). The entire experiment lasted approximately 30–40 min.

4 Result

The method by Ward’s hierarchical clustering was used to analyze the post-interaction questionnaire. The cluster analysis was later ascertained by one-way ANOVA to see if there was significance among the clusters and to determine the level of the clusters.

4.1 Personality Clustering

In consistent interaction, the items regarding personality are grouped into four clusters: 1st cluster (cute, boring), 2nd cluster (interesting), 3rd cluster (cheerful, easy-going, gentle) and 4th cluster (relaxed, preference, open, dominant). In continuous interaction, the items regarding personality are grouped into four clusters: 1st cluster (cheerful, relaxed, easy-going, preference), 2nd cluster (open, dominant), 3rd cluster (cute, gentle) and 4th cluster (boring). In complementary interaction, the items regarding personality are divided into four clusters: 1st cluster (open, easy-going, relaxed, cheerful), 2nd cluster (dominant, interest, preference), 3rd cluster (cute, gentle) and 4th cluster (boring).

4.2 Personality in Multi-screen Interaction

In consistent interaction, the clusters differed significantly in terms of personality (F3,156 = 31.914, P = 0.000 < 0.001). With the LSD post-hoc test applied, it was revealed that cluster 4 is larger than the other three clusters. Participants thought consistent experience had a relaxed, preference, open and dominant personality. In continuous interaction, the clusters differed significantly in terms of personality (F3,156 = 40.195, P = 0.000 < 0.001). With the LSD post-hoc test applied, it was revealed that cluster 2 is larger than the other three clusters. Participants thought continuous experience had an open and dominant personality. In complementary interaction, clusters differed significantly in terms of personality (F3,156 = 37.72, P = 0.000 < 0.001). With the LSD post-hoc test applied, it was revealed that cluster 1 is larger than the other three clusters. Participants thought complementary experience had an open, easy-going, relaxed and cheerful personality (Fig. 2).

Fig. 2.
figure 2

Personality clusters in multi-screen interaction

Full size image

4.3 Emotion Clustering

In consistent interaction, the items regarding emotion are grouped into four clusters: 1st cluster (determined, active, attentive, interest), 2nd cluster (enthusiastic, inspired, strong), 3rd cluster (distressed, irritable) and 4th cluster (hostile). In continuous interaction, the items regarding emotion are grouped into four clusters: 1st cluster (determined, attentive, enthusiastic, inspired), 2nd cluster (active, interest), 3rd cluster (strong, distressed) and 4th cluster (irritable, hostile). In complementary interaction, the items regarding emotion are grouped into three clusters: 1st cluster (inspired, interest, active, enthusiastic, determined), 2nd cluster (strong, attentive) and 3rd cluster (irritable, hostile, distressed).

4.4 Emotion in Multi-screen Interaction

In consistent interaction, the clusters differed significantly in terms of emotion (F3,156 = 53.122, P = 0.000 < 0.001). With the LSD post-hoc test applied, it was revealed that cluster 1 is larger than the other three clusters. Participants felt determined, active, attentive and interested. In continuous interaction, the clusters differed significantly in terms of emotion (F3,156 = 103.52, P = 0.000 < 0.001). With the LSD post-hoc test applied, it was revealed that cluster 2 is larger than the other three clusters. Participants felt active and interested. In complementary interaction, the clusters differed significantly in terms of emotion (F2,157 = 127.38, P = 0.000 < 0.001). With the LSD post-hoc test applied, it was revealed that cluster 1 is larger than the other two clusters. Participants felt inspired, interested, active, enthusiastic and determined (Fig. 3).

Fig. 3.
figure 3

Emotion clusters in multi-screen interaction

Full size image

5 Discussion

The purpose of this study is to understand user perception in multi-screen interaction experiences, in particular what emotions would be perceived as well as the personality traits that users would assign to the interaction. From the result of the experiment, similarities were discovered among three interaction experiences: consistent, continuous and complementary. Participants thought all multi-screen interactions possessed an open personality that is perceived to be curious, creative, flexible and more opportunities. An open personality is part of the openness trait in the five-factor model described by Carver and Connor-Smith [25]. They demonstrated that openness trait is related to engagement coping, which refers to problem solving and cognitive restructuring. That is, the three multi-screen interactions have the tendency to provide positive impacts as the interactions serve to assist participants in accomplishing their goals and accommodate stressor in the activity process. In terms of emotion for the three kinds of interactions, similarities were also found as participants thought all multi-screen interactions elicited emotions that are active and interested. It was found that all of the interactions have been mentioned to be impressive, and participants felt interested and attracted toward the interaction they thought to be the most impressive and wanted their devices to interact in the same way. Overall, the multi-screen interactions are perceived to be open, active and interested.

The results of the experiment showed that different multi-screen interactions tend to associate with certain personality traits and emotions (Table 2). In consistent experiences, preference and attentiveness are the most representative personality and emotion in this experience. Preference usually occurred when the design elements are in concordance with the characteristics of its user group [22]. It can also be explained from the interview conducted, as participants’ prefer consistent design in icon position, layout and style that facilitate easier searching and positioning across multiple devices. The experience was perceived to be deliberate and provide users with a sense of familiarity. Therefore, consistent experience is suitable to apply to services dealing with enormous information content or complex information architecture when users are expecting simple, familiar interaction experience with minimum adjustment required across multiple devices.

Table 2. User perception in multi-screen experience
Full size table

Result showed that continuous experience is perceived to be open, dominant, active and interested. The term dominant is used to describe a person exhibiting tendencies to command, control or lead others to take action, and a dominant interaction is usually perceived from the order of interaction, the confidence level and the language displayed on the computer [26]. In the services where continuous experience takes place, users often got asked to make certain decisions for the actions to be performed across multiple devices for continuation of the tasks. A too dominant interaction may make users feel imposing and forceful, while a less dominate interaction may be too weak and disorganized. It is therefore vital for designers to take into consideration the personality traits of the experience and their impact since these may interfere with users’ reaction to an activity. Services dealing with user behaviors and contextual information such as scheduling or quantified self movement, the spontaneous and convenient characteristics of continuous experience may be preferable to enhance such services. The term “convenience” has been emphasized several times by participants to address the freedom of the interactions. The technical foundation of continuous experience thus needs to be well designed so that every device can work smoothly with on another to provide the freedom of device shifting for the users.

In the complementary scenarios, users perceived this kind of experience as cheerful, easy-going, enthusiastic and inspiring, which were perceptions not perceived in either consistent or continuous interaction experiences. In an information-seeking approach, an easy-going person usually prefers easy access to information that requires less effort and thoroughness [27]. Therefore, complementary interaction experiment is perceived with a spontaneous, uncomplicated and relaxed feeling. Users could utilize their existing devices to collaborate or control other devices. This low entry barrier reinforce the anytime, anywhere concept, which makes users feel emotionally positive and inspires them to pursue further activities with other devices. Moreover, the combination use of multiple devices not only deepened the content knowledge in a service, but also elevated users’ experiences to another level that are novel, attractive, smart and well-executed. With these perceived characteristics, complementary interaction experience is best adopted to enhance the fun experiences in entertaining services, and to enrich with constant inspiration and motivation in exercising or productivity services.

6 Conclusion

In multi-screen services, interaction experiences were the results derived from the contexts of activities, the number of available devices, and users’ personal goals. The results of this study showed that user perception could be elicited from multi-screen experiences, and variations of emotions and personalities can be found among consistent, continuous and complementary interactions. As mentioned earlier in this study, users interact with computers through social norms similarly to human interactions. As a consequence, future research could investigate relationships among user interfaces, interaction emotions and personality traits in the multi-screen services. This would verify the effects on whether the attributes of user interface such as color, input/output, message route, or information content are compatible with the types of interaction experiences.

The characteristics of the different interaction experiences may serve as a criterion to be adopted in an activity, as it is undeniable that certain interaction is more suitable for certain activity. This study wants to aggregate on top of what has already known and provides additional insights from user perspective for designers to adopt the appropriate interaction to match with their design goals. It is also useful to enhance existing services so that the conveyed interaction emotion and personality is in consistence with the purposes of the services. Furthermore, it can be utilized as an evaluation tool to examine the discrepancies between user perception and the services they use. Even though the subject of this study focused on the issue of exercise, the experiment was conducted through simulating real world situations under well-controlled experimental conditions. Therefore, results from this study can be further analyzed and used in different subject matters, providing further insight in the field of multi-screen interactions.