Keywords

1 Introduction

For a long time, designing interactive systems and devices has mainly concerned itself with identifying target users, activities to be supported by interacting with the each type of device, appropriate interfaces and ways of interacting with the devices [1]. With usability for single use of device ensured, cross-device interaction approaches largely focus on maintaining the consistent interfaces and content synchronization [2, 3]. While these established design guidelines and practices are applicable to the shift from one device to another, they tend to limit the issues within the interactivity of 1-to-1 device. When two or more devices are used together, again, the usability aspect of user interaction with devices is constrained within that particular combination of devices [4,5,6,7].

Usability issues caused by the above self-contained usability practices start to surface when the users spontaneously change the use of devices in the manner that moves away from intended context for which the devices, systems or services are designed and built for. They escalate along with the appearances of new interactive devices, the diversity of user activities that potentially benefit from employing multiple devices, especially when the users create their own combinations of device usage which may not be foreseen by designers or developers. Furthermore, would existing UI design principles, guidelines and practices–which had mostly evolved assuming a single device use–still hold true when extended to different combinations of devices? If not, in what ways could existing body of knowledge/know-how be revised or complemented in view of this?

This work aims to take the multi-device shifting as a lens to re-visit the usability factors to evaluate the effectiveness of such interactivity and user experiences. Specifically, we explore how device transition could support spontaneous shifts in device engagement such that the level of usability is maintained or improved throughout the task. We discuss how transition-support design principles should be treated as a holistic view of pre-transition, transition and post-transition phases. These 3 phases form a transition pathway which allows easy communications among user-device, device-device and user-user through its informative transition interface. Shifting cues–the main component of the informative transition interface–are introduced and described with respect to each phase in the transition pathway. We conclude the paper with the description of our prototype that is guided by the proposed transition-support design concept.

2 Related Work

Studies on multi-device interaction have proposed various ways to utilize multiple devices to improve the usability of single device alone. This section reviews and discusses the current landscape of usability issues for the shifts in device engagement during a task.

2.1 Device Substitution (Sequential Interaction)

There are a variety of reasons for which a user decides to continue a task on a device other than the one where it has been started: re-accessing content from different devices [4], unavailability of applications/devices (e.g. required applications are not supported, device runs out of battery), user’s preferences on device usage (e.g. different devices for different usage), or the transformation in the characteristics or nature of the task [8].

In the context of within-task device switching, interface consistency and content continuity between the states before and after the substitution of device have been identified as essential factors for a coherent experience [2]. The former can be achieved by synchronizing data and its structures across devices (e.g. Google cloud services) while the latter can be ensured by migrating the state of user activity from one device to another [3]. Usability factors such as how the user is informed about the “continuity” capability, what steps would be required to achieve it, or which device would be optimal for the task in the user’s context are under-explored. For example, after switching from a phone to a tablet, a user may not know that the pages browsed on the phone can be re-visited easily on the tablet (e.g. via the history menu item under the settings of Google Chrome) if the same credential is used to sign in on both devices. The user is left with the transition hurdles: learning the basic requirements, applying the required setup steps and deciding which device to use to suit the context of the task and the user. In addition, in situations when the pending task is to be resumed on a device belonging to a different user–thus may have different ways of interaction with device–would the interface of the first device still be desired to maintain its consistency?

2.2 Device Adding (Intermittent/Simultaneous Interaction)

Adding devices to assist the operation of a single device is prevalent both in research community and the practices of general public today. The added devices can be used intermittently for utilizing resources offered by other devices. For example, in avoiding the need of switching between apps on the same device, people sometimes consult a digital language dictionary app running on a phone while reading an article written in a foreign language on a tablet.

A great number of research has focused in leveraging parallelly two or more devices for a single task such as placing multiple mobile devices next to one another on the same surface to form a larger display for co-viewing of shared content (e.g. Pass-them-around [9], JuxtaPinch [10]) or for moving an object beyond its screen boundary (e.g. Pinch [11, 12]). When combination of device screens could not be placed on the same plane, the smaller screen devices were often used as sub-displays and/or extended input devices to the larger screen devices. Examples of such device combinations include phone-large display (PocketPIN [13], PresiShare [14]), smartwatch-phone (Duet [15], TakeOut [16]), and smartwatch-large display [17, 18]). Again, the user must know and remember how to initiate/disconnect ad-hoc communication (e.g. bumping 2 devices against each other [12], performing pinching gesture across the displays of the 2 devices [9,10,11], scanning visual marker [13, 14]), what kind of input commands can be used for the joint interactions (e.g. using knuckle-touch to move icons on the home screen, the same knuckle-touch is also used for selecting multiple items of a list in an email management app [15]). At usability level, it requires either the users or the systems to remember and adapt to the interaction strategy when using different device combinations.

2.3 Device Removal (Sequential/Simultaneous Interaction)

Reasons for removing device(s) from a pre-setup ecosystem of devices include unavailability of device(s) (e.g. running out of batteries, device is used for other task, lending device to a family member) or changes in user context making it no longer favorable for such device combination (e.g. leaving home for work). Transition between the states before and after device removal seems to be a neglected topic, resulting in unexpected disruption when such withdrawal of device happens. For example, lifting up a device would abruptly break the joint display [10, 11], stopping the use of phone as TV remote control when it is used for answering phone calls.

As reported, changes in device engagement in the above solutions lack the device-to-device transition support, thus imposing this burden on manually learning, discovering and combining devices on the user in a “self-service” style. In the next section, we explore the usability factors that foster transition between devices or device combinations.

3 Device Transition

As can be easily assumed, engaging and coordinating different devices would require a suitable UI and interaction strategy that supports the user’s intended task/activity. Usability aspect has been studied exhaustively for single use of device that is mostly used today, but not for those interactive devices that might enter the market in the near future. Likewise, many studies have focus on creating appropriate UIs and interaction strategies for various but expected and planned device ecologies, but not for a spontaneous combination of heterogeneous devices. It is impractical to design for every possible combination of devices that end-users may think of, so how would we achieve a sustainable design that can be reused, easy to disassemble to components to be recomposed for new interactive device or device combinations?

With usability mostly ensured within the known device or device ecology boundary in which each device takes up its own role in the overall interaction, basic design components can be formed from there. In our preliminary study [19], we refer the packing of device/device combination and such appropriate UI and interactivity together as device composite. The simplest form of a device composite is a single device that comes with its established UI supporting possible user interactions (Device Solitude). More complex forms of device composites involve two or more devices, each composite corresponding to a different generalized scenario of device engagement (Exclusive Input-Output, Shared Input, Shared Output, Shared Input-Output and Device Companion). In the context of shifting between device composites, interaction does not just happen between the user and the device but implies the coordination in device-to-device communication and user-to-user collaboration when such device engagement takes place. With the categories of device composites as the basis, in this paper we further abstract the situations of shifting from one device composite to another and structure the usability issues that need to be addressed in those situations.

3.1 Transition Usability

Usability has traditionally been characterized by learnability, efficiency, effectiveness, memorability, reliability in use, and user satisfaction [20, 21] and still widely used in evaluating the level of usability of user-interfaces today. When examining the shift from one device composite to another, we asked ourselves: what could be added or modified to the our understanding of usability in the form of design principles, heuristics and wealth of guidelines available today, in order to maintain or to improve usability when there are shifts in device engagement during a task?

We sought the answer for the above question through our preliminary study [19]– which we briefly describe here. 18 students and researchers (9 males)–age range from 21 to 55–who use multiple devices for their daily tasks/activities volunteered in the study. Participants brought a friend or came alone; in the latter case, one of the investigators acted as the participant’s friend. They were presented a situation when shifting from one device composite to another could be easily carried out to support the goal of the tasks. The tasks were designed to challenge participants with the typing of varied-length search queries on the small screen of the smartwatch. At any time, participants were able to (but not required to) temporarily borrow the friend’s smartphone and shift their input between the smartwatch and the smartphone, or coordinate with the friend in interacting with the system. We encouraged participants to think aloud during the session.

After completing all tasks, participants were asked to fill out a questionnaire which consisted of open-ended questions about the experiences they had during the session. Participants were asked to rate on five-point Likert scales how easy/difficult they considered it was to initiate the device transition and subsequent shift between them. We encouraged participants to give short explanations for their rating.

Findings from the preliminary study show that participants’ decisions on within-task device changes were mainly driven by the ease of transition and the low transition cost–the cost incurring before, during and after transition. Other factors include the usability of an interface modality for a given action, the complexity of the task and the expectation of potentially faster, easier task completion. The 5 design considerations drawn out from the user study (determining ideal UI and interactivity, support for changes, smart shifting cues, situational feedback and informative environment for spontaneous device shift) emphasize the 2 key objectives in device composite transition: minimizing the resources expended in discovering and handling potential shift in device engagement, and maximizing awareness of potential collaborative interactions. The reported results and findings further led to the construction of the following design principles that stimulate the opportunistic joining or leaving of device involved in a user’s goal.

Partnership Discoverability.

This principle calls for design that lets devices within proximity to learn about each other’s capabilities, the possibility and impact of potential collaboration. Learning the existences of devices surrounding a user’s device can be achieved by using spatial sensing of devices (e.g. [22]) or through location-based services. Current ad hoc networks (e.g. Bluetooth, Wi-Fi Direct) supporting device discovery can be augmented with additional information about the potential impacts of collaboration and the friendliness of device-to-device social relationship [23] (e.g. 6.4 inch screen, loud speakers for music, Swype keyboard ~30-35 wpm, SocioCon-FriendsFootnote 1).

Nearby devices can be emphasized or be made easier to choose using the user interface technique known as proximate selection [24]. Device candidates can be recommended by applying criteria to suggest the best suited device or a group of devices that could be connected within the space a user can interact with. The criteria could be based on the information about the nearest distance to a user’s device, the device-to-device social relationship or the highest percentage of matching between the goals and the capabilities to be provided by the shift-to device. The main issue that this principle tries to capture is that with these available technical solutions to support the discovery of connectable devices, what type of information and in what ways the discoverability should be informed to the end-user who faces the device shifting situation.

Role Election.

This principle calls for design that allows a device to nominate itself or other device for a suitable role (input device/output device/input-output device) in the combined use of devices or when the context of use changes. The common 1-way practice we often see is when connecting a device to auxiliary device(s), the latter would take over the intended modality, for example a projector will automatically act as the output device when it is connected to a laptop. This principle tries to capture the usability implications of such role election needed to be conveyed to the end-users in a suitable way. Role election would likely be followed by the UI-interaction election–to be discussed below.

UI-Interaction Election.

This principle calls for design that enables a pair/ecosystem of devices to vote for the most suitable UI and interaction strategy for the context of use. Adaptation in UIs-interaction techniques may be needed when there is a mismatch in input or visual output space due to differences in device form factors, or when replacing a device that is part of a special UI-interaction technique applied system (e.g. [13, 17, 25]). Pushing suitable UI when there are changes of number of devices engaged for the task has been demonstrated in [9,10,11] when a shared photo was displayed across two phones placing next to each other. Lifting up either phone resulted in the photo snapped back to the hosted device. As another example, a user was presented 4 choices of media consumption on the phone when moving away from a TV: continue, continue with downscaled content, buffer the content for later use, and disconnect [26]. This principle tries to capture the aspect of device-shifting situations where the user can choose the most optimal way of UI to continue the task depending on the actual context of use.

Figure 1 presents the bi-directional shifts between device composites. In the graph, device composites are ordered according to the minimum number of devices within the respective device combination (number of devices increases from left to right). Shifting from a device composite towards the right/left side of the graph involves adding/removing one or more devices respectively. Shifting towards the top/bottom direction mainly involves the changes in device roles.

Fig. 1.
figure 1

Bi-directional shifts between device composites involve either the addition (+) or removal (−) of device(s), followed by Role Election (RE) in some cases. Input Shift (IS) and/or Output Shift (OS) take place within the device composites having shared input and/or output device(s).

Full size image

Recall that each device composite corresponds to a different generalized scenario of device engagement, thus a new interactive device will be classified as Device SolitudeFootnote 2 while any spontaneous combination of devices would be classified into the device composite that best characterizes its inter-device interaction. Characteristics of changes in device engagement will call for the application of appropriate design principles. Pretend that after opening a lesson slide on a phone (Device Solitude), a lecturer receives a notification of an available nearby interactive wall display situated just around a corner. He and his students walk towards it. The lecturer casts the lesson on the wall display and uses his phone as a laser pointer (Exclusive Input-Output) while explaining the concept to his students. His students can also turn their smartwatches or phones into laser pointers to join in the discussion (Shared Output). This fictional scenario contains 2 levels of shifts: between device composites and within a device composite. The first shift from Device Solitude to Exclusive Input-Output involves adding a new device (wall display) and a nomination of the phone becoming an input device. This role election triggers the election for UI-interaction strategy that is suitable for the interactivity. The second shift from Exclusive Input-Output to Shared Output happens when more devices joining the interaction session. Role election takes place, followed by internal Input Shift (shift within a device composite) when the lecturer authorizes his students devices to become auxiliary input devices to the wall display. The Input Shift and Output Shift models are parts of the transition models that describe the shift of input or output modality from one device to another [23].

3.2 Transition Pathway

Transition-support design should consider all usability factors and attributes as an interconnected whole that is part of the 3 phases: pre-transition, transition and post-transition. The transition pathway can be traversed either forward (forecast) or backward (backcast) in time. In the forecast approach, recognizing the user’s potential choices of device engagement (pre-transition) is the starting point to determine user efforts, potential conflicts in resources required for device shift and UI-interaction strategies (transition), and the consequences of such decision (post-transition). The backcast approach starts with the speculation of desirable outcomes (post-transition) that meet the user’s goals in terms of different perspectives such as resource optimization, user satisfaction or context matching. From there, tracing back to identify the gaps/conflicts between the current and the expected/optimal UI-interaction strategies, determines the user efforts and resources utilization needed for achieving the user’s goals (transition), and finally identifies the best matching device (pre-transition). Both directions share the common key steps: prioritization of user perspectives/choices, identification of appropriate UI-interaction strategy for targeted shift-to device composite, and estimation of user effort and resources required for the device composite shift. Figure 2 details building blocks of the transition pathway.

Fig. 2.
figure 2

Building blocks of the transition pathway.

Full size image

What happens after the post-transition phase? Looking back, a device composite shift starts from the pre-transition phase, goes through the transition phase before reaching the post-transition phase. With each phase’s achievement comes the closeness to the user’s goal–a device composite with its usability that supports the user’s task or needs. The time spending at each phase varies and so does the interaction between user and device composite after the post-transition phase. The user can continue the task by interacting with the newly-shifted device composite. It could be possible that another shift in device composite (e.g. reverting back to the previous device composite, shifting to another device composite) is needed either immediately or after a short interaction with the device composite. And so, the interaction cycle of user-device composite goes on with each new instance of device composite resulted from the device composite shift. We call this The User-Device composite Cycle, of which the transition pathway is responsible for maintaining the continuous interactions between users and devices participating in various instances of device composites.

3.3 Informative Transition Interface

The key challenge in shifting between device composites is how to transcend differences between device composites. Transition phase plays important role in bringing together a variety of existing UIs-interaction design approaches, managing potential gaps and conflicts between device composites. As such, navigating through the transition pathway requires efficient communications of information necessary for a task to be resumed smoothly on a another device composite.

We formulate shifting cues, a set of independent user interfaces designed to inform users about available choices of device composite shift and to convey user’s intention on device shifts. Perceived as 2-way cues between user and device, shifting cues include explicit cues in the form of UI elements (e.g. buttons), natural input modalities (e.g. voice commands) and implicit cues in the form of reading user’s intention (e.g. facial recognition, emotion recognition). How shifting cues should be presented to users is critical because they could be either perceived as a disruption to the user interaction and attention, or welcomed as a stepping stone for a more efficient way to complete the task. Current design practices suggest that system alerts would be used when they convey information that will benefit users, those with high importance should be pending for user’s response–unless the situation is resolved–while others with low importance can be bypassed automatically after some time. However, the system should not send users myriad of real-time request, feedback or response for every shifting cue, considering the increasing frequency of device-shifting environment envisaged.

Table 1 details a non-exhaustive list of the shifting cues for the building blocks of the transition pathway. This way of tabulation then allows how the usability evaluation could be structured and measured by that includes a device transition as part of the task-completion interactivity.

Table 1. Shifting cues in pre-transition, transition and post-transition phases.
Full size table

4 Prototype

To illustrate and validate the proposed transition design principles and informative transition interface, we extend our prototype developed in [19] to involve spontaneous device engagement to happen any time during a task session. Multimedia Browser is a prototyped mobile app running on Android mobile and wearable devices. When it runs on a single device, input and output modalities are self-contained on the same device (Fig. 3a). User can browse through media content using direct touch input on the device screen. However, when the content is casted to a nearby large display, user will be prompted with a message (Fig. 3b) for converting the device role (role election principle). As soon as the mobile/wearable device is turned into a remote input device for the large display (Fig. 3c), the hop-to-select traverse strategy [17] is voted to facilitate the user’s visual focus on browsing media content on the distant display (UI-interaction election principle). With this interaction strategy, the user can perform motion gestures (e.g. shaking device) and coarse relative touch gestural input (e.g. tapping, swiping) on the smartwatch screen without looking at it.

Fig. 3.
figure 3

Single device use: smartwatch screen displays photo (a) and prompts user to change the device role (b), co-using devices: smartwatch screen becomes input surface when the watch is used in conjunction with a simulated large display (c).

Full size image

In addition to content navigation and browsing, the user can invoke the device’s on-screen soft keyboard to enter the search query for the content of interest. In situations when the user finds it difficult to type on the small screen of the smartwatch and wishes to seek for an available, better suited device for the task, tapping on the Input-Shift button (Fig. 4a) will bring up the information of the closest device (Fig. 5a). Alternatively, an alert message (Fig. 4b) is presented to users when certain number of repeated errors (e.g. 10 typographical errors in keyboard typing) are made. If this request happens on a device having larger screen (e.g. phone, tablet), a proximate selection map–employing the proximate selection technique [24]–visualizing surrounding device locations with respect to that of the current device (Fig. 5c) will be presented (UI-interaction election principle).

Fig. 4.
figure 4

Shifting cues in co-use of smartwatch, phone and simulated large display by two users.

Full size image
Fig. 5.
figure 5

Detailed view of a nearby device on the smartwatch screen (a), proximate selection map of nearby devices on the smartwatch (b) and the phone (c). This map shows each potential partnership device as a circle containing a device icon. The size of each circle is determined by the distance from the candidate to the current device. The circle color denotes its social relationship to the current device. (Color figure online)

Full size image

The proximate selection map shows each potential partnership device as a circle containing a device icon and information about distance, device name and social relationship between the two devices (partnership discoverability principle). The circle color denotes its social relationship to the current device. We use the 5 device-to-device social relationships identified in [23]: SocioCon-Buddies, SocioCon-Family, SocioCon-Friends, SocioCon-Friends of Friends and SocioCon-Public. The size of each circle can be determined by the distance from the candidate to the current device, or by the friendliness in device-to-device social relationship. This map can also be showed on the smartwatch when the user chooses to toggle between the single view (Fig. 5a) and full view (Fig. 5b) (partnership discoverability principle). Successful engagement of devices happens after the owner of the target device agreed to the request (Fig. 4c), after which user interaction can be shifted between both devices spontaneously.

5 Conclusion

While usability is ensured when using a single device or a pre-determined pair/ecology of devices, it might not be the case when changing to or involving another device. By considering device/device combination with its appropriate UI and interaction strategy as a basic design component–a device composite, any unexpected combination of devices can be classified to the most suitable type of device composite after which the established/interim UI-interaction strategy is activated.

Now that the co-use of multiple devices is considered, how should we see the usability issues different from conventional single-device use?: (1) optimal UI and interaction strategy for the combined use of devices needs to be studied, designed and developed; a mechanism of how unplanned device combinations can push a particular UI when it was not pre-conceived needs to be studied; (2) design principles, guidelines/heuristics we have so far need to be revised/revisited, for example, principles and guidelines for combined use of devices with large discrepancy in screen sizes and resolution (e.g. smartwatch and public display); and (3) evaluation of usability needs to take into account not only the usability of single setup, but the trade-off in usability before-during-after device shift and the consequent experience of it.

In this ongoing project, we identify the partnership discoverability, role election and UI-interaction election as enhanced transition design principles that facilitate spontaneous shifts in device engagement. Our prototype–guided by the proposed design principles–is designed for supporting opportunistic joining/leaving of device involved in a user’s task. Our next steps include the development of a complete eco-system of multi-device prototypes and conducting usability study to validate the proposed design.