Providing Daily Casual Information Through Eye Contact with Emotional Creatures

Abstract

Our everyday life has become increasingly enhanced through the progress of advanced information technologies. In particular, the daily lifestyles of urban residents have changed significantly because of smartphones that keep them continually connected to the Internet. As a result, citizens are able to access diverse information sources anytime and anywhere. However, the current progress of these digital technologies dramatically increases the possibility of cognitive overload. In this paper, we propose Ambient Bot, a notification system that ambiently presents practical everyday information to people. Ambient Bot requires people to wear a lightweight head-mounted display (HMD) that shows the real world through a camera attached to the HMD. Users see an empathetic virtual creature in the real world via augmented reality technologies. The empathetic creature not only speaks but presents everyday information that people might be interested in only when they focus their attentions on the creature itself. We first present a design approach for the Ambient Bot; then, we describe a prototype system and its evaluation through a user study.

1 Introduction

Our everyday life has become increasingly enhanced through the progress of advanced information technologies. In particular, the daily lifestyles of urban residents have changed significantly because of smart phones that always connect people to the Internet, offering access to diverse information sources anytime and anywhere. However, the current progress of these digital technologies dramatically increases one’s cognitive overload. For example, in our everyday commute in Tokyo, a large amount of information is offered to citizens via public displays located on trains, stations and streets. As shown in [11], modern social media systems try to steal people’s available attention by using recent advanced information technologies. To make everyday life more comfortable and peaceful, information should be more ambiently and intimately delivered to individuals only when the information is truly necessary.

This paper proposes Ambient Bot, a notification system that ambiently presents practical everyday information to people. Ambient Bot requires people to wear a lightweight head-mounted display (HMD) that shows the real world through a camera, through which they see an empathetic virtual creature in the real world via augmented reality technologies. This empathetic creature speaks and presents everyday information of possible interest but only when users focus their attention on the creature itself. We first present a design approach for Ambient Bot; next, we describe a prototype system and its evaluation through a user study.

This paper is organized as follows. Section 2 presents an overview of Ambient Bot. In Sect. 3, we present its design approach. In particular, we discuss possible interaction styles as well as content that a virtual creature may say and/or present. Next, in Sect. 4, we describe the implementation of a prototype system. Section 5 presents the results of a user study that uses the prototype system and discusses some issues found during the user study. In Sect. 6, we show some research related to our study. Finally, Sect. 7 concludes the paper.

2 An Overview of Ambient Bot

Ambient Bot provides an agent that ambiently exists around a user; it presents information that he/she wants to know using augmented reality (AR) technologies without interfering with his/her current everyday activities. The aim of Ambient Bot is to offer an implicit low-cost interaction method wherein it is easy to access necessary information, in the various places and situations of everyday life; for example, when a user is in a public place, such as a station or when walking down the street. Ambient Bot adopts the user’s gaze as the basic input modality, while Ambient Bot’s visual text messages and spoken audio are the output modality. Various input modalities, such as hand gestures, voices, and controllers, have been proposed for AR environments; however, these methods are not naturally implicit interactions, especially in public places. On the other hand, interactions using eye contact cannot be used for complicated interactions or for accessing complex information, but these interactions can enable implicit natural interactions, bypassing explicit interactions like those related to the use of controller devices.

Ambient Bot allows a user to acquire information simply by using eye contact with a virtual creature that floats continually within one’s field of view, ready at hand when the user needs information. The virtual creature not only speaks but also displays information in a translucent window against an image of the real world. Figure 1 shows a user’s perspective, i.e., what a user sees via an HMD when using Ambient Bot. When eye contact has not been made, the virtual creature automatically moves to a position that does not interfere with the user’s view, ensuring that the user is not strongly conscious of the creature’s existence. To actualize interaction and for the creature to make an eye contact, it needs float around the user’s field of vision as described above. In the current design, we chose a jellyfish character as our floating virtual creature, as shown in Fig. 1. More detailed discussions related to choosing the jellyfish as a virtual creature can be found in [5]. Since such an animated virtual creature is popular in recent animations and games, especially for young Japanese people, this approach is not unnatural as a means of accessing information. In addition, we adopt this interaction design because it would be inappropriate to display humans, dogs or cats as floating creatures [5].

Fig. 1.

A screenshot of Ambient Bot presenting the news

3 Interaction Design for Ambient Bot

The current version of Ambient Bot adopts the pull-based interaction method defined in [6], where a user explicitly accesses information that he/she wants to know. In this method, since his/her objective is to access clear information, he/she needs to indicate to a notification system what kinds of information he or she is currently seeking. In conventional information access services, people usually select a specific application on the desktop of a personal computer or the smartphone home screen, acquiring the information they want by clicking the target application’s icon. For example, if a user wants to know the weather tomorrow, he/she usually picks up his/her smartphone, taps the weather forecast application icon, and thus, acquires the expected information. However, in the case of the basic Ambient Bot concept, user interaction instead entails making eye contact with a jellyfish¹. However, this eye-contact approach is likely insufficient for accessing complicated information because it is difficult to select the appropriate information with only eye contact.

This basic interaction design can be easily enhanced for accessing multiple information sources. A user can configure the types of contents to be accessed in advance, according to the time and situation. This design allows the user to acquire the desired information without hindering the implicit and natural interactions offered by the basic Ambient Bot interaction design. In the enhanced Ambient Bot, multiple virtual creatures appear around a user, with each virtual creature representing a different informational category. Figure 2 shows an overview of this interaction design. In this design, a user chooses a desired information category by selecting a virtual creature floating around him/her, and making eye contact with one of them, akin to selecting an application icon from a smartphone home screen. Showing multiple virtual creatures in the real world (where a user is) could lead to a cognitive overload, yet this load could also be reduced by carefully arranging the viewable positions of the creatures, with respect to the time and specific situation, ones that were previously configured by the user.

Fig. 2.

Basic interaction design in Ambient Bot

4 Implementation

The current prototype system of Ambient Bot for Android-based smartphones (Google Nexus 6²) was implemented so that it is feasibility for use in people’s daily life could be investigated. In this prototype system, a virtual creature is superimposed on video captured by a camera equipped smartphone. The video is shown in the smartphone’s display with Virtoba X6, where a user can watch the 3D VR image of the real world³. In this prototype system, we prepared three kinds of jellyfishes, each with different colors and patterns, as shown in Fig. 3. Additionally, three types of information content were chosen: news, weather forecast, and SNS, in which Web APIs like Google News API⁴, WeatherHack API⁵, and Twitter API⁶ were used to retrieve the latest information. Furthermore, a user can personalize these interfaces, choosing the virtual creatures that will provide each type of information content. Additionally, as shown in Fig. 4, a user can configure the appropriate information contents according to the current time and situation. For example, if the weather forecast is configured for 6–8 am in the morning, upon reaching this time, the weather-forecast-providing jellyfish appears in the user’s view. Specifically, in this prototype system, the jellyfish appears immediately outside of the user’s HMD viewing angle.

Fig. 3.

Three jellyfishes as virtual creatures

Fig. 4.

Configuring the personalization in Ambient Bot

In addition, we used two methods of presenting information to the user: the first entails showing information within a window and displaying it in the real world; the second was to read the information displayed within the window in synthesized speech, where Voice Text Web API⁷ was used for the synthesized speech. It was anticipated that this method would not always be desirable. For example, when a user wants to check the news while walking, it may not be appropriate to display an information window with a large amount of information in front of a user’s view. Therefore, a user can also easily disable to the information window when he/she feels that it is not desirable.

5 User Study and Discussion

We conducted a user study experiment to investigate the feasibility of the current Ambient Bot interaction design. We recruited 10 participants, including 8 males and 2 females, all of whom are university students between 21–24 years old who usually access information with their smartphones. In this experiment, we examined whether the interaction design of displaying multiple virtual creatures (as shown in Fig. 3) is effective or not, whether personalization of the interaction supports this interaction design well, and whether voice notifications and information windows disturb a user’s daily activities.

First, participants freely used Ambient Bot at the beginning of the experiment. Then, they configured the system with the appropriate information, according to the current time and by matching the virtual creature with its corresponding content. Next, participants used Ambient Bot in the following three situations: working on daily tasks at a desk, sitting in a train, and walking outside. Figure 5 presents photos showing each of the three scenes in our user study. Figure 5(a) shows a photo where a participant is working on daily tasks on a desk, Fig. 5(b) shows a participant seated in a train, Fig. 5(c) shows a participant walking outside. The working on daily tasks at a desk and walking outside conditions entailed the use of real environments, but the sitting in the train situation used a simulated environment, in which a user watched a video while sitting on a sofa in a room. The video was taken in advance on a public train using a 360° camera. In this situation, the prototype system presented a pre-recorded 360° movie instead of live video captured in the real world and displayed on a smartphone. In each situation, participants configured their Ambient Bot displays, deciding whether information would appear in a window or be read via voice synthesizer. They also decided the information delivery method appropriate for them. Finally, we administered a survey and conducted interviews with each participant.

Fig. 5.

Some scenes captured in our user study

5.1 Multiple Virtual Creatures Design

We investigated whether our interaction design of a user choosing a virtual creature much like they would choose an application icon on a computer desktop was effective as shown in Fig. 2. We compared the time required to obtain information using a smartphone versus that when using Ambient Bot. We examined whether a participant could quickly choose the appropriate target creature, similar to the way they choose a target icon of an application. Because all participants accessed information more frequently through Twitter than through the news or weather, we decided to use the latest Tweets from each participant’s timeline as the target information in this experiment.

In the case of a smartphone, we measured the time it took to remove the smartphone from a participant’s pocket, release the lock, and view the latest Tweet by launching the Twitter application. Alternatively, in the case of Ambient Bot, a participant wears an HMD and uses the Ambient Bot with no virtual creature in sight. Next, we measured the time required to search a target creature, to make eye contact with that creature and to view the latest Tweet through an information window. As a result, all participants accessed target information faster when they used the Ambient Bot (Fig. 6).

Fig. 6.

Required time for accessing information

One of the advantages of the Ambient Bot system is its simple interaction design. It enables users to access information just by making eye contact with a creature without ever having to take their smartphones out of their pockets, much less unlock the smartphones. The results of our experiment suggest that this advantage is maintained even if the interaction is changed to choosing a target creature from multiple creatures.

Figure 7 shows the results from a questionnaire about the design of the virtual creature. The creatures made the participants feel a sense of incongruity; in particular, several participants felt as if they were watched by the creatures. Many participants felt their presence, but easily came to accept the creature. This was likely the case as such virtual creatures are ubiquitous in daily life. In particular, these creatures have become popular in many fictional animations and video games for young people. Some participants did not feel as if they were being watched by these creatures, simply because their appearance was not sufficiently realistic and was easily perceived as a virtual object. However, when the creatures looked human, participants had unpleasant feelings. It is worth noting that our virtual creatures did not affect the reliability of the information. Many participants felt that the information from Ambient Bot was just as reliable as popular news sources because the articles presented by Ambient Bot used a formal literary style typical of news sources.

Fig. 7.

Questionnaire about virtual creature design

Figure 8 shows answers to the question “Which one do you prefer: a creature or an icon as Ambient Bot object?” From this we found that in general, showing a creature is better than showing an icon. Some participants said, “This is because it is not so strange for creatures to float and move around a person. I can empathize with these creatures, as virtual creatures are familiar to me due to fictional animations and video games.” In particular, one participant said, “It is strange that an icon delivers information to me directly because I think of the icon as little more than an application launcher.”

Fig. 8.

Which do you prefer: a creature or an icon as Ambient Bot object?

On the other hand, all participants who chose an icon claimed, “The icon enables me to identify associated information quickly as it is otherwise hard to remember which information is associated with which creature”. Even if participants said that a virtual creature was better than an icon, 8 of 10 participants associated a blue jellyfish with Twitter information because Twitter’s application icon is blue. We assume that one’s impression of content sources is strongly related to the icon images. Therefore, we have to evaluate the virtual creature design associated with an application icon, like a creature with a weather icon on its body.

5.2 Personalization

Figure 9 shows the results of a questionnaire about the use of personalization in Ambient Bot. Approximately 80% of the answers to the question, “Does personalization make it easy to access the information you want?” were positive. Personalization in Ambient Bot can compensate for the pitfalls of a gaze modality that users cannot use and that complicates inputs.

Fig. 9.

Questionnaire about personalization

Half the participants thought personalization was not really troublesome. Many of them said, “This is because once I configured Ambient Bot at the beginning, I did not have to change the configuration every day.” On the other hand, some participants wanted additional personalization, such as a default waiting position for the virtual creatures, or the visual equivalent of an information window.

Table 1 summarizes the participants’ preferred personalization, the kind of information they liked to receive with respect to the time they liked to receive it. All 10 participants configured “today’s weather” for when they were waking up in the morning, while the wake-up time was nonetheless different from person to person. In addition, more than half of the participants configured “tomorrow’s weather” at night. The participants usually wanted to know tomorrow’s weather so they could decide what clothes to wear and whether they needed to bring an umbrella or not. Many participants configured common news for the commuting hours in the morning. Together with the wake-up time, 7 participants wanted to check the news in the morning hours. Thus, there were several common options that many people wanted to configure. Ambient Bot can be programmed in advance with these configurations as defaults.

Table 1. Time-content personalization

Figure 10 shows participants’ final decisions with respect to three situations based on their decisions regarding configurations that either displayed an article within an information window or read the article using a voice synthesizer. In the first situation, when working at the desk, nine out of ten participants displayed an information window, and whether or not they required that information to be voiced depended on the participants involved. Participants who rejected using the voice during work read only the critical part of the article, and then felt annoyed at listening to the entire article. On the other hand, a participant who wanted to listen to the article via voice said, “I was less burdened when I used the voice assistance; rather than reading the whole article by myself, I could concentrate on my work”.

Fig. 10.

Appropriate methods of information delivery

In the second situation, when sitting in a train, all participants wanted to display an information window, yet their opinions about using the voice were divided. Participants who refused to use voice noted that, “The voice conflicts with in-train broadcasting,” while participants who wanted to use the voice noted that, “The broadcast information on a train is not necessary for me to listen to.” Unlike music, the reading of an article in a voice, seriously conflicts with in-train broadcasting. Using voice information in Ambient Bot may disturb those who normally use in-train broadcasting to provide their current location. On the other hand, for most people, visual information on a train is unnecessary and displaying an information window did not disturb their activities on the train.

In the third situation, when walking outside, many participants refused to display an information window. To ensure people’s safety while walking, they needed to carefully distinguish the necessary information from within their eye-sight. As a result, the presentation of an information window, disturbing their eye-sight, should be avoided.

In these ways, the degree to which attention needs to be paid through ones’ eyes and ears depends on the different situations confronted. Thus, in an effort not to disturb people’s daily activities, the method of delivering information should be changed according to the situation at hand.

6 Related Work

Augmented reality technologies allow virtual information to be represented in the real world [3]. Recently, the introduction of Google Glass⁸ and Microsoft HoloLens⁹ has made the technologies closer to our daily life; the future progress of the contact-lens displays will also enable people to use augmented reality technologies anytime and anywhere. The progress will offer promising possibilities to change our daily lifestyle significantly because the technologies allow people to receive information without extra time and effort, unlike the current method of receiving the information through smart phones that need to be held and watched in order to receive the information.

HoloLens, Microsoft’s HMD, has recently attracted people to develop new services in a variety of fields. Microsoft HoloLens is able to deliver a mixed reality (MR) user experience [1], allowing people to interact with virtual objects and entities within real-world settings. MR enables designers to develop new types of advanced services that incorporate virtuality into the real world. The software platform that accompanies the Microsoft HoloLens hardware makes it easy to develop MR applications without requiring advanced skills. Various visions of possible novel services have already been presented. Incorporating virtuality in the real space offers opportunities for developing new types of advanced services [9, 14]. Practical examples have been shown in various fields such as education¹⁰, games¹¹, medical care¹², the space industry¹³ and the manufacturing industry¹⁴.

In [2], Welbo is an AR-based agent system similar to Ambient Bot, where a virtual agent appears in the real world, and the agent can talk to a user. However, in Welbo, a user needs to speak to the virtual agent for interacting with the agent. The approach is heavy weighted because speaking requires more human consciousness than eye contact that is adopted in Ambient Bot. Also, natural and intimate creatures like a jellyfish are more ambient than artificial agents used in Welbo.

In past research, AwareMirror was proposed as a technology-enhanced mirror that ambiently presents information to a user who is washing his/her teeth in front of the mirror [4]. The concept named Pervasive Ambient Mirrors enhanced AwareMirror to reflect people’s current situation to influence their behavior [10]. MIT Media Lab proposed the tangible bit concept that allows a user to interact with ambient information by touching the information directly [8].

Recently, in particular, for Japanese young adults, the boundary between fictionality and their usual daily real life becomes more and more ambiguous [9, 14]. In their pop culture based lifestyles, the fictionality is already becoming alternative reality and they like to enjoy the social relationship with virtual creatures in the hybrid world [12, 13]. For example, a jellyfish is natural and intimate for them because it is a popular agent in one of the most popular Japanese animations that represent our near future high-tech society. Then, technologies used in animations and video games are plausible for them even if they appear in our present daily life.

7 Conclusion

In this paper, we described Ambient Bot, which is an ambient notification system that uses augmented reality technologies to offer everyday information using casual interactions. This paper described the design and prototype system implementation of Ambient Bot, and it also described some insights that emerged from a user study.

Of course, the current HMDs are not comfortable enough for accessing daily information in an anytime, anywhere fashion; however, in the future, HMDs will become smaller and more lightweight [7]. Consequently, every person may eventually wear an HMD in daily life. Therefore, our approach will ultimately become practical for accessing various everyday information, superseding even smartphones.