Extracting intermediate-level design knowledge for speculating digital–physical hybrid alternate reality experiences

Abstract

This paper reports a process to derive intermediate-level knowledge as a service design and analysis framework for designing digital services to offer alternate reality experiences, and analyzes the possible opportunities and pitfalls of the framework. The user experience felt by refining the meaning of real space through virtuality is defined as alternate reality experiences. Alternate reality experiences are typically achieved by modifying our eyesight or replacing our five senses to others, and they make our world interactive by implicitly influencing human attitudes and behaviors. First, the paper extracts observations for deriving the intermediate-level knowledge through the discussions raised in exploration workshops. In the workshops, the three digital services that utilize diverse strategies to offer alternate reality experiences are chosen. The workshops’ main focus is to examine how a person could have a sense of values in alternate reality experiences via the three digital services. Second, the paper shows how to derive the proposed service design and analysis framework from the extracted observations through expert analysis, then an overview of the framework is explained. Finally, the paper presents feasibility analysis of the proposed framework through a new digital service named Mindful Reminder as a case study for refining the service through focus group discussions. The approach described in the paper is to report a concrete process through which extracted observations can be converted into intermediate-level knowledge that can be used to design alternate reality experiences. Traditionally, the process for generating intermediate-level knowledge has not been well-documented; however, documenting the process is very important in theorizing the design of alternate reality experiences and helps effectively develop a variety of emerging advanced digital services that will offer alternate reality experiences in the future.

Appendices

Appendix 1: Case study demonstration sequence

In the experiment to use a prototype system of Mindful Reminder, its participant wears an HMD and continuously watches two movies: one shows the scene that the participant is currently viewing, and the other is recorded in advance in the same location. However, the scenes in the second movie seem like they are occurring in front of the participant. Therefore, the participant is unable to differentiate the scenes in the second movie from the scenes currently playing out in the real world.

In the experiment, the participant is asked to stay in the room and to wear an HMD (see Fig. 5). Then, the experimenter leaves the room, and the participant, who is alone in the room, watches the movies through the HMD. The second movie was recorded by a camera attached to the HMD (see Fig. 5a); therefore, the participant sees real-world scenes recorded in advance through the HMD. In the demonstration, a notification card appears in the top left-hand corner of the HMD (see Fig. 5b), and the participant is notified that the demonstration will be completed soon and is asked to wait for a moment. Then, the experimenter speaks to the participant and asks him/her to look at the experimenter and to remove the HMD (see Fig. 5c). After taking off the HMD, no one is actually there in front of the participant. Because the experimenter speaks to the participant in pre-recorded scenes, the participant believes that the experimenter is actually there, although the scenes do not currently occur in the real world.

This demonstration allows the participant to actually experience the different effects of the system notification in the form of a card appearing in the top left-hand corner and the notification from a human appearing in front of the participant and declaring that the experiment of Mindful Reminder is over.

Appendix 2: A detailed sequence of a user study

Introduction:

The moderator welcomed all the participants to the focus group discussions. The participants and the moderator were then seated in chairs in a semi-circle.

Demonstration:

The moderator presented a prototype system of Mindful Reminder so that the participants well understood the behavior and functionalities of Mindful Reminder. The user watched movies through an HMD (see Fig. 5).

Case Study Overview and Scenario Presentation

The moderator, who is one of designers of Mindful Reminder, then presented an overview of Mindful Reminder, particularly how it works and how the technologies were used. The moderator did not show any of her subjective opinions regarding Mindful Reminder and showed the objective merits and shortcomings of Mindful Reminder. The scenario descriptions shown in Section 5.1 were then distributed to the participants. The moderator instructed the participants to read the scenarios, and before they started to read the scenarios, she explained the background of the fictional world behind the scenarios. For each focus group, the participants were encouraged to imagine an ideal world through Mindful Reminder. The scenarios were presented to all the focus groups in the same order.

Scenario Development

After a brief general discussion on the demonstration and the technologies used in Mindful Reminder, the moderator asked participants to develop their own scenarios to use Mindful Reminder. The scenarios depicted a scene in which Mindful Reminder is used based on their experiences with the demonstration of Mindful Reminder. The developed scenarios are shown in Appendix 3. This part lasted approximately 15–20 min.

Framework Explanation:

The moderator presented an overview of the two sub-frameworks presented in Section 5, and provided an example of how to use the sub-frameworks. The example used Enhanced TCG described in Section 3, particularly with regard to how to use values when refining the design. After providing this example, the moderator was asked questions about the framework. This part took approximately 10–15 min.

Scenario Improvement:

The moderator asked the participants to speculate on the potential pitfalls and opportunities of the scenario based on the proposed sub-frameworks. First, they used the first sub-framework to structure their discussions about the potential pitfalls in terms of values in each step, and values were used to frame the discussions in each step. Additionally, they used the second sub-framework to discuss the plausibility of the virtual values in the alternate reality experience. In terms of grappling with magic circles when refining the scenarios, the participants only considered the community magic circle, where stakeholders who were directly related to the scenarios took into account the plausibility of alternate reality experiences in the scenarios. After the discussion, the participants in each focus group were asked to refine the scenario to overcome the extracted pitfalls. Appendix 3 shows the refined scenarios and the values and plausibility in the framework that were discussed. Finally, they discussed the effect of the framework on their analysis. The discussions lasted approximately 45–70 min.

Debriefing:

Following the scenario discussions, the moderator asked the participants to discuss the merits and demerits of using the framework and the research method used in this study when refining the scenarios. These discussions lasted 5 min per person. Finally, the moderator thanked the participants for participating.

Appendix 3: Scenarios developed in foucus groups

1.1 Focus group A

Scenario at the Beginning of the Focus Group Discussion:

“Keisuke is a university student. He took a train to his university. On the train, he observed a person who offered his/her seat to an elderly person. This act of kindness allowed Keisuke to think about himself and to reflect that, if necessary, offering his seat to someone would be desirable.”

In the focus group, the participants initially discussed all the dimensions and their respective values. They then observed that an elderly person may have provided some empathetic value when a seat was offered to him. However, if the elderly person had appeared healthy, he may not have been as empathetic. Therefore, they considered behavioral plausibility to be important when analyzing the plausibility of living things. Table 2 summarizes the frames in the framework that the participants discussed. Additionally, they discussed the landscape dimension. If the train is not crowded, the person may not need to give up his/her seat to others. The focus group participants mainly emphasized the authentic value and the factual plausibility, not the visual and behavioral plausibility. The landscape in the scenario should present a crowded train; therefore, the realistic landscape is not needed.

Table 2 Dimensions, values and plausibility focused on in focus group A

Scenario at the End of the Focus Group Discussion:

“Keisuke is a university student. He took a crowded train to his university. He heard the sound of a cane from a corner near where he was sitting, and he turned to look in that direction. He then observed an elderly person with a cane walking across the train. Because the elderly person was a bit too far away from him, he could not do anything. However, someone closer to the elderly person kindly offered his/her seat to him/her (an occurrence virtually generated by Mindful Reminder). It was a crowded train; therefore, everyone had to be a little irritated. Nevertheless, a kind person offered his/her seat to the elderly person with a cane. Keisuke then thought that if he had the chance in the future, he would like to offer his seat to someone.”

1.2 Focus group B

Scenario at the Beginning of the Focus Group Discussion:

“Ryosuke went shopping at a supermarket. One day, there was a sample-tasting section in the supermarket. Although he wanted to try some of the samples, he had some hesitations about going to the sample-testing section. Then, a woman in the sample-tasting section encouraged him to do so. “Would you such as to try this?” With her encouragement, he tried the samples and thought that trying something without hesitating worthwhile. This occurrence served as a good opportunity for Ryosuke to reconsider to try unfamiliar things by embracing a healthy curiosity about the unknown.”

In the focus group, the participants often discussed the reliability of the woman at the sample tasting. In step 5 of the framework, they decided to use a famous actress in the sample-tasting scenario; in doing so, they aimed to focus on the authentic value by increasing reliability because a famous actress is actually there and talking about the sample tasting. Table 3 summarizes the frames in the framework that the participants discussed. In this scenario, behavioral plausibility is important because the virtualized famous actress appears to be a familiar person.

Table 3 Dimensions, values and plausibility focused on in focus group B

Scenario at the End of the Focus Group Discussion:

“Ryosuke went shopping at a supermarket. One day, the supermarket had a sample-tasting section for nattō, i.e., sticky steamed soybeans. He did not want to eat nattō because he did not think that he liked it, even though he had never tried it. However, when he looked at the section closely, he noticed that an actress famous for her involvement in a TV food show was promoting the consumption of nattō. The actress was saying that nattō was nice and delicious, and the woman in the sample-tasting section had just asked Ryosuke to try nattō. He then tried the nattō just to taste it and found that it was surprisingly much better than expected. As such, he reflected that it was better to try unknown things without negative preconceived notions, and the occurrence served as a good opportunity to remind him about the negative effects of not trying something because of preconceived notions.”

1.3 Focus group C

Scenario at the Beginning of the Focus Group Discussion:

“Yukiko was distressed by serious incident that occurred one day. That night, she was at home reading her friends’ posts on an SNS. The posts were interesting and fun; nevertheless, she suddenly remembered the distressing incident that had occurred earlier in the day and posted her grievances on the SNS. Unexpectedly, a virtualized friend then appeared and said, “Are you okay?”, which was actual content that the friend had posted on the SNS. Yukiko was encouraged by her friend’s unexpected appearance in front of her, and she was able to forget the distressing event for a while.”

In the focus group, the participants were always anxious when the friend appeared in front of them. The user knows the friend very well. If the appearance and behavior of the virtualized friend differ from that of the real friend, the plausibility of the scenario will diminish significantly. Therefore, the virtualized friend should look like the real friend as much as possible. Table 4 summarizes the frames in the framework that the participants discussed. After deciding to focus on the empathetic value of the narrative of the virtualized friend, the participants found that the visual and behavioral plausibility of living things and occurrences are essential to increase the persuasiveness of the narrative.

Table 4 Dimensions, values and plausibility focused on in focus group C

Scenario at the End of the Focus Group Discussion:

“Yukiko was distressed by serious incident that occurred one day. That night, she was at home reading her friends’ posts on an SNS at her home. The posts were interesting and fun; nevertheless, she suddenly remembered the incident that had occurred earlier in the day and posted her grievances on the SNS. Then, Hiroki, who is Yukiko’s best friend, unexpectedly appeared in her room and encouraged her by telling her a short joke. The content was fun, and Hiroki had actually posted it on the SNS. Yukiko had really been encouraged by Hiroki, who actually appeared in front of her, and she was able to forget the distressing situation for a while.”