Keywords

1 Introduction

1.1 Motivation

In-vehicle emotion detection and mitigation have become an emerging and important branch of research within the automotive domain. Different emotional states can greatly influence human driving performance and user experience (UX) today in manual driving, but still in future automated driving conditions. The sensing and acting upon relevant emotional states is therefore important to avoid critical driving scenarios with the human driver being in charge, and to ensure comfort, acceptance, and to enrich the user experience in automated driving. With the evolving development of progressively automated vehicles, i.e., high and full automation respectively SAE levels 4 and 5, the role of a nowadays car driver executing the driving task or at least monitoring the automation and simultaneously the driving environment is turning gradually into the future role of a mere user or passenger of the automated system. Prospectively, a car driver does not even have to serve as a fallback instance in cases of automated system failure. Therefore, the current primary task of self-driving will become increasingly irrelevant and the nowadays secondary tasks are turning into prospective primary tasks the user will be allowed to exclusively focus on. This raises questions of changing interplays in human-vehicle interaction resulting in prospectively new affective use cases and hence the needs for different user-centered empathic Human-Machine Interfaces (HMI). Interestingly, researchers and experts in HMI research expect that technical systems in general and vehicles in particular will be better able to react to human needs when being equipped with emotion recognition capabilities [1, 2]. This is the motivation of this paper and the presented research. Hence, we will focus on the role of the user’s affective state for user experience and user acceptance in the context of highly automated driving. Within this context, the central goal will be to reveal and discuss crucial use cases for empathic automated cars thereby providing a roadmap of future research and applied issues of designing user-centered empathic vehicles for future mobility.

1.2 Empathic Automated Vehicles

According to the scientific literature, the term empathy refers to the capability or disposition to share and understand the other person’s internal world including thoughts and feelings [3]. Generally, research has differentiated between the ability to understand another person’s perspective, i.e., cognitive empathy, and to feel what the other person is feeling, i.e., affective or emotional empathy [4]. This differentiation is very important not only when studying empathy in humans, but also from a technical system design perspective, because, given the current state of the art in artificial intelligence, it seems unlikely that technical system will soon be capable of affective empathy [5]. To add, technical systems do not require affective empathy to be of help for humans, because cognitive empathy, i.e., the capability to understand the user, can already enable systems to provide users with tailored, situation specific support [5]. According to Stephan [5], technical systems do not even need empathy in every sense of a full understanding of the emotional spectrum of the user, but in some cases already the understanding of a limited set of user states can be sufficient to be helpful. Following these considerations, we here conceptualize empathic vehicles as vehicles that have the ability to understand a user state of interest and the possibility to react to this user state tailored to the user given a set of interaction strategies.

Because emotions are supposed to have a critical influence on driving behavior, use cases for empathic vehicles have been widely discussed for manual driving [6,7,8,9]. In contrast, their full potential is not straight-forward during highly automated driving when the driver is not in control anymore (level 4 and 5 according to SAE international [10]). Interestingly, some recent projects consider higher automation levels, but these projects concentrate either on a very limited set of clearly defined use cases [11,12,13,14], or they rather discuss a general agenda towards the development of empathic vehicles without defining the exact use cases.

Therefore, the aim of this paper is to generate a set of affective use cases and ideas that can guide research activities towards designing empathic fully automated vehicles in SAE Level 5.

2 Method

2.1 Participants

An expert group workshop was conducted as part of the pre-conference workshops of the ‘Mensch und Computer’ Conference 2019 in Hamburg, Germany (MuC’19 – the largest annual human-technology interaction conference in the German language area) [12]. The expert workshop aimed at Human-Machine (HMI) and Human-Computer Interaction (HCI) researchers as well as at user experience (UX) and usability practitioners, designers, and developers in the field of automotive HMI interested in designing empathic cars within highly automated driving (SAE 5). Nine (six female) international participants visited this expert workshop. They varied widely in age (25–41 years; M = 30.22, SD = 5.40) as well as vocational experience in the field of usability and User Experience (1–9 years; M = 2.28, SD = 2.94). The participants brought with them a diverse academic background: three came from computer science, three had a degree in psychology, one participant had studied graphic design another one interaction design and one was a graduate student of usability engineering.

2.2 Procedure

Throughout the workshop we followed a design thinking and building ideas procedure and used participatory ideation methods [15]. The procedure of the workshop can be separated into four phases (see Fig. 2): (1) getting to know each other, (2) getting into the right mood, (3) generating ideas, and (4) integration of results.

The workshop started with a brief round of introduction of all participants (1), followed by a warm-up exercise that helped participants to ‘warm-up’ with each other so that they can better start working together (transition from (1) to (2)). This is helpful for the participants to overcome prejudices and (before things get serious) to interact with each other. This was accomplished using the Lego® Serious Play® methodology [16]. In this way, we were able to gain a first reflective and evaluative confrontation with one’s own view on the topic of self-driving cars. Furthermore, it is a methodological way to put the context in relation to the topic, while the task for the participants can be kept very open. On the one hand, the use of Lego® bricks facilitates visualization for the participants and on the other hand, the presentation in the form of metaphors also helps them to deal with the topic on a deeper level. When building with Lego® bricks, participants literally think with their hands, since sensory stimulation of the hands also triggers certain thinking processes in the brain [16, 17]. It should be pointed out when presenting each individual Lego®−model, it should be ensured that each brick used is briefly explained - what it stands for and what the participant was thinking. Exemplary impressions of the Lego® Serious Play® methodology can be found in Fig. 1.

Fig. 1.
figure 1

Participants prototyping future automated empathic cars to get into the topic. Images courtesy of Mensch & Computer 2019.

Full size image

After that, the research area was introduced with definitions of automated driving, emotions, and possibilities of in-car emotion recognition as well as car manufacturers’ ideas of empathic cars (feeding (2)). In this block, challenges and boundaries of cutting-edge research methods from ongoing projects and results in the field of emotion detection and regulation within affective automotive HMI were presented. This way, participants would go about the workshop tasks with a common understanding for the topic and state of the art.

To start the idea generation phase (3), participants further learned about the basic principles of design thinking and the particular methods which would be used in the following ideation session. Because investigating future scenarios and technologies is not straightforward [18], we introduced the ‘25th Hour’ project by Audi [19] as a contextual base for general guidance throughout the creative work. This project is a nice illustration of people’s visions of future automated vehicles look like and captures possible activities passengers might want to engage in during an automated vehicle ride (for other studies, see [18, 20]. Following the insight that future vehicle interiors should be designed for users’ needs, the ‘25th Hour’ project [19] investigated how future self-driving cars will change our everyday lives and activities while spending our time in a self-driving car. In this project, among others, three time modes in the sense of overarching use themes were defined that are conceivable in future cars of SAE level 5 automation: quality time, productive time, and down time (or time for regeneration), which were seen as the framework for the use cases for the group work in our workshop. Here, participants worked together in small groups to come up with prospective use cases for affective user interfaces in the age of automated driving. Participants were randomly assigned to three groups of three expert members and one workshop instructor each. Each group was tasked to develop use cases for empathic automated vehicles in SAE level 5 within the context of only one of the aforementioned use themes (down time, productive time, quality time).

Fig. 2.
figure 2

Sketch of the expert workshop’s procedure. The goals of the phases are shown on the left. The time for each of the phases is presented on the right.

Full size image

In each group, an initial brainstorming was realized using the speed ideation method, in which participants are urged to generate a bulk of divergent ideas in indirect collaboration. Each person records their own ideas on sticky notes and after a few minutes, when everybody has a set of ideas in front of them, they move seats like in a speed dating scenario. After that, they iterate on the ideas on their new seat, before rotating again and again until they are back in front of their initial ideas, which now have been annotated with improvements and new spins. An ensuing group discussion was used to distill this flood of ideas into a list of the five ideas with the most benefit for daily life for each scenario. The groups showcased their findings on a poster and built a paper prototype for their single top idea, on which they had to elaborate further. Then, each group had to present its ideas to the plenary.

In order to integrate the results (4), the instructors summarized the main findings of the workshop and discussed open questions in the plenary. In this phase also an interesting discussion about cultural and ethical issues of empathic systems came up.

3 Results

In the following, the use cases that were generated in the course of the workshop for the three time modes, i.e., use themes, are described. An overview over the main results can be found in Table 1.

Table 1. Affective use cases for empathic vehicles produced by the groups.
Full size table

3.1 Down Time

The first group, dealing with the ‘down time’ scenario, identified the use case of a bored passenger as the most important. An empathic car would then notice the passenger’s boredom and provide entertainment in order to change the feeling into a more lightened mood (for an illustration, see Figure 3). Entertainment could come in the form of suggestions for movies, games, audio books, or phone calls with available friends. Even talking with the vehicle itself could be an option – provided the vehicle’s AI allows for engaging conversations.

Fig. 3.
figure 3

Illustration of ‘down time’

Full size image

Other use cases produced by this group were: A sad passenger, in which case the car would offer to phone a close friend or family member in order to cheer up the user. Other than changing the passenger’s negative feelings, the vehicle could also support positive ones. A use case presented for this was the creation of a peaceful ambience by providing sensations associated with peaceful landscapes like the sound of waves combined with a slight smell of salt or the sound of wind coupled with a smell of pine trees. Another one providing relaxation via massage knobs built into the seat, support by playing meditations possibly adjusted to the current season. An example mentioned was warm light, jingly music and cinnamon smell during Christmas season. Finally, the car could just allow the passenger to sleep.

A last use case provided by group 1 was that of a passenger who feels embarrassed and wants to use the car as a safe space or as some kind of a shelter. In this case the vehicle could darken the windows and tone down the light for the passenger.

3.2 Productive Time

The second group in the workshop dealt with application for a time when the vehicle’s user wants to be productive. The use case that was identified by this group to be the most important was the enhancement of the user’s ability to concentrate. If the vehicle identifies the passenger’s concentrated mood it would react by adjusting the light to be more suitable for working, cancelling outside noise, and increasing the windows opacity. Further, the vehicle would automatically connect to relevant devices like the laptop and cell phone and suppress apps that might be distracting. Also, the route would be planned in such way that cell phone coverage is ensured at all the times. Lastly, the car would analyze whether the user is getting tired and offer short breaks approximately every 50 min (see illustration in Fig. 4).

Fig. 4.
figure 4

Illustration of ‘productive time’

Full size image

The second use case presented by this group envisioned the vehicle as an office for conference calls and was entitled ‘passionate’. The vehicle would than automatically phone all the scheduled participants and provide a shared working space. It could even project virtual avatars of the other conference participants into the vehicle’s interior. Further a ‘motivated’ use case was proposed in which the vehicle provides an opportunity to compare your work with friends and colleagues in a gamified way awarding level-ups for vocational achievements. Moreover, a ‘feeling supported’ use case was suggested in which the vehicle provides necessary resources like information, software and online libraries automatically. For breaks a ‘bored’ uses case was proposed as well, which is rather identical to the one presented by group 1.

Additionally, to these uses cases an ‘angry’ use case in which the vehicle, when it detects the user getting irate, would aim for a calmer more passive driving style, adjust the lighting and suppress non-priority mails. Finally, a use case was presented in which the vehicle would provide happy stimuli, like the user’s favorite playlist, to support the user’s productiveness.

3.3 Quality Time

The third group dealt with the scenario of ‘quality time’ (illustrated in Fig. 5). The use case this group identified as most important was the synchronous consumption of media with passengers in different vehicles. The vehicle would suggest contacting people who are also driving in an empathic vehicle at the moment with whom the user could watch movies and TV shows together. Which movies are suggested for watching could be based on the viewing history of the users, but also on the conditions. For example, if the vehicle is en route to Paris, the empathic vehicle could propose a movie centered on Paris.

Fig. 5.
figure 5

Illustration of ‘quality time’

Full size image

In another use case, it was suggested that the vehicle could moderate games that a family travelling together could play. ‘I spy with my little eye’ was mentioned as an example. Further the vehicle could play music and invite the family to sing together, with the music suggested fitting to the potential destinations. If the vehicle realizes that the passenger is sad it could offer to contact family members or friends by phone. For this case, it was also proposed to let the vehicle offer to drive a different route, e.g., going to the beach instead of the original destination.

Lastly a ‘worried’ use case was suggested by this group in which the vehicle would connect to the security systems of the user’s house letting him or her check back whether everything is alright at home to get rid of these worries.

4 Discussion

4.1 Summary of Results

In this paper we presented use cases for empathic automated vehicles. These use cases were generated in an expert workshop, in which participants first had to identify relevant emotions user of autonomous vehicles are likely to experience during different overarching use themes (productive time, down time, quality time). Based on these, potential adaptation strategies tailored to the emotions were ideated and elaborated in group work. In spite of the different themes, the groups were looking into overlapping affective use cases that emerged during the session and that were deemed applicable for each of the scenarios. For example, a use case similar to the ‘bored’ use case for down time was described in every group, i.e., for every overarching use theme. As a bigger pattern, it could be observed that the participants tended to name solutions counteracting negative emotions rather than fostering positive ones. The mitigation of negative emotional states and disturbances could be seen as the common ground of most suggested affective use cases. Not surprisingly, many concrete mitigation strategies aimed at distracting the user from his negative emotional state, e.g., by suggesting him or her to watch a movie. Interestingly, this focus on the recognition and mitigation of negative emotions can also be seen in several ongoing research projects dealing with frustration, stress, uncertainty or sadness [6, 11,12,13]. However, participants also brought up use cases, in which the empathic vehicle suggests entertainment fitting to the mood and preference of the user. This may seem like a logical step considering that internet platforms, like Youtube and Netflix, already make use of algorithms to suggest videos tailored to the preference of individual users.

4.2 Technical Considerations on Empathic Vehicles

At the technical basis, an empathic vehicle needs a comprehensive system architecture. Such an architecture is developed in the project ‘AutoAkzept’ [11]. It consists of several functional modules beyond user models for emotion recognition, such as a context model, and an integrated situation model, a user preference model and a module selecting the best adaptation in the given situation. This comprehensive architecture is necessary because not only does the empathic vehicle need to recognize an emotion, it also needs to understand the context and integrate user state and context into a situation. Only if the empathic vehicle is able to understand the whole situation, it can recognize the human needs and suggest the best interaction or adaption to meet that need. This is true because emotions are elicited by stimuli, e.g., an event or a thought, and the facial expressions displaying an emotion (as one way to express/detect emotions) are often ambiguous and even human observers use context to disambiguate emotions [21].

4.3 Challenges and Future Perspectives

The results of this workshop provide a very first glimpse into the possibilities that affective technologies may provide for interaction in automated vehicles. In order to advance this field towards feasible implementations accepted by users, the initial expert input needs to be enhanced applying a user-centered design process. Related work by Li et al. [22] provides an approach for iteratively generating use cases in user workshops and car-storming sessions which could be applied to our initial feature ideas. They also make a point for more inclusive design approaches, for example, by reaching out to users from different social and cultural backgrounds. In fact, an expert workshop as reported in this work is a reasonable step towards the exploration of new research fields; in the long run, however, requirements for affective innovations need to be derived from real user needs. User acceptance of and user needs during the interaction with affective technology such as empathic vehicles are likely to differ strongly based on cultural upbringing. The context culture theory by Hall [23] suggests a major contrast between eastern and western cultures in the way people from these cultures communicate information, either with low context and based on the very meaning of said words, or high context, meaning more implicitly through expressions of emotions. Thus, especially emotional interaction in terms of detection and mitigation might be susceptible to cultural influences. Apart from communication, also other factors like disparate design preferences, general taste and legislation call for a culturally sensitive design approach [24].

Acceptance of empathic vehicles may not be influenced by cultural differences only. Even within the same cultural context different individuals not only express emotions differently, posing a challenge for the recognition of the user’s emotion, but different individuals also have different preferences of mitigation and these may even change over time, posing a challenge to select the most suitable interaction for a given user at a given time. For example, in a sad state some individuals may prefer to keep these to themselves, whereas others will cheer up when the vehicle suggests to call some friends or family (see suggested intervention in use case 2 in the ‘down time group’ and use case 4 in the ‘quality time group’). So a vehicle suggesting calling a friend to someone who has just lost a friend is likely to make the situation even worse. This rather dramatic example illustrates the fine line between helpfulness and harmfulness of empathic vehicles or technology more generally. To be on the helpful side, the system needs to have the capacity to learn individual user preferences and needs to be able to apply them.

A whole bouquet of questions concerning privacy presents itself when more than one person is riding in the vehicle. If the passengers have different moods and needs or even just different preferences, who should the car adapt to? What happens to the person whose mood and needs are not catered to? Another unsolved question that arises when several people share the ride concerns socially unaccepted emotions such as frustration. Will users accept frustration reduction measures suggested by the vehicle when another person can watch and should they even be suggested?

So in a nutshell, we have to conclude that this attempt and journey to address the user-centered question of exploring affective use cases in highly automated driving has just begun.