1 Introduction

Nowadays, the user experience study of autonomous car is getting more and more attention. As seen in auto shows, there are lots of HMI concepts describing the outlook of future autonomous car interaction systems. Nissan PIVO uses a physical robot as interaction system. Eleven years later, in their IDS concept car, we can see the same concept with a robot on a movable screen. Daimler uses holographic display in Vision Tokyo concept car. BMW uses HUD display, LED ambient light, at least 9 interactive devices and so on in their HMI system. Not only in concept cars, we can also find the big change of design in production car like Tesla model 3 which only has screen, steering wheel, screen, brake, and accelerator. The HMI concept and study of autonomous car are widely divergent. The interaction of autonomous car today is much like the “faster horse” which people want a century ago. There are enough technology and ideas about the future autonomous cars interaction. So we hold the opinion that the insight of user need and design process is more necessary now.

Highly automated system like autonomous car has a concomitant rise in the breadth and complexity of interaction. Thus, we need a clear, consistent, logical, and holistic design method to design and analyze human-vehicle interaction environment [3]. As most of the product focus on human-centered design process, researchers always involve users in participatory design. But in our study, we found that it is hard for users to put forward ideas in a brand new product which they never experience before. So, we want to propose an innovative method about need insight and interaction design for highly automated human-vehicle interaction system, then design a HMI system using our method and compared our system with another one using normal design method to evaluate our method.

2 Related Works

The work of Sven Krome introduces a context-based design process and a method called “car storm” to provide unique experience for the autonomous car passengers [4]. Bo Zhou utilizes a four steps design process based on service system design method to define the Service-Defined Intelligent Vehicle [5]. Ingrid studies the design techniques for exploring automotive interaction and discuss unmet needs in interaction design for the future [6]. In another area of research, some low cost experimental methods to simulated unmanned vehicle environment are studied due to the difficulty of building an autonomous car. Raphael uses a scale model to do the user test [7]. Tom uses “The Wizard of Oz” [8]. We use a real autonomous test car provided by an enterprise to do the on-road experiment. Our main contribution is that we introduce a design process to insight interaction needs and design HMI system for the autonomous car.

3 The Interaction Model and Definition of Self-driving Car

3.1 The Context of Self-driving Car

The U.S. Department of Transportation’s National Highway Traffic Safety Administration (NHTSA) has created the most common definition breaking down autonomous vehicles into 5 categories, which shows the deployment path from none-automation to fully self-driving automation. Since driving task still exists in level 0–3, all the interaction have to be designed as a secondary task [9]. We want to set a free environment that makes our passengers get rid of driving. So we choose level 4 full self-driving automation as the target we design for.

3.2 The Model of Self-driving Car Based on Intelligent Agent

As described in Patrick A.M. Ehlert’s work, self-driving car can be defined as an intelligent agent. An intelligent agent is an autonomous, computerized entity capable of sensing environment and acting intelligently based on its perception [10]. Obviously, compared with normal interactive system, to interact with an intelligent agent can result in a more complex interface and task. To be specific, we are facing inaccessible, non-deterministic, non-episodic, dynamic and continuous environment [11].

Our autonomous car can be defined as an intelligent agent through comparison with the model described in Nilsson’s “principles of artificial intelligence” [12]. Based the definition in Nilsson’s book people should also be seen as an intelligent agent. So in our design process, we do not only think about the users’ needs but also find the “car needs”. For example, to recognize the user is one of the car needs.

3.3 The Interaction Model of Autonomous Car: Application Scenario

We need to define an interaction between people and vehicle. The human-vehicle interaction is transforming from an interface for controlling a machine to an interface helping people to communicate with information, intelligent agent and other people [13]. To find a universal model in our system, we need to describe the relationship between people, vehicle, technologies, user needs and so on. With reference to the PACT model put forward by David Benyon, we propose an application scenario used to describe the interaction model between intelligent agents.

As shown in Fig. 1 we add an agent to the model to make up the new scenario. That is because, in the old model, the car should be part of the technologies. But some scenario doesn’t contain people. For example, the Summon function updated in Tesla v7.1 can pick you up wherever you want by the car itself. Autonomous car should be the same as people in the system. Both people and car are agents. They use technologies to act in context. So it should be concluded as a PCACT model as the figure shows. In this model, interface exists between people and technology, people and car, people-car-technology system and activity. Furthermore, there are three different interfaces between people-car-technology system and activity. They are P-T-A, C-T-A, P-C-T-A. What we need to do next is to design interaction system for the 5 different kinds of interface.

Fig. 1.
figure 1

PACT model (left) and PCACT model (right)

Full size image

4 Interaction Needs Insight and Design Method

4.1 Principle

First of all, we need to define interaction design for autonomous cars. In our opinion, it is to find application scenario, which can prove the technology is useful through the understanding of human need and activities. Finally, we can generate requirement of technology [14]. Thus, we will introduce a design process from need insight to design tool and find the technology needs finally. That will help us to translate users’ needs to technical demands.

Design should focus on human needs. There are serval methods to insight the human needs such as user interview, brainstorm, and concepts from other products and so on. But, there will be difference of needs in terms of description and integrity. Therefore, in order to standardize the requirements, we find with different methods. We use the interactive model described in this paper as a script, by supplementing the necessary content in the script, to obtain a large number of application scenario list from different sources but have the same format. The format is the PVACT model we describe before.

In our study, we conducted user interview, brainstorming and network survey to collect application scenarios with details filled and similar ones merged. Finally, we obtained 153 scenarios. There are 21 techniques satisfying the requirements in the 153 scenarios. Obviously, if we put all of the 21 techniques in one product, it will result in a bad user experience because of high complexity and poor efficiency although it can meet all the user needs. Thus, we need a technology convergence to satisfy the needs selectively.

It should be noted that keeping an open mind when filling PCACT model helps to find more needs.

4.2 Need-Technology Matrix

We collected 16 classes of people, 9 classes of autonomous vehicles, 21 classes of technologies, 153 classes of activities and 9 classes of contexts. In order to assist decision making, it is better to show these elements in a visual way. We can fill them in the two-dimension matrix. In general, in the early phase, different classes of people and vehicles do not need to be seen as variables in the matrix (Fig. 2).

Fig. 2.
figure 2

PCACT matrix

Full size image

So we can show all the PCACT we found in the two-dimension matrix as the picture show. But it is a concept model, the real matrix in our study should have more information.

The matrix shows us the relationship between the technologies and user needs clearly. Based on this matrix and the needs finding before, we change from design-driven to technology-driven in this phase. Then, we need to analysis the matrix to obtain technology requirements.

4.3 Design with the Matrix

The design process can be summarized in the following steps

  • Step 1: To Filter High Frequency Ratio Technologies.

    It is easy to find that some technologies can be chosen for multiple times in one raw compared with others. Some technologies can only fit for 1, 2 requirements. To make a universal and cost-effective system, we should choose high demanded technology and just site the low demand technology aside.

  • Step 2: To Identify Special Requirements

    After the first requirement screening, we choose the technology which can fit for user needs as much as possible. But the needs can’t be satisfied due to technical reduction maybe are important for users. So we should double check the low demand technologies in case of missing some key application scenario. For example, the accelerometer in your smart phone is a high demand technology. You can use it in serval different application scenarios such as Game, human motion data acquisition, auto rotate screen and so on. Fingerprint identification looks like a low demand technology in the matrix which only have two scenarios. But fingerprint unlocks and pay are high frequency scenarios. So Finger print identification is also a high demand technology. In a word, there are two indicators needed to be considered to calculate the demand for technologies, one is the frequency at which the scenarios are triggered, the other one is the number of scenarios.

  • Step 3: To Optimize Technical Path

    After the two focus step, we can get a rough HMI layout and technology list. Next step, we should optimize the technical path. We can find serval different paths to satisfy the application scenario although using the same technologies. What we need is to find the best path. Then, to optimize the best expressing method. For example, when you want to communicate system status to your user. You have at least two expressing methods. One is system centric expressing like “Reliability 64%”. Another is human centric expressing like “Engagement 36%” [16]. The expressing method decides the ease and experience of use.

  • Step 4: To Define Your Own Product

    We have already translated the user needs to technical demands. The final step is to define the product with your own needs, commercial needs, branding needs and so on. We can combine the technologies to create a product. Ideally, there is no limit to what product it is. In our study, we do not have to add all of the interactive devices on the car. We can also design a mobile device using the same technologies to make a better in-vehicle experience. In one sentences, the combination of technologies is product design.

5 Result

5.1 Result

Finally, we solicited 4 from the 21 technologies we collected to form the whole vehicle HMI system. We hope to use a minimum of technologies to achieve the greatest effects. To ensure the integrity of the system, there should be at least one input and one output device. For a better user experience and richer application scenarios, we need to combine soft and hard interaction together [15]. So we need to make sure that input and output system have both implicit and explicit interaction devices. Depending on the ranking of technologies demands, we choose the top one to fill in each quadrant. And we get the HMI technology matrix as the Fig. 3 shows.

Fig. 3.
figure 3

HMI technology matrix and result 1

Full size image
Fig. 4.
figure 4

1:1 concept car

Full size image

Based on the matrix, we redesign the HMI layout, as shown in Fig. 5. This system has two inputs and two outputs. The eye-gaze can be used to monitor user status, identify users and so on. Speech control can input the users’ command. And the status monitored by eye-gaze can help the system understand the users’ voice command better. We cite an instance when user tells the car to stop there the system will park in the area where the user stares at. The output devices consist of 3D sound cue and ambient LED light. They can provide spatial light, 3D VUI, 3D earcon and so on as the output. The light and sound combine together via the space position. Cite an instance, we can set up the voice and LED move together from back to front to express speed up. Based on the HMI layout we designed with our process, we make a 1:1 concept car to test our HMI system (Fig. 4).

Fig. 5.
figure 5

Result 2

Full size image

5.2 Result Designed with General Method

In order to evaluate how the design method impacts on output of design result in this paper more intuitively, we design another HMI system using the normal design method to compare with result 1. Method 2 is an on-road test. The user is told before test that the car would be driving itself and we have an official security to protect passengers. We encourage the user to think aloud. The users can express their dissatisfaction, creativity, improvement suggestions and so on. Then, all the ideas we collected from the users can be used in the focus group. Finally, we received 89 ideas about the interaction design. After evaluated by users, we designed a HMI layout as shown.

In consideration of the limit of in-vehicle time, users prefer to use their own devices to reduce learning cost. But they still need a larger screen as an extend screen to get a better visual effect. So we will allow user to connect their smart devices with our car system to adapt to our system quickly.