Keywords

1 Introduction

Recently, the development trend of HMI in vehicles is complex information system that includes car to car, and Car to X [1]. Apart from driving styles, drivers using in-vehicle information systems (IVIS) can also cause distraction [2]. 60% to 70% of the visual attention resources are accounted on the driver’s main driving task in their driving.

With the increase of driver’s workload, the degree of distraction increases, which is one of the causes of accidents. And the drivers’ workload was revealed to increase with task difficulty and it have relation with the development of smart driving interface designs in the future [3].

The layout of the in-vehicle system must be identified and operated quickly to improve user’s efficiency, avoiding serious consequences due to involvement. Many studies had demonstrated that driver’s attention will be distracted while operating the IVIS, which will result in a deteriorated driving performance [4].

The design of vehicle information system is an important factor to evaluate the safety impact of interaction between primary driving tasks and secondary tasks while driving [5]. And the development of IVIS is diversity, the manipulation on the system is becoming more and more complex. Therefore, it is meaningful to establish an effective evaluation method for the design of HMI.

In some previous studies, subjective evaluation, questionnaires and interviews were commonly used for evaluation of HMI. In addition, facial expressions, eye tracking system and skin resistance were recently involved in HMI evaluation [6]. In the automobile field, task response time, lane deviation and other indicators based on the state of vehicle were measured for HMI evaluation [7].

Driving simulator, including NASA-RTLX combined with lane deviation and steering wheel angle were used in evaluation of effect on driving performance in the context of music selection [8].

In a driving simulator study of Jeong et al. [9], the position of in-vehicle display was evaluated by physiological and eye movement data. The results showed that the drivers focus of visual field varies with the position of the in-vehicle display, and the conventional location of a display (center console) gave the lowest level of GSR, which means lower mental stress.

Jin et al. [10] established an evaluation method based on driving simulator and eye tracker, which could be used for the evaluation of driving performance when drivers performed in secondary tasks.

Harsham [11] extracted meaningful indicators from vehicle CAN and history data, to evaluate the complexity of the HMI. And came out prediction methods for user behaviors and intentions to improve in-vehicle task manipulation performance.

In the study of Pankok [12], display clutter may have its greatest effect on performance or attention allocation in driving. The more display clutter, the more effect on driver’s distraction when operating secondary tasks. And drivers more focus on display clutter than display clear.

Ooi [6] used electrodermal activity (EDA) to investigate stress and anger as primary emotion leading to accidents. They established highly reliability frame of driving emotion recognition by comparative experiments.

Nowadays, there are many devices in the vehicle. These devices are used by drivers to perform secondary tasks related to visual, auditory, vocal, manual or cognitive, and they also were normally associated with more than one type of distribution [13,14,15,16]. Especially in the vehicles, driver’s hands are mostly engaged in primary driving task, but secondary tasks involved finger movement such as playing music, Bluetooth connection and navigation are increasing rapidly. Thus manipulation an IVIS while driving can distract drivers’ attention and further reducing the drivers’ performance, drivers’ ability to manipulate the IVIS [17, 18] and leading accidents [19].

Fingers are the one of the most important part on the body while operating HMI system. Due to drivers directly manipulate in-vehicle information system by fingers, which behavior can directly reflect driver’s experience of it. Finger movement indicators are also significant for evaluation of HMI. Therefore, it is meaningful to measure the movement of fingers.

This paper presents a new human-machine interaction evaluation method from both objective and subjective perspective that combined with eye movement tracking, finger tracking, questionnaire and interview which cloud be used in design of HMI, especially in the field of automobile. In this study, we conducted a music selection task to evaluate a domestic popular IVIS based on the evaluation method mention above.

2 Method

2.1 Apparatus

This experiment adopted a CHANGAN EV200 as the test car. An onboard IVIS system including music was equipped in the middle of the control panel of the car as shown in Fig. 1. The screen was a capacitive screen, and the size was 17.7 (w) × 10.0 (h). Music software was installed on the system, and the icon was on the home page of the IVIS system.

Fig. 1.
figure 1

The middle of the control panel of the car.

Full size image

Figure 2 was a screen shot of the screen display. It was located on the right side of the steering wheel so that it can be easily viewed and manipulated.

Fig. 2.
figure 2

Home page of IVIS.

Full size image

As shown in Figs. 3 and 4, the two main interfaces of music function are music playing interface and music list interface, which are also the main interfaces tested in music selection function.

Fig. 3.
figure 3

The music playing interface.

Full size image
Fig. 4.
figure 4

The music list interface.

Full size image

EthoVision XT 11.5 software was commonly used to study motion of small animals in two-dimension. We found that the EthoVision XT 11.5 can properly measure the two dimensional motion of thumb on screen. The participant’ forefinger movement images were recorded by a camera (Logitech C920) from the vertical direction of the screen. The camera’s pixel count of 5 million, and the maximum FPS was 30 frames per second. The camera quality met the experimental requirements. The video files created by the Logitech C920 will be imported into the EthoVision XT 11.5 software, and thumb movement speed was acquired based on image recognition technology. A marker which color was distinguish from the environment will be attached on the nail of thumb, then movement of the marker was considered to be the movement of thumb. The indicators of finger movement total distance, finger movement average angular velocity, finger movement average liner velocity, finger movement total angular, number of clicks and total time of finished task.

A Dikablis Professional eye tracker (Ergnoeers, Germany) was adopted in this experiment to record the image of eye movement at 60 Hz. Two infrared cameras were used record the images of left eye and right eye respectively. Fixation frequency, heat map and visual search pattern were calculated by eye movement analysis software D-Lab (Ergnoeers, Germany).

2.2 Participants

Ten participants (6 males and 4 females ranging in age from 21 to 25 years) were recruited using the following criteria: (1) resident in the Chongqing area; (2) in possession of a valid driver license for at least two years. All the recruited participants didn’t use this IVIS before. All participants were in good health and free from eye diseases. Participants were instructed to avoid tea, coffee and cigarette on the day of the experiment.

2.3 Procedure

Before the formal experiment, participants were informed the objective of the study and their task. Each of participant was requested to music selection task. Simultaneously, eye movement and finger movement data were recorded.

We divided the music selection task into seven small manipulation tasks. Thus, to complete the music selection task, participants have to carry out the steps that included (1) clicking music icon (2) clicking music list icon (3) clicking icon of “song” (4) click the “search” (5) click input box (6) input name of song (7) click playing music (8) click icon of collecting. Each task instruction was pre-recorded the sound file and played to participants through audio. After listing to a given item instruction, participants performed the tasks through manual manipulation of center control. Once the last target had been chosen, the experimenter signalled to participants that the task had been completed. Having a five-minute break. After the experiment, the participants were requested to fulfill a questionnaire (PANAS), which was used to reveal relations between positive and negative affect with personality stats and traits.

2.4 Date Analysis

Data analysis had two parts: objective data and subjective data. Objective data include finger behavior data and eye movement data. Subjective data include questionnaires and interview.

The analysis of finger behavior data was recorded the whole process of participants completing interactive tasks with Logitech camera, and then import the video into the EthoVision TX. The finger movement data obtained in this study included two parts, video observation and software processing.

Video data included task completion time, completion task number of steps, each step completion time and number of steps. More specifically, (1) Task completion time referred to the time between the participants’ issuance of orders and the completion of tasks. (2) The number of steps referred to all the steps in which participants completed the task. (3) Each step completion task time referred to the time spent between two adjacent correct steps. (4) The exploratory number of steps referred to the keys are pressed the number of errors between two keys being pressed [20].

According to ISO15007-1, a saccade was defined as brief, fast movement of the eyes that changes the point of fixation and a glance was defined as the maintaining of visual gaze within an area of interest [21]. In this study, the details of scanning were extracted by MATLAB encoding, including total fixation time, number of fixation and gaze-on time. To solve the above questions, after testing the participant’s manipulation behavior, we found the expert who was very familiar with the manipulation of this car screen to test him in the same way as the participants.

According to the experience, with error rates generally within 30% of observed values, in line with expectations for expert’s predictions. Statistical significance was accepted at p < 0.05 and conducted using SPSS for Mac 24.

3 Result

Ten participants took part in the experiment. However, three participants’ data were excluded from analysis, because of breaking down eye tracker.

As the Figs. 5 and 6 showed, participants required more mean completion the music task time (113 s) than expert (18 s). There was a significant difference between participants and expert on task completion time(p < 0.05). This result indicated that the task completion time of participants were significant longer than expert.

Fig. 5.
figure 5

Task completion time of the music selection task

Full size image
Fig. 6.
figure 6

Number of step of music selection task.

Full size image

As can been seen in Fig. 6, the number of step data were analyzed and revealed that participants took significantly more steps to finish task than expert (p < 0.05).

As the Figs. 7 and 8 showed, finger manipulation data of participants were analyzed using One-Samples t-test and showed a significant different in total finger movement distance between participants and expert (p < 0.05).

Fig. 7.
figure 7

Total finger movement distance of the completion music selection task (mm).

Full size image
Fig. 8.
figure 8

Total finger movement angle of the completion music selection task (rad).

Full size image

Compared with the expert, a significant different in participants on total finger movement angle (p < 0.05). Therefore, it can be inferred that participants’ fingers take much more workload than experts.

However, analysis of the accuracy data using a single sample t-test showed no significant difference in average finger movement speed and average finger movement angular speed between the participants and expert (Figs. 9 and 10).

Fig. 9.
figure 9

Average finger movement speed of completion the music selection task (mm/s)

Full size image
Fig. 10.
figure 10

Average finger movement angular speed of completion the music selection task (rad/s).

Full size image

Based on the above data, it can be inferred that there are problems in the interactive design of music selection task in this IVIS. Therefore, we analyzed completion time and number of steps for each step to find problems in road of music selection. Each step data including completion time and number of tentative steps of participants were also analyzed by One-Sample t-test.

There were significant differences among the result of five steps in the process of accomplishing the task. Some steps including, step 2, step 4, step 5 and step 6 had no significant different in the completion time of each step with expert. It showed that the interactions involved in these steps are reasonable. However, some steps including step 1 (p < 0.05), step 3 (p < 0.05), step 7 (p < 0.05) and step 8 (p < 0.05) had a significant different in the completion time of each step with expert (Fig. 11).

Fig. 11.
figure 11

Completion time of each step (s).

Full size image

The number of completion steps in step 1, step 2, step 7, step 8 was not significantly different from the number of exploratory steps prescribed by experts. Step 3 (p < 0.05), step 5 (P < 0.05), step 6 (P < 0.05) had a significant different between participants and expert (Fig. 12).

Fig. 12.
figure 12

Each step number of steps.

Full size image

However, eye movement indicators such as (fixation time, number of fixation, gaze-on time) that were commonly used cannot reflect these problems accurately.

Specifically, there was no significant difference in fixation time, number of fixation and gaze-on time between participants and expert.

To finish music selection task, participants covered two interface. Fixation frequency was used to whether participants cognitive load exceeded normal when their manipulated the music interface. It was found that fixation frequency (number of fixation divided by fixation time) were no significant difference in music playing interface and the music list interface between participants and normal value (4).

Generally, information presentation format or layout has a long history of research in safety-critical domains other than road safety [22]. According to attention track of participants, we found the participants’ common operating characteristics.

The PANAS questionnaires was a self-report questionnaire including two scales which were measured both positive and negative affect. Each cell task was scored on a scale of 1 (not at all) to 5 (very much) [23]. Clinical and non-clinical studies have found that PANAS was a reliable and effective tool for evaluation both positive and negative effects [24]. Participants filled out this form after finishing music selection task. The figures showed that Positive emotions accounted for only 42%, while negative emotions accounted for 58%. It can be seen that the experiential feeling of the music selection task on IVIS.

Next, participants filled in some satisfaction questionnaires. These problems could reflect participants’ satisfaction with the music selection after completing manipulation. Questions, number of participants and average scores are shown in the Table 1 (1 (not at all) 5 (very much)).

Table 1. The satisfaction scale of music selection function.
Full size table

If the mean score reached 80%, it was indicated that most participants were satisfied with the interactive design of music selection. But only 64% of participants were satisfied with the music selection function.

In the interview, participants also asked some questions about manipulation of music task. These problems were summarized and classified.

4 Discussion

As the complexity of the HMI increased, drivers’ distraction can contribute to accidents [11]. In previous studies, many scholars have evaluated IVIS by driving performance, emotion, eye tracking or subjective scale. Their concluded that only the usability of IVIS or analysis of driving distraction caused by manipulation IVIS. Now, there is no suggestion on the design of IVIS.

Unlike the previous studies, this paper added finger indicators combined with eye movement indicators and subjective questionnaires to test the music function of IVIS. It was found that, participants’ finger behavior data, video data and eye movement data reflected interaction design problems of IVIS and we could accurately find the root where caused of the problem.

The experimental results showed that the new evaluation method was feasible and effective to evaluate In-vehicle information system. Based on the above experimental results, we found three problems in the function of music selection.

First, the task completion time and the task number of step were more than baseline. It suggested that participants had many irrelevant steps when they completed the music task. Experience showed that it was very complicated to complete a task in more than five steps, even expert spent eight steps on completion task. And there is a “15-Second Rule”, in a stationary vehicle, if the drivers can complete the task within 15 s, the task may be safe and this task also didn’t increase risk of vehicle accidents while driving [25]. It shows that the design of interaction in this task may be reasonable. In this experiment, participants spent 113 s on music tasks and expert also spent 18 s on it. And, in the interview, five participants mentioned that completed task was a lack of consistency. It indicated that the logical framework of music selection task was too complex.

Total finger movement distance and total finger movement angle also had a significant difference with participants and expert. It guessed that participants spent more attention to search the targets and participants’ finger had a large number of horizontal movements during music selection tasks, which resulted in a significant increasing in the finger movement angle speed. And this guess was confirmed by video data.

Particularly, according to the data of each step completion time, there were issues with step 1, step 3, step 7, step 8. But the number of steps showed that there were problems with the manipulation of steps 3, step 4, step 5, step 6. Participants had no more steps in step 4 and the number of steps in step 5 and step 6 were less than expert.

Step 1, step 7 and step 8 have significant differences in completion time of each step but no significant difference in number of steps.

It was found that, by watching the participants’ manipulation video, all participants could find the music icon in the first time and clicked it. But participants took a lot of time to complete step 1 because the screen wasn’t sensitive. So the second problem is that the center console display isn’t sensitive.

Step 7, clicking playing music, video data indicated that most participants were considering which song to play when they completed step 7. It looked like a very simple step, participants spent more time on manipulation but it was close to the baseline for participants to complete step 7. Thus it was irrelevant to interaction design.

Step 8, clicking the collection icon, had a significant different with participants and expert. Video data showed that most of participants didn’t see quickly the collection icon in music list interface. In interview, I asked participants why they did not quickly find the collection icon. The unanimous answer of the participants was not to see it. Because of the icon color was similar to background color.

Thus there were problems with the interaction design in step 3.

Step 3, clicking icon of “song”, from eye movement data, we found that there was no significant difference in fixation time, number of fixation and gaze-on time between participants and expert. It indicated that participants saw the “song” icon, but they did not click on it. So, participants’ perception of the “song” icon was low. It also reflected that finger movement data could find problems which eye movement data couldn’t find.

As the Fig. 13 showed, we inferred that participants had two common habits in searching for the target in the process of completing the task. First, we found most of participants’ eyes swept from left to right and from top to bottom. Secondly, what’s more interesting is movement of the fixation point changes with the movement of the finger when participants searched the target and participants searched for target by the way of finger-centered to surroundings. If the guesses are confirmed, it can provide a theoretical basis for the layout of interface information to improve manipulation efficiency.

Fig. 13.
figure 13

One participant’s attention track in music selection task.

Full size image

For example, distribution from left to right according to functional importance and distribution from top to bottom according to using frequency as so on.

5 Conclusion

Different from other research, one hallmark of it is that this method could find the problems in HMI design. The experiment was mainly analyzed finger behavior data and compared finger data between participants and expert. We analyzed problems in the HMI from finger movement data and eye movement data combining with questionnaires and interview. These were results that were not possible from eye movement data.

Further, we resolved the task into simple steps and analyzed those steps by multiply indicators. This evaluation method of HMI can help with design IVIS for manufacturers. And it is also good for reducing driving distraction and decreasing accident risk. Further, the evaluation model built in this study could estimate IVIS effectively under different vehicle.

Analysis of each step should add more finger movement indicators and eye movement indicators to improve the reliability of the analysis results and find more problems in IVIS. There is no improving design and no experiment data of improving design to compare with origin designing data. If the improved data were better than the improved data, the evaluation method would be proved useful. The experiment was carried out in static condition. However, the participants were told to simulate the driving situation in the real vehicle, which was different from the operation of the vehicle information system in driving situation. Therefore, it is suggested that the driver’s manipulation data of IVIS should be tested accurately in driving.