Keywords

1 Introduction

As a milestone product of the AI era, the development of autonomous vehicles has attracted tremendous attention from the whole society. The future of autonomous transport service should be a seamless, on-demand, all-weather service with no restrictions on age, gender, physical function or any other aspect, and thereby freeing the public from driving tasks. From the city and government levels, a transportation network of autonomous vehicles can provide a safer traffic environment, improve vehicle utilization, reduce fuel consumption, and lower the traffic volume. However, being able to be accepted by the public is a prerequisite for the success of autonomous transport service.

In the initial stages of technological penetration, the passenger’s primary need for autonomous vehicles is to ensure safety. Once this need is sufficed, the next focus is on the comfort level of the ride. Ride comfort is a continuous variable for evaluating ride experience, which can be perceived by users and also be measured (e.g. evaluation standards such as Sperling Method and UIC513). Ride comfort of passengers should also be taken as an important consideration for autonomous vehicles. However, the measurement scheme for conventional vehicles is not entirely suitable for driverless areas where scenarios are complex, scoring is required to be more precise, and users’ psychological experience of the security and intelligence of the system is involved. Comfort is an important experience for passengers as they ride on the vehicle. As such, comfort level should be considered as an important evaluation indicator in developing autonomous vehicles, and it is therefore particularly important to establish an evaluation model for ride comfort of autonomous vehicles.

Many scholars have defined the perceptual comfort of human beings. Hertzberg [1] believes that comfort is a state of human perception, and comfort means that people do not have uncomfortable experiences. Coreltt [2] points out that human comfort is the feeling of balancing all parts of the human body, when the balance of external disturbing factors is broken, the human body will feel uncomfortable. Slater [3] points out that comfort is a state of physical and psychological pleasure and harmony in the environment. De Looze [4] recognizes that comfort is a state of physical and psychological pleasure and harmony in the environment. To be comfortable would be affected by physiological, psychological and material factors, which are the subjective response of human beings to the environment. To sum up, comfort is first of all a subjective response of the human body to the stimulation of the external environment, and the condition for human body to feel comfortable is that the external physical conditions are suitable; comfort contains two meanings: one is that the physiological mechanism of human beings is in a relaxed state, the other is that human beings feel happy psychologically.

There have been many studies on ride comfort both in China and abroad for conventional transportation vehicles such as cars, trains, buses, etc., involving influencing factors, evaluation indicators, and evaluation methods. We summarized the current research status based on the influencing factors, evaluation indicators and evaluation methods of ride comfort, and according to this, we considered the contents and methods of the differentiated research on ride comfort for autonomous vehicles.

1.1 Influencing Factors and Evaluation Indicators

There are many studies on the ride comfort of conventional vehicles, and it is found that the ride comfort of passengers is mainly affected by two factors: “ride environment” and “vehicle driving characteristics”.

The riding environment is one of the factors influencing riding comfort, which can be divided into internal environment and external environment. In evaluating the internal driving environment, Liu et al. [5] and Zhu et al. [6] adopted the concept of environmental factors, including specific indicators such as space size, noise intensity, temperature level, and seat performance. Wang [7] used noise and pressure, air quality, vehicle decoration environment, etc. to evaluate the internal driving environment; in evaluating the external driving environment, Taghirad [8] built the function model of ride comfort, including indicators such as road roughness. A recent study by Telpaz et al. [9] also found that the distance between pedestrians, buses and other objects close to the vehicle will also affect ride comfort. Richards [10] and others collected the physical characteristics of bus and train environments and subjective scores of passengers on the interior environment to determine vehicle comfort. Suzuki et al. [11] defined comfort degree, and studied the influence of vibration and noise factor, olfactory factor, seating factor, visual environment, skin factor on human riding comfort. Masakazu [12] et al. developed a facial expression-based evaluation method to evaluate the quality of in-car ride. The comprehensive psychological feelings of users in-car can be judged by facial factors such as eyes and mouth. Sivilevičius [13] et al. established various factors describing users’ satisfaction with trains, such as comfort, safety, travel time and price, and established a multi-objective optimization mathematical model to calculate the comprehensive quality index of trains.

Vehicle driving characteristics is another important factor affecting ride comfort. Ni, Wang [14] and Ma [15] pointed out that the two indicators widely used by Chinese and foreign railway transport agencies are stationarity and comfort. With regard to stationarity, some more classical evaluation standards, such as Sperling Method, are used to evaluate the running quality of vehicles by calculating stationarity using amplitude, vibration frequency, and acceleration, etc. Nastac and Picu [16] have adopted this method. As for comfort, many researchers such as Ma et al. [15] mentioned UIC513 comfort index, in which the acceleration in the transverse, longitudinal and vertical directions of the measured points is comprehensively weighted to evaluate comfort. In addition to Sperling Method and UIC513, there are many different indicators/combinations of indicators for evaluating vehicle driving characteristics. Earlier studies by Smith et al. [17] predicted passengers’ feelings based on vehicle acceleration. Strandemar [18] also evaluated ride comfort by acceleration parameters; Sun et al. [19] directly measured vehicle vibration; Taghirad [8] evaluated ride comfort by the passengers’ movement (left and right sway, upward and downward bounces), tire deflection, etc.

1.2 Evaluation Methods

After determining the influencing factors of ride comfort and specific evaluation indicators, it is necessary to evaluate the comfort of riding a certain vehicle through specific evaluation methods. The academia mainly uses subjective feedback of passengers and objective data collection.

Subjective feedback from passengers. Data were collected through questionnaires and interviews. For example, Liu et al. [5] asked passengers to report their subjective feelings and built a subjective evaluation model for riding comfort. Yang and Zhang [20] used the passengers’ subjective evaluation table on ride comfort to collect user feedback on vehicle noise, ride vibration, up and down jolt, left and right sway, front and back sway, ride comfort, discomfort during the ride, etc.

Objective data collection, such as Sperling Method, UIC513 and other standards calculate the stationarity or comfort index by collecting the amplitude, vibration frequency and acceleration data of the vehicle in the lateral, longitudinal and vertical directions, thus rating the stationarity and comfort level of the vehicle during a ride.

Most of the existing studies on conventional vehicles will be mainly carried out based on one of two above methods, aiming at building a ride comfort model system. However, the study of this paper combines subjective and objective methods to build a more efficient evaluation system that can automatically rate the comfort level of autonomous vehicles.

1.3 Summary

The purpose of this study is to establish an evaluation method for ride comfort of autonomous vehicles. Currently, there are few studies specifically for autonomous vehicles. The influencing factors and evaluation indicator mentioned in conventional studies can be referenced but cannot be directly applied. As for the “riding environment”, the influence of the external environment on riding autonomous vehicles such as surrounding vehicles and pedestrians needs to be considered. However, the internal environment of autonomous vehicles will remain in a stable state for most of the time, and through regular cleaning and maintenance, passengers’ feelings will not change greatly. Therefore, the internal environment is not the focus of this study. Regarding the “vehicle driving characteristics”, there are differences in driving characteristics between different types of vehicles and different products of the same type of vehicles. In this area, autonomous vehicles also need to be paid close attention to. In addition to these two factors, because driverless technology is closely related to safety, intelligence, etc., the “system intelligence” factor should also be considered in evaluating ride comfort. Specifically, “system intelligence” can be regarded as the ability to handle special situations, the ability of the system to maintain accurate and stable, etc. This study will also split specific evaluation indicators based on this factor.

When summarizing the existing research on conventional vehicles, we found that most of the researches only adopted either subjective evaluation or objective data, which could not directly guide the update of autonomous driving technology and had certain difficulties in converting results. In addition, a subjective evaluation model can be established through subjective scoring, but the cost of purely manual scoring is relatively high. With the increase of autonomous vehicles, running scenarios and locations, manual scoring will no longer be feasible and more efficient scoring methods are needed. Therefore, we need to consider how the subjective evaluation corresponds to the objective behavioral data of the vehicles, so as to combine both the subjective and objective methods, and finally form an automatic scoring tool.

The structure of this paper is as follows. In the second part, we will roughly describe the overall research ideas and methods. In the following third part, we will discuss the details of the research steps and how to obtain the research results through derivation. In the fourth part, the results are summarized. And through discussion, there is a reflection on the achievements and experience gained from this work.

2 Research Overview

2.1 Research Steps

A long-term user research is conducted based on the prototype of autonomous vehicles (see Fig. 1). Nearly one thousand users were invited, known as the “Test-riding Pioneers”, to experience autonomous vehicles.

Fig. 1.
figure 1

Research process

Full size image

In the process of inviting users to test, influencing factors of ride comfort on autonomous vehicles have been identified, and the subjective evaluation model is gradually built and verified. Based on this, the vehicle’s driving data were simultaneously collected, such as acceleration/deceleration, steering angle, etc. Also, a one-to-one correspondence between the driving data and the passenger’s real-time feedbacks was made. Finally, an automatic evaluation tool for ride comfort on autonomous vehicles was formed.

2.2 Tools

The autonomous vehicle used by the research institute was a modified Lincoln MKZ (shown in Fig. 2);

Fig. 2.
figure 2

The prototype of an autonomous vehicle used by passengers for the trial ride (a modified Lincoln MKZ)

Full size image

The Lincoln MKZ vehicle used for the test has been modified with a driverless system on it and has the capability of Level 4 automatic driving on open roads. Each test vehicle has an automatic driving test license issued by the Beijing municipal government, and a safety officer will be assigned to supervise the safe driving at all times during the test. The subjects all sat in the back seat.

In addition, a data collection app for instant comfort evaluation is developed to evaluate passengers’ subjective experience.

The subjects were asked to evaluate the ride experience in real time through a data acquisition APP on mobile phones (shown in Fig. 3) during the ride of the autonomous vehicle, and the APP would timestamp each evaluation.

Fig. 3.
figure 3

The comfort evaluation and data acquisition app

Full size image

2.3 Test Scenario

The study has set up a driving test route for an autonomous vehicle in Beijing’s Daoxianghu road section (as shown in Fig. 4). The road was in good condition (traffic density was not high, traffic was slow, and obstacles were few). After traveling 16.2 km from the intersection of Beiqing Road and Daoxianghu Road, the car would return to the starting point.

Fig. 4.
figure 4

The test route in Beijing Daoxianghu road section

Full size image

The average speed of the autonomous vehicle is 48 km/h, and it takes about 20 min for it to complete a full round. In the autonomous vehicle, two safety officers were arranged to take over the car under special circumstances. In addition, an instructor was arranged to guide the tasks to be performed by the subjects and give explanations when the subjects ask questions.

3 Procedure and Results

In this part, we will illustrate in detail the procedure and results obtained through the above methods and steps. The part can be mainly included into four parts as below.

3.1 Set up Subjective Evaluation System

At the beginning of the study, we first need to build a subjective evaluation system for the ride comfort of autonomous vehicles. In the first part, we listed the factors that affect the comfort of vehicles in reality and believed that the dimensions that are more relevant to autonomous vehicles are driving factors and external environmental factors. Then based on the analysis in the first part, we will analyze and sort out the influencing dimensions of the ride comfort on autonomous vehicles in this part.

Research Set

We firstly invited 30 test-riding pioneers in the early tests of unmanned vehicles. The gender, age, driving experience, and other attributes were evenly distributed, thus reducing the impact of sampling.

They had a complete experience of the nearly 17 km driverless journey on the Daoxianghu section. During the test, there was a safety officer in the front row, and the passengers were seated in the back row. At the same time, there was a tester in the back row who followed them all the way.

The study would invite the subjects to take the vehicle, ask them to think aloud their evaluation of riding comfort. Their comments had been recorded during the whole test with passengers’ consent.

Semantic Analysis

We conducted semantic analysis on the evaluation from 30 passengers. We analyzed the collected information according to different types of driving scenes. And we filled the subjective evaluation in each scene. In the end, we sort out the words related to comfort by word segmentation (Table 1).

Table 1. The analysis of the passenger’s evaluation on testing experience
Full size table

By sorting out the interview records of all 30 passengers, we found out some main factors that affect passenger comfort and then further cluster these factors.

Finally, three types of comfort evaluation dimensions are sorted out: driving factors (the feeling of leaning forward, the feeling of braking, the feeling of swaying, the feeling of shaking, the feeling of bumping, the feeling of being pushed back), external environment factors (the sense of safety related to speed, the sense of safety related to distance), and specific factors (accurate driving, flexible driving, and stable driving). Among them, the specific factors are the special influencing factors of ride comfort of autonomous vehicles, which mainly reflect the degree of intelligence of autonomous vehicles. The passengers repeatedly mention that the degree of intelligence of autonomous vehicles will directly affect the psychological comfort of the passengers, which is also the reason why we include it into the subjective model.

After naming all the factors and confirming the corresponding meaning, a preliminary subjective comfort evaluation model was formed (Table 2).

Table 2. The subjective evaluation model framework of AV’s riding comfort
Full size table

3.2 Verify and Determine Subjective Evaluation Model

Based on the preliminary subjective evaluation model, we will confirm the validity of the model through quantitative verification, modify the evaluation dimension and finally determine the model.

Research Set

We have continuously invited 377 test users to experience autonomous vehicles in nearly half a year, and these users would fill in the ride comfort chart through the data recording app after a complete experience of autonomous vehicles. We used a 5-point system to measure the comfort level of each type of ride. We set one item to match each layer of subjective model. No. 2–6 fit detail factors of somatosensory. No. 8–9 were for sense of safety and No. 11–13 were for sense of intelligence. No. 2, 7, 10 were set to measure general feeling of 3 main factors. The whole feeling of riding comfort can be measured by the first item. Please refer to Table 3 for specific items.

Table 3. Questionnaire of AV’s riding comfort
Full size table

In order to reduce the differences in each individual’s understanding, we defined each scoring dimension, taking the feeling of leaning forward as an example (Table 4).

Table 4. Scoring criteria for riding comfort-the feeling of leaning forward
Full size table

Verify Through Model.

Based on the above definitions, we collected 377 questionnaires.

The validity of the model was verified by the structural equation model, and finally a complete subjective evaluation model was formed (as shown in Fig. 5).

Fig. 5.
figure 5

Structural equation model for riding comfort (subjective evaluation) of autonomous vehicles

Full size image

When verifying through the model, we removed two dimensions: the feeling of being pushed back and the feeling of bumping, both of which had little influence on ride comfort. We found that somatosensory perception is an important factor that affects ride comfort. Secondly, the sense of security also directly affects ride comfort, while the sense of intelligence affects overall comfort by affecting the sense of security.

The model fitted well, Chi-square = 22.00 (p = 0.341), CFI = 0.999, RMR = 0.005, RMSEA = 0.017.

3.3 Mapping Vehicle Data Based on Subjective Models

In the third step, vehicle data are mapped to the third dimension of the subjective model, and through machine learning, a subjective-objective automatic model is established, so that an automatic score of the third dimension can be obtained. Then the overall comfort score can be deduced based on the influencing weight of the dimensions in the subjective model.

Acquiring Subjective-Objective Corresponding Data Sets

Based on the subjective model established above, we need to collect vehicle data under different comfort scores for each third-level dimension. Taking the feeling of leaning forward as an example, we need to obtain the vehicle data sets under 1–5 scores respectively, and in order to suffice machine learning, the number of data sets under each score needs to reach 1000.

We continued to invite nearly 400 passengers to experience the autonomous vehicles and used data app to collect data. The data app provides such a function that whenever a passenger feels uncomfortable with certain behaviors of the vehicle, the data app will record the timestamp of the current evaluation. For example, if passengers feel that leaning forward is relatively uncomfortable, they click the button of “the feeling of leaning forward” and give corresponding scores, and the app will record the timestamp of the rating. Using this timestamp, we can extract the current data set of the test vehicle from the automatic driving system, including transverse acceleration and deceleration, lateral acceleration and deceleration, steering angle, etc., thus forming a data packet with the subjective data corresponding to the objective ones. With this process, nearly 8,000 data packets from 8 experience dimensions were collected.

Develop the Automatic Scoring Tool

Based on the data packets for each evaluation dimension, we used the following methods for data training. Taking the feeling of leaning forward as an example, we found out the vehicle data that really affects the change of the score of the feeling of leaning forward through machine learning from the data packets with scores of 1–5, thus forming a function that calculates the score of the feeling of leaning forward according to the vehicle data.

We adopted two major types of calculation models commonly used in 6 industries to train the data. One is depth learning models such as MLP and TCN; The other is integrated learning models such as Decision Tree, Random Forest, GBDT, GBRT. Finally, we would calculate the average score based on the model data included in the overall tool to obtain the score of the dimensions of the evaluation, and the formula is as follows:

$$ S_{avg} = \frac{1}{M}\sum\nolimits_{M = 1}^{M} {f_{m} \left( x \right)} $$
(1)

By analogy to the other 7 three-level evaluation indicators, the function of the vehicle data for calculating the scores of the 8 indicators is finally obtained.

Based on this, we can automatically calculate the scores of eight three-level latitude indicators according to the data of any autonomous driving segment, and finally write the weights of all levels of indicators in the subjective model into the calculation system to automatically calculate the final total score of ride comfort.

3.4 Validation of the Automatic Scoring Tool

When we finish developing the automatic tool, we will continue to verify the validity of the model in the next step.

Research Set

We verified it by comparing manual scoring with machine scoring. We will continue to invite users to take a test ride in the following time. After users complete the test ride, they will fill in an optimized questionnaire. At the same time, the machine will also score the same journey. Finally, we will examine the difference between manual scoring and machine scoring.

Validation Results

We used the Mean Score Error method to verify the consistency between manual scoring and machine scoring. As shown in the following Table 5, the comprehensive accuracy rate of the models of the three somatosensory indicators is close to 0.8, which we believe has reached the usable level.

Table 5. Verification of accuracy of the models of the three somatosensory indicators
Full size table

Because inviting users to take a test ride and the data processing take a long time, we have currently completed the validation of the secondary dimension somatosensory indicators, and the validation of other dimensions is still in progress.

4 Model Application

After the somatosensory model passed the validation, we applied it in two directions:

4.1 Verify Change of Capability

We used the model of the somatosensory data to backtrack the road test data of autonomous vehicles in the past year, and calculated the somatosensory score of the daily road test data. In order to facilitate the monitoring of abnormal data, we cut the vehicle data into 10-min segments, and the somatosensory score of the day was the average of all segments. As shown in Fig. 6, the horizontal axis represents the date, and the vertical axis is the distribution of mean, maximum and minimum scores of the somatosensory score for the day.

Fig. 6.
figure 6

Structural equation model for riding comfort (subjective evaluation) of autonomous vehicles

Full size image

In the whole scope, we can find that the change trend of somatosensory scores in the past year shows an overall upward trend, representing that passengers’ somatosensory perception on autonomous vehicles is also gradually optimizing.

4.2 Real-Time Monitoring Experience

On the other hand, we applied the model to the current daily road test data monitoring. When the autonomous vehicle completed a day’s road test, the technician uploaded the day’s data to the data platform. The automatic scoring tool will trigger the automatic scoring mechanism in the early morning of each day to query and download the data of Daoxianghu road section in the past day and perform automatic scoring.

For the scoring results, on the one hand, we can use it as a measure to monitor the somatosensory change curve; On the other hand, it can help technicians locate the abnormal scores of the day. If abnormal low scores are found, it can quickly locate the low score data packet, check the video playback of the road test to locate the problem, and solve the technical problems instantly and efficiently.

5 Conclusion

Through this study, we have formed a set of scientific and universal evaluation criteria and scoring tools for the riding comfort of autonomous vehicles. This model can conveniently, quickly, and stably measure the riding comfort of autonomous vehicles, and can play its role and bring value both internally and externally.

Internally, based on this model, we can continuously observe the riding comfort changes of our own autonomous vehicles. Such changes can guide the technological R&D team to continuously optimize their technological algorithms to create products with better ride experience. Compared with the past methods of technicians randomly extracting data for playback and testers recording somatosensory problems on cars, it not only saves manpower, but also can locate problems more quickly and improve unmanned driving technology.

Externally we hope that this model can become a unified evaluation standard, which can be used to conduct horizontal comparison and evaluation of different autonomous vehicles. Also, we hope to cut in from the perspective of riding comfort to promote the autonomous vehicle industry.

We also plan to continuously optimize this comfort model and apply it to a wider range of scenarios.

Continuous observation and calibration of the model: After the development of the automatic scoring tool based on the riding comfort model of autonomous vehicles, we still need to use the same method to continuously collect new subjective and objective data and verify whether there is a difference between the scores given by the tool and the users. If there is, the model will be calibrated using the newly collected data, and will iterate step by step, until the newly collected subjective and objective data are completely matched with the fitting function, and then the model is considered to be calibrated.

Testing and application in more scenarios: At present, the scenarios set for the autonomous vehicle riding are relatively loose: the speed is slow, the surrounding road conditions are simple, and the riding is accompanied by safety officers. In such an environment with many favorable factors, the problems existing in the autonomous vehicle riding experience cannot be fully exposed. In the future, autonomous vehicles will certainly be used in various scenarios, including complex roads (which may affect passengers’ sense of safety) and faster driving (which may affect passengers’ feelings of being pushed back and centrifuged), etc. We will consider more testing scenarios and apply and verify this riding comfort model.