1 Introduction

With the increasing numbers of car, our life has become more and more convenient but several issues raised at the same time. The traffic safety problems attract many researches, and intelligent transportation could better support their solution along with the increasing number of vehicles and drivers.

According to the statistical analysis data shown in Fig. 28.1, most accidents are caused by driver’s behavior and personal factors [1, 2]. Therefore, it is of great significance to study and analyze the driver’s driving operation to improve the level of traffic safety management.

Fig. 28.1
figure 1

Distribution of traffic accidents statistics [1, 2]

Full size image

Various literature studies have introduced driving operations recognition methods based on computer vision, multi-sensor fusion, etc. Škrjanc et al. presented an evolving cloud-based algorithm for the recognition of drivers’ actions [3]. The general idea is to detect different maneuvers by processing the standard signals that are usually measured in a car, such as the speed, the angle of the steering wheel, and the position of the pedals without additional intelligent sensors. Deng et al. presented a driving style recognition method using braking characteristics based on hidden Markov model [4]. Chen et al. proposed a driving behavior analysis method based on the vehicle on board diagnostic (OBD) information and AdaBoost algorithms [5]. Tran et al. proposed a vision-based framework where Hidden Markov Model (HMM) was used to analyze the driver’s foot behavior [6].

Smart cushion was also used for activity and sitting posture recognition. Huang et al. proposed the sensor array-based smart-chair to recognize sitting postures [7]; artificial neural network (ANN) was applied to classify eight different postures. Ma et al. proposed a cushion-based posture recognition system [8, 9]; they developed several applications such as wheelchair user’s posture recognition, fatigue detection [10], also by combining cloud platforms to develop a wheelchair assist system to help the caregivers monitor the wheelchair user’s status [11].

According to the above literature, current major research directions of driving operations recognition include driving data collection and modeling algorithms. Driving data collection includes automotive video capture, car-mounted sensors, and the on board diagnostic (OBD). In terms of driving operations modeling algorithms, there are HMM, support vector machine (SVM), and decision trees, among other methods. This chapter proposes a novel driving operations recognition method based on smart cushion and deep neural network (DNN) algorithms. The proposed method collected pressure data when the driver is seated on the car seat equipped with four pressure sensors. After preprocessing and reducing the dimensions using principal component analysis (PCA) for the data, this chapter constructs a driving operations classification model to finally recognize driving operations with a DNN algorithm including normal driving, stepping on the brakes, stepping on the accelerator, stepping on the clutch, and rotating the steering wheel. The experimental result shows that the proposed driving operations recognition method can achieve an average recognition accuracy rate of about 97%.

The remaining sections of this chapter are organized as follows. The detail of the proposed driving operations recognition method is presented in Sect. 28.2. The experiments and results are detailed in Sect. 28.3. Finally, Sect. 28.4 concludes the chapter.

2 Methods

The proposed driving operations identification consists of hardware design, data processing and constructing classification model. The hardware design is used to collect driving operations data. The driving operations data will be divided into a training set and a test set by 8 to 2 ratio. The data processing module includes signal filtering, sliding time window extracted signal, and PCA dimension reduction. The data processing applies to both the training set and the test set. The trained classifier passes the test sample to the driving operations recognition model based on the DNN algorithm to classify the test sample class. The number of correct or misclassified samples divided by the number of total samples in the test set is the classification accuracy or error rate, respectively.

The proposed method first collects pressure data for five different driving operations. Then, such data is divided into training set and test set. Data is filtered, extracted features by sliding time window and processed by PCA dimension reduction. Each processed sample slice can be considered as a representation of the driving operations. This chapter uses the training set to establish a driving operations classification model based on DNN algorithm, and then uses the test set to judge the accuracy of the model. Finally, the feasibility of this method is analyzed and the advantages and disadvantages of using this method to classify driving operations are illustrated. Figure 28.2 shows a workflow of the entire proposed method.

Fig. 28.2
figure 2

The overall workflow of the proposed method

Full size image

2.1 Hardware Design

2.1.1 Cushion and Pressure Sensors

The smart cushion used in this experiment consists of a cushion, four pressure sensors, a Bluetooth module, a micro control unit, and a battery.

The cushion is a common 3 cm thick sponge cushion, which is neither thick enough to degrade the accuracy of the sensor nor affect the normal operation of the driver.

The pressure sensor is the FSR406 produced by Interlink Electronics. Its sensing value changes according to the weight exerted onto the pressure sensor. Table 28.1 reports the main parameters of the FSR406 sensor.

Table 28.1 The main parameters of FSR406 pressure sensor
Full size table

2.1.2 Distribution of the Sensors

In order to collect pressure sensor data more accurately, efficiently, and reduce the cost of the cushion, this chapter provides a sensor distribution method considering the human anatomy. The pressure distribution of the body–seat interface could be converted to image data [12]. Some researches found that the body–seat interface could divide into two ischial tuberosity (IT) [13,14,15] regions and two thigh regions as shown in Fig. 28.3a. When sitting posture changes, the movement of the trunk or leg will cause the ischial tuberosity area or thigh area pressure values to change. These four areas are highly sensitive to the body movement. On the basis of these observations, we developed a smart cushion with the sensor distribution as shown in Fig. 28.3b.

Fig. 28.3
figure 3

(a) Body–seat interface of human body; (b) pressure sensor deployment

Full size image

2.2 Data Processing Method

2.2.1 Finite Impulse Response Filter Method

The raw data are typically affected by noise, e.g., caused by the driver’s body movement or electrical signal noise from the smart cushion itself. In order to reduce the impact of noise on the experimental results, the original signal needs to be preprocessed before analysis. This chapter selects the Finite Impulse Response (FIR) [16, 17] filter to reduce the noise of the original signal.

The FIR filter is defined as Eq. (28.1).

$$ y(n)=\sum \limits_{m=0}^{N-1}h(m)x\left(n-m\right)=h(n)\otimes x(n) $$
(28.1)

where N is the length of the filter, h(m) is the specific pulse of filter system, and x(n) and y(n) are the input and output. The filtering system is shown in Eq. (28.2).

$$ h(n)={w}_n(n){h}_d(n) $$
(28.2)

where h d(n) is ideal filter and w N(n) is form function. We use Hamming window function as shown in Eq. (28.3).

$$ {w}_N(n)=\left\{\begin{array}{c}0.5-0.5\cos \frac{2\pi n}{N-1}\kern0.75em 0\le n\le N-1\\[5pt] {}0\end{array}\right. $$
(28.3)

2.2.2 Sliding Time Window

The driving operation is a time series signal. In addition, in order to remove the interference caused by abrupt motion, bumpy signals, and other invalid actions, and to ensure that the extracted signal can reflect real-time and near-term behavior before establishing the recognition model, in this chapter we select the sliding time window to extract the real-time effectiveness of the driver information. The sample taken by the sliding window is then used as a training sample set.

As shown in Fig. 28.4, W 1 is the sliding time window, the time window length is T 1, and the sampling frequency of the information is F 1. The length of the time window should not be too long nor too short. If it is too long, the real-time performance will be affected. If it is too short, it will affect the recognition accuracy. After many experiments, we selected a rectangular window with a time window length T 1 = 25, and the data sampling frequency F 1 = 10 Hz.

Fig. 28.4
figure 4

Sliding time window schematic

Full size image

In order to improve the efficiency of model construction and programming, we merge the pressure signals of four pressure sensors that are processed by filtering and sliding time windows, that is, time window signals on four pressure sensors in series.

2.2.3 Principal Component Analysis Dimensionality Reduction

The merged data set is with a very high dimension and has high similarity between samples. If it is directly introduced into the training of the DNN network model, the final accuracy of the model will be affected. In addition, training in ultra-high-dimensional data will require increasing neural network model depth, reducing model training speed and wasting computer memory.

Therefore, this chapter uses PCA dimension reduction after merging data and reduces the dimension so that fewer dimensions can represent more data information, which can increase the efficiency of model construction and improve the final accuracy of the model. We found that the relationship between the accuracy of the model and the retention dimension was parabolic through multiple experiments, and the information retained after PCA dimensionality reduction accounted as shown in Table 28.2.

Table 28.2 Information retention ratio after PCA dimension reduction
Full size table

Considering the model accuracy and the ratio of retention information comprehensively, this chapter finally chooses to reduce the high-dimensional sample data to 5 dimensions, which can retain more original data information and ensure the accuracy of the final model identification.

2.3 Classification Model Construction

The data processed by the PCA will be directly input into the DNN model for training and testing. Since the current DNN still belongs to the black box model, the selection of some hyperparameters in the DNN model is still in an empirical stage. This chapter experimentally compares the model accuracy and training time to select a better DNN model structure and its hyperparameters. The final hyperparameters in the model are shown in Table 28.3.

Table 28.3 DNN model structure hyperparameters
Full size table

3 Experiment and Result

The specific process of the experiment is shown in Fig. 28.5. First, the pressure data of the smart cushion is collected; then the data is preprocessed. Data is further divided into a training set and a test set, and the classification model is constructed and trained through the training set. Finally, the trained classification model is verified through the test set.

Fig. 28.5
figure 5

The flowchart of driving operations recognition experiment

Full size image

3.1 Data Collection

In our experiments, we used a vehicle driving simulator instead of a real vehicle to collect pressure data. Vehicle driving simulator, shown in Fig. 28.6, contains the same operating components as the real vehicle: steering wheel, clutch, throttle, and brake.

Fig. 28.6
figure 6

Vehicle driving simulator

Full size image

The purpose of the experiment is to verify that the previous model can effectively identify the driver’s single driving operations. Therefore, in this chapter we selected five basic single driving operations for recognition classification. The description of these driving operations is as follows:

  • Rotating the steering wheel: rotating the steering wheel to the left or the right.

  • Step on Brakes: Contact brake pedal—brake pedal pressed to the end.

  • Step on the accelerator: Contact accelerator pedal—accelerator pedal pressed to the end.

  • Step on clutch: Contact clutch pedal—clutch pedal pressed to the end.

  • Normal: Keep normal driving behavior.

Considering that individual differences of the test subjects (e.g., height, weight, gender) may affect the seat pressure distribution, to obtain more general experimental results, when determining the experimental subjects, we recruited 5 healthy workers (4 males (A, B, C, D) and 1 female (E)) with driving experience with a weight of 50–75 kg and a height of 150–180 cm. The personal characteristics of the participants to the experiments are shown in Table 28.4.

Table 28.4 Personal characteristics of the participants
Full size table

We correctly installed the cushion on the seat of the vehicle driving simulator. The subjects sat in a comfortable position on the seat and operated the steering wheel, accelerator, brake, and other operating components to become familiar with the simulated driving operations platform.

After the participants are ready, we request to start the driving operations through the computer in front of the driving simulator. The five types of driving operations are performed separately. After one type of operation is completed, the following one is performed. Each type of driving operation is carried out 20 times (each execution is separated from the previous one by 10 s); before starting a new type of driving operation, the subject rests for 30 s. The five participants completed the described protocol once. In order to prevent increasing the identification error due to the imbalance of the samples in each type of operations, the number of valid samples of the various types of operations finally acquired must be basically the same. Each participant performs 20 operations of each type, each execution takes 0.5 s, and the sampling frequency of the four pressure sensors is 10 Hz. So the size of the original sample data set is 500 × 4.

3.2 Data Preprocessing

Due to individual differences of the driver’s body, noise signals generated by small movements not related to driving operations, and disturbing electrical signals caused by voltage instability or other factors, in order to ensure the accuracy of the processed data, filter preprocessing must be applied to the original signal before analyzing it. In this chapter, FIR filter is used to smooth the original signal. During the filtering process, the sampling frequency of the pressure sensor is 10 Hz. The detection period is determined according to the specific content of the experiment. Each set of data has 500 points and the filter form length is N = 100.

In order to transfer the pressure signal to the classification model and identify the signal over a period of time, this experiment uses the sliding time window with a sliding step of 1 to extract the first 25 sample data points each time as part of the new sample data, and merge the features of the new sample data of 4 sensors as a complete sample data point.

The data preprocessing workflow for one type of driving operations is shown in Fig. 28.7. Among them, S1, S2, S3, and S4 are the data of four pressure sensors in sequence, and each data has 500 sampling points. The data size of each sensor is 500 × 1. First, it is filtered with FIR and its data size is unchanged. Then, by sliding the time window to extract features, the size relation between the data before and after the feature is extracted from the sliding time window is as follows:

$$ {r}_2={r}_1-\left(T\times S\right)+1 $$
(28.4)
$$ {c}_2=T $$
(28.5)

where r 1, r 2 are the number of rows before and after the sliding time window is extracted, and T is the window size of the sliding time window. T is set to 25 in this chapter. S is the step size of the sliding time window. In this chapter, we select S equal to 1. C 2 is the number of columns of data after the sliding time window is extracted. Therefore, the size of each sensor’s data after extracting features through the sliding time window is 476 × 25. The data features of the four sensors are then combined into a matrix of size 476 × 100. Finally, this experiment combines the data of the five types of driving operations into a feature matrix of size 2380 × 100 for subsequent analysis.

Fig. 28.7
figure 7

Data processing workflow for a type of driving operations

Full size image

In order to verify whether the collected data can identify five types of single driving operations after preprocessing, this chapter visualizes the data after preprocessing. In this chapter, the high-dimensional feature matrix is firstly reduced to a 3-dimensional data matrix by PCA, as shown with scatter plots in Fig. 28.8.

Fig. 28.8
figure 8

3D data visualization of five driving operations. (a) 3D data scatter plot; (b) Partial enlarged plot

Full size image

In particular, Fig. 28.8b is an enlarged side view of the data on the brakes and the accelerator in Fig. 28.8a. After reducing the high-dimensional feature matrix to 3 dimensions and visualization, it can be found that normal driving, operating steering wheel, and pressing clutch are clearly separable. Although there are confusion points between brake and throttle, the classification boundary still exists.

3.3 Model Construction

As mentioned in the method, this article uses experiments to synthesize the accuracy and efficiency, selects the DNN model that is used to reduce the 100-dimensional feature data to 5 dimensions, and uses the DNN model parameters determined in Table 28.3 to train a high precision classifier.

We firstly train the DNN model with the prepared training set and then we obtain the appropriate parameters through back propagation and gradient descent, which increases the model accuracy for the data set (including the training set and the test set). The accuracy of the training set is shown in Fig. 28.9.

Fig. 28.9
figure 9

Accuracy of training set

Full size image

As shown in Fig. 28.9, the average accuracy of the training set increases with the number of iterations, and after 210 rounds of iterations, the accuracy rate rises slowly reaching 97%. Therefore, this chapter chooses the iteration number as 300, and then apply the trained DNN model to the test set.

3.4 Recognition Results

Table 28.5 shows the confusion matrix for the classification results, and Table 28.6 shows the performance of the classifier for different driving operations. The results in Tables 28.5 and 28.6 show that the accuracy of the classification model for brake and stepping on the accelerator were 94.35%. The rest of the driving operations was almost 100%, and the average accuracy was 94.5%. The results show that stepping on clutch, rotating the steering wheel, and normal driving all have their own distinct behavioral characteristics, so the recognition accuracy is very high. However, the results show that the recall rate of stepping on the accelerator and the precision of the brakes are low, and the stepping on the accelerator is particularly easy to be mistakenly recognized as brake. Through the review of the details of the experiment, it was found that since both the step on the brake and the step on the accelerator are right-footed, the actions are very similar, and it is easy to misclassify. Therefore, the next step will be to identify the driving operations in combination with multiple sensors to enrich the identified driving operations and improve the recognition accuracy.

Table 28.5 Confusion matrix of classification results
Full size table
Table 28.6 Performance of the classification model
Full size table

4 Conclusions

For the purpose of recognizing driving operations, a DNN-based classification algorithm and model are proposed. In this system, our aim is to recognize five kinds of driving operations like stepping on the accelerator, stepping on the brake, stepping on the clutch, rotating the steering wheel, and normal driving. The sensors deployed according to the physiological characteristics of the human anatomy. In order to reduce noise, the data collected from the sensors was preprocessed by FIR filtering. Then, in order to identify the time segment signal, the signal is subjected to a sliding time window and feature fusion processing [18]. Finally, the merged data is passed to the model training classifier, and five kinds of single driving operations can be recognized.

Recognition of driving operations remains an open research challenge, and in our vision, the use of smart cushion can provide a new perspective for solving this problem. As a smart object in a car, the driver’s convenience and alarm function are of great significance in the field of traffic safety. Although our results are still preliminary, this smart, convenient and non-invasive method can provide a reference for future research. In particular, we will focus on enriching operations; for this purpose, more experiments are needed. Finally, in the light of the fact that the pressure cushion alone is not able to provide reliable data for the detection of complex activities, in the future we will add more kinds of sensors to enrich the functions of the proposed system.