Detecting driving stress in physiological signals based on multimodal feature analysis and kernel classifiers
Introduction
Some unfavorable mental states, e.g., fatigue (Cyganek and Gruszczyński, 2014, Lee and Chung, 2012, Liu et al., 2015, Yang et al., 2010), sleepiness (Åkerstedt et al., 2013, Hallvig et al., 2013, Morris et al., 2015); stress (Healey and Picard, 2005, Tamrin et al., 2014, Zheng et al., 2015) and distraction (Craye et al., 2015, Horberry et al., 2006) contribute to traffic accidents, leading to considerable number of vehicle crashes, injuries, and fatalities annually. In addition, physical discomfort such as extreme ambient temperature (Daanen et al., 2003, Pimenta and Assunção, 2015), uncomfortable driving position (Smith, Mansfield, Gyi, Pagett, & Bateman, 2015) and muscle fatigue in the neck/shoulder/back area (Hostens & Ramon, 2005) might impact driving behavior. Searching for the countermeasures to reduce the amount of traffic accidents and enhance public road safety has become an urgent issue for both governments and automakers. It is very crucial to develop an automatic system that intelligently detects the driver's unfit status and makes a warning once necessary.
For developing an automatic system to measure drivers’ workload and detect their internal status, it is crucial to select effective measurements. Usually, we can classify the measures into three categories: (1) vehicle behavior measures; (2) video-based measures; (3) physiological measures.
Vehicle behavior mainly includes: vehicle speed, acceleration, lane position deviation, steering, braking and gear changes (Horberry et al., 2006, Morris et al., 2015, Tamrin et al., 2014, Wang and Xu, 2015). These measures are easy to obtain however they are strongly dependent on the type of vehicle and the handling skill of drivers. Additionally, some vehicle features such as lane tracking technology are prone to data loss under the situation of lane markers missing, bad weather or darkness (Wang & Xu, 2015).
Video-based detectors that track head movements, monitor eye gaze status and interpret facial expression have been widely explored (Azim et al., 2014, Cyganek and Gruszczyński, 2014, Garcia et al., 2012, June, González-Ortega et al., 2013, Jo et al., 2014, Kholerdi et al., 2016, Zhang et al., 2012). However, visual features cannot always return reliable results especially in the conditions of poor light, night driving and wearing glasses. Besides, inaccurate interpretation of the facial expression might be encountered when some introverted subjects intend to control their emotions during the process of driving test.
Recently, features extracted from physiological signals such as electrocardiogram (EKG), galvanic skin response (GSR), respiration, electromyogram (EMG) and electroencephalogram (EEG) show relatively high identification accuracy and get insight into driver's states directly (Correa et al., 2014, Healey and Picard, 2005, Lee and Chung, 2012, Lee et al., 2014, Li and Chung, 2013, Li et al., 2012, Liu et al., 2015, Liu et al., 2010, Singh et al., 2013, Vicente et al., 2016, Wang et al., 2015, Zhao et al., 2012, Zhao et al., 2011). The disadvantage of these measurements is that they require sensors and cables attached on the body, which constrains the behavior of drivers in some extent. And the wireless and wearable acquisition solutions allow subjects participate the drive task more conveniently and comfortably (Fu and Wang, 2014, Lee et al., 2014). These physiological measures could be used automatically by in-vehicle intelligent systems in various approaches to help the drivers better manage their negative driving status.
It is a challenging and rewarding work for consecutive detecting of driver states in real driving environments. Computational techniques such as feature extraction, feature selection & reduction, and classification have the capacity to determine optimal sensors and automate driving status recognition.
- (1)
Many feature extraction methods have been developed for physiological signals such as waveform information (Correa et al., 2014, Healey and Picard, 2005), power spectral analysis (Vicente et al., 2016), wavelet coefficients (Li & Chung, 2013), and nonlinear analysis (Chen, Zhao, Zhang, & Zou, 2015).
- (2)
In order to reduce the dimensionality of feature sets or select the optimal features from the primary measures, Principal Component Analysis (PCA) (Wang, Lin, & Yang, 2013) and Linear Discriminant Analysis (LDA) (Correa et al., 2014) are the most commonly used. Besides, Mutual Information (MI) methods and Sparse Bayesian Learning (SBL) have been employed in the feature selection of Brain-Computer Interface (BCI) system and obtained outstanding performance (Atkinson and Campos, 2016, Hoffmann et al., 2008).
- (3)
Several modeling techniques have been developed to detect driving status such as support vector machines (Zhao et al., 2011, Zhao et al., 2009), artificial neural networks (Correa et al., 2014, Singh et al., 2013), fuzzy logic inference (Chiang, 2015) and Bayesian networks (Yang et al., 2010).
These computational techniques will be discussed and compared in the discussion part.
Commonly, stress is viewed as a response to particular events. It is a normal reaction of human body preparing itself in the face of difficulties with focus, strength and improved alertness. Usually, stress can be both physical and mental. If subjects are involved in a long, monotonous driving or under sleep deprivation condition, physical stress might become the major consideration. In this research, three well-defined driving conditions are designed mainly to induce different levels of mental stress. Furthermore, there is a difference between eustress and distress, where eustress is a positive stress helping individuals to better cope with the challenging situations. However, in this research, we focus on distress only, which is usually caused by increased drivers’ workload. A lot of literature has reported the relationship between driving distress and the diminished vehicle handling abilities, reduced alertness, and increased probability of traffic accidents (Bakker et al., 2011, Chiang, 2015, Okada et al., 2013, Sharma and Gedeon, 2012, Sun et al., 2010, Villarejo et al., 2012).
Stressful events could cause dynamic changes in autonomic nervous system (ANS), reflected as increased activity in sympathetic system and decreased activity in parasympathetic system. Physiological signals such as galvanic skin response (GSR), electrocardiogram (EKG), and electroencephalogram (EEG) are the main measures used for stress detection in literature. Heart rate variability (HRV) is a common non-invasive indicator to detect ANS activities and is used as a primary measure for stress detection in many monitoring systems (Chiang, 2015, Okada et al., 2013). HRV can dynamically reflect the accumulation of mental workload which makes it a good estimator for stress level. Short-term reduced HRV indicates acute stress, reflecting that stressful events negatively affect HRV. Skin conductance is another reliable measure for stress. Increased skin conductance reflects the individual under stress and reduced skin conductance indicates the individual under less-stressful situation (Bakker et al., 2011, Sharma and Gedeon, 2012, Sun et al., 2010, Villarejo et al., 2012). Other research demonstrates that relationships exist between brain activities and emotional stress. Stress assessment system has been developed using a new fusion link between EEG and peripheral signals such as blood volume pulse, skin conductance and respiration (Hosseini, Khalilzadeh, Naghibi-Sistani, & Homam, 2015).
Monitoring driving status has great potential in helping us decline the occurrence probability of traffic accidents. However, automatic detection is usually restricted to the stage of laboratory research mainly due to signal's features and noise, measurement constraints and subject-dependent issues. The aim of this paper is to develop an efficient system to determine driver's relative stress level during real-world driving tasks based on the analysis of physiological data. A novel system combining multimodal feature analysis and kernel-based classifiers is proposed. Considering the test differences between individual drivers and within individual drivers, two kinds of analysis are made: Analysis I used features from short intervals of well-defined data to discriminate three levels of diving stress at per-drive and cross-drive levels respectively. Analysis II made continuous stress evaluation throughout a complete driving test for each drive.
It is expected that: 1) different levels of driving stress could be characterized by specific set of physiological measures; 2) more features drawn from multimodal analysis are favorable for assessing the drive status more reliably and accurately; 3) the system combing efficient feature selection methods and kernel-based classifiers could enhance the capacity for identification of driver states.
In rest of this paper, Section 2 describes the materials including experiment design, participant information and data collection. Section 3 gives an introduction of our methodology including feature extraction, feature selection & reduction, and kernel-based classification technique. Section 4 presents our experimental results. Section 5 reviews the similar work and discusses the contributions and limitations of the present study. And the conclusion has been made in Section 6.
Section snippets
Experiment design
We use the database, contributed to PhysioNet by Healey & Picard, which collected a set of multiple physiological recordings from healthy subjects when they were driving along with a specified route in and around Boston.
The prescribed route was designed to make the drivers experience the situations which might produce different degrees of stress and also restore the stressful events encountered in daily commute. Fig. 1 shows the whole driving task which is mainly divided into six sections:
Methodology
The general block diagram of the proposed system is shown in Fig. 2. Physiological signals are acquired by multi-sensors. After the filtering and time segmentation, features extraction is executed including the wavelet decomposition, the time and the spectral analysis. The sparse Bayesian learning (SBL) is employed to select the optimal feature sets. To further reduce the feature dimensionality, principal component analysis (PCA) is utilized. Finally, two kernel-based classifiers, i.e., Support
Feature analysis with ROC
Receiver operating characteristic (ROC) curve is usually applied as a visual approach to demonstrate the relationship between true positive rate and false positive rate along with the change of a threshold parameter. Three driving conditions, i.e., rest, city and highway, were divided into three 2-class cases, i.e., rest to others, city to others, and highway to others. Fig. 6(a–c) are ROC curves for each bio-signals and the fusion of all signals; Fig. 6(d–f) are ROC curves for time, spectral
Comparisons with similar work
This research, presented an analysis of multimodal features of physiological signals, to identify drive-related stress symptom. Then, it was proposed an automatic detection system to distinguish three different stress levels and provide detailed stress assessment during continuous drive tasks. Three important conclusions could be drawn from this study: 1) different levels of driving stress could be characterized by specific set of physiological measures. 2) multimodal features are favorable for
Conclusions
Monitoring drivers’ internal status has great potential in declining the occurrence probability of traffic accidents. The present research developed an automatic system for detecting the driving related stress level based on multichannel physiological records. A variety of features were extracted using wavelet decomposition, time and spectral analysis. Sparse Bayesian Learning (SBL) and Principal Component Analysis (PCA) were combined and adopted to search for the optimal and compact feature
Acknowledgement
This work is partly supported by National Natural Science Foundation of China (Nos. 61201124, 51407078) and Fundamental Research Funds for the Central Universities WH1414022.
References (54)
- et al.
Improving BCI-based emotion recognition by combining EEG feature selection and kernel classifiers
Expert Systems with Applications
(2016) - et al.
Fully automated real time fatigue detection of drivers through fuzzy expert systems
Applied Soft Computing
(2014) - et al.
Automatic detection of alertness/drowsiness from physiological signals using wavelet-based nonlinear features and machine learning
Expert Systems with Applications
(2015) - et al.
Hybrid computer vision system for drivers' eye recognition and fatigue monitoring
Neurocomputing
(2014) - et al.
Driving performance in cold, warm, and thermoneutral environments
Applied Ergonomics
(2003) - et al.
Sleepy driving on the real road and in the simulator-A comparison
Accident Analysis & Prevention
(2013) - et al.
Driver distraction: The effects of concurrent in-vehicle tasks, road environment complexity and age on driving performance
Accident Analysis & Prevention
(2006) - et al.
Assessment of muscle fatigue in low level monotonous task performance during car driving
Journal of Electromyography and Kinesiology
(2005) - et al.
Extreme learning machine: Theory and applications
Neurocomputing
(2006) - et al.
Detecting driver drowsiness using feature-level fusion and user-specific classification
Expert Systems with Applications
(2014)