Elsevier

Expert Systems with Applications

Volume 85, 1 November 2017, Pages 279-291
Expert Systems with Applications

Detecting driving stress in physiological signals based on multimodal feature analysis and kernel classifiers

https://doi.org/10.1016/j.eswa.2017.01.040Get rights and content

Highlights

  • Automatic driving stress detection system in physiological records was proposed.

  • Features were extracted from multimodal analysis.

  • Efficient feature selection and reduction methods were employed.

  • Several kernel-based classifiers were adopted and compared.

  • Our proposed method shows a promising application in intelligent vehicle systems.

Abstract

Monitoring driving status has great potential in helping us decline the occurrence probability of traffic accidents and the aim of this research is to develop a novel system for driving stress detection based on multimodal feature analysis and kernel-based classifiers. Physiological signals such as electrocardiogram, galvanic skin response and respiration were record from fourteen drives executed in a prescribed route at real drive environments. Features were widely extracted from time, spectral and wavelet multi-domains. In order to search for the optimal feature sets, Sparse Bayesian Learning (SBL) and Principal Component Analysis (PCA) were combined and adopted. Kernel-based classifiers were employed to improve the accuracy of stress detection task. Analysis I used features from 10 s intervals of data which were recorded during well-defined rest, highway and city driving conditions to discriminate three levels of diving stress achieving an averaging accuracy over 99% at per-drive level and 89% in cross-drive validation. Analysis II made continuous stress evaluation throughout a complete driving test attaining a high coincidence with the true road situation especially at the switching interval of traffic conditions. Experimental results reveal that different levels of driving stress can be characterized by specific set of physiological measures. These physiological measures could be applied to in-vehicle intelligent systems in various approaches to help the drivers better manage their negative driving status. Our design scheme for driving stress detection could also facilitate the development of similar in-vehicle expert systems, such as driver's emotion management, driver's sleeping onset monitoring, and human-computer interaction (HCI).

Introduction

Some unfavorable mental states, e.g., fatigue (Cyganek and Gruszczyński, 2014, Lee and Chung, 2012, Liu et al., 2015, Yang et al., 2010), sleepiness (Åkerstedt et al., 2013, Hallvig et al., 2013, Morris et al., 2015); stress (Healey and Picard, 2005, Tamrin et al., 2014, Zheng et al., 2015) and distraction (Craye et al., 2015, Horberry et al., 2006) contribute to traffic accidents, leading to considerable number of vehicle crashes, injuries, and fatalities annually. In addition, physical discomfort such as extreme ambient temperature (Daanen et al., 2003, Pimenta and Assunção, 2015), uncomfortable driving position (Smith, Mansfield, Gyi, Pagett, & Bateman, 2015) and muscle fatigue in the neck/shoulder/back area (Hostens & Ramon, 2005) might impact driving behavior. Searching for the countermeasures to reduce the amount of traffic accidents and enhance public road safety has become an urgent issue for both governments and automakers. It is very crucial to develop an automatic system that intelligently detects the driver's unfit status and makes a warning once necessary.

For developing an automatic system to measure drivers’ workload and detect their internal status, it is crucial to select effective measurements. Usually, we can classify the measures into three categories: (1) vehicle behavior measures; (2) video-based measures; (3) physiological measures.

Vehicle behavior mainly includes: vehicle speed, acceleration, lane position deviation, steering, braking and gear changes (Horberry et al., 2006, Morris et al., 2015, Tamrin et al., 2014, Wang and Xu, 2015). These measures are easy to obtain however they are strongly dependent on the type of vehicle and the handling skill of drivers. Additionally, some vehicle features such as lane tracking technology are prone to data loss under the situation of lane markers missing, bad weather or darkness (Wang & Xu, 2015).

Video-based detectors that track head movements, monitor eye gaze status and interpret facial expression have been widely explored (Azim et al., 2014, Cyganek and Gruszczyński, 2014, Garcia et al., 2012, June, González-Ortega et al., 2013, Jo et al., 2014, Kholerdi et al., 2016, Zhang et al., 2012). However, visual features cannot always return reliable results especially in the conditions of poor light, night driving and wearing glasses. Besides, inaccurate interpretation of the facial expression might be encountered when some introverted subjects intend to control their emotions during the process of driving test.

Recently, features extracted from physiological signals such as electrocardiogram (EKG), galvanic skin response (GSR), respiration, electromyogram (EMG) and electroencephalogram (EEG) show relatively high identification accuracy and get insight into driver's states directly (Correa et al., 2014, Healey and Picard, 2005, Lee and Chung, 2012, Lee et al., 2014, Li and Chung, 2013, Li et al., 2012, Liu et al., 2015, Liu et al., 2010, Singh et al., 2013, Vicente et al., 2016, Wang et al., 2015, Zhao et al., 2012, Zhao et al., 2011). The disadvantage of these measurements is that they require sensors and cables attached on the body, which constrains the behavior of drivers in some extent. And the wireless and wearable acquisition solutions allow subjects participate the drive task more conveniently and comfortably (Fu and Wang, 2014, Lee et al., 2014). These physiological measures could be used automatically by in-vehicle intelligent systems in various approaches to help the drivers better manage their negative driving status.

It is a challenging and rewarding work for consecutive detecting of driver states in real driving environments. Computational techniques such as feature extraction, feature selection & reduction, and classification have the capacity to determine optimal sensors and automate driving status recognition.

  • (1)

    Many feature extraction methods have been developed for physiological signals such as waveform information (Correa et al., 2014, Healey and Picard, 2005), power spectral analysis (Vicente et al., 2016), wavelet coefficients (Li & Chung, 2013), and nonlinear analysis (Chen, Zhao, Zhang, & Zou, 2015).

  • (2)

    In order to reduce the dimensionality of feature sets or select the optimal features from the primary measures, Principal Component Analysis (PCA) (Wang, Lin, & Yang, 2013) and Linear Discriminant Analysis (LDA) (Correa et al., 2014) are the most commonly used. Besides, Mutual Information (MI) methods and Sparse Bayesian Learning (SBL) have been employed in the feature selection of Brain-Computer Interface (BCI) system and obtained outstanding performance (Atkinson and Campos, 2016, Hoffmann et al., 2008).

  • (3)

    Several modeling techniques have been developed to detect driving status such as support vector machines (Zhao et al., 2011, Zhao et al., 2009), artificial neural networks (Correa et al., 2014, Singh et al., 2013), fuzzy logic inference (Chiang, 2015) and Bayesian networks (Yang et al., 2010).

These computational techniques will be discussed and compared in the discussion part.

Commonly, stress is viewed as a response to particular events. It is a normal reaction of human body preparing itself in the face of difficulties with focus, strength and improved alertness. Usually, stress can be both physical and mental. If subjects are involved in a long, monotonous driving or under sleep deprivation condition, physical stress might become the major consideration. In this research, three well-defined driving conditions are designed mainly to induce different levels of mental stress. Furthermore, there is a difference between eustress and distress, where eustress is a positive stress helping individuals to better cope with the challenging situations. However, in this research, we focus on distress only, which is usually caused by increased drivers’ workload. A lot of literature has reported the relationship between driving distress and the diminished vehicle handling abilities, reduced alertness, and increased probability of traffic accidents (Bakker et al., 2011, Chiang, 2015, Okada et al., 2013, Sharma and Gedeon, 2012, Sun et al., 2010, Villarejo et al., 2012).

Stressful events could cause dynamic changes in autonomic nervous system (ANS), reflected as increased activity in sympathetic system and decreased activity in parasympathetic system. Physiological signals such as galvanic skin response (GSR), electrocardiogram (EKG), and electroencephalogram (EEG) are the main measures used for stress detection in literature. Heart rate variability (HRV) is a common non-invasive indicator to detect ANS activities and is used as a primary measure for stress detection in many monitoring systems (Chiang, 2015, Okada et al., 2013). HRV can dynamically reflect the accumulation of mental workload which makes it a good estimator for stress level. Short-term reduced HRV indicates acute stress, reflecting that stressful events negatively affect HRV. Skin conductance is another reliable measure for stress. Increased skin conductance reflects the individual under stress and reduced skin conductance indicates the individual under less-stressful situation (Bakker et al., 2011, Sharma and Gedeon, 2012, Sun et al., 2010, Villarejo et al., 2012). Other research demonstrates that relationships exist between brain activities and emotional stress. Stress assessment system has been developed using a new fusion link between EEG and peripheral signals such as blood volume pulse, skin conductance and respiration (Hosseini, Khalilzadeh, Naghibi-Sistani, & Homam, 2015).

Monitoring driving status has great potential in helping us decline the occurrence probability of traffic accidents. However, automatic detection is usually restricted to the stage of laboratory research mainly due to signal's features and noise, measurement constraints and subject-dependent issues. The aim of this paper is to develop an efficient system to determine driver's relative stress level during real-world driving tasks based on the analysis of physiological data. A novel system combining multimodal feature analysis and kernel-based classifiers is proposed. Considering the test differences between individual drivers and within individual drivers, two kinds of analysis are made: Analysis I used features from short intervals of well-defined data to discriminate three levels of diving stress at per-drive and cross-drive levels respectively. Analysis II made continuous stress evaluation throughout a complete driving test for each drive.

It is expected that: 1) different levels of driving stress could be characterized by specific set of physiological measures; 2) more features drawn from multimodal analysis are favorable for assessing the drive status more reliably and accurately; 3) the system combing efficient feature selection methods and kernel-based classifiers could enhance the capacity for identification of driver states.

In rest of this paper, Section 2 describes the materials including experiment design, participant information and data collection. Section 3 gives an introduction of our methodology including feature extraction, feature selection & reduction, and kernel-based classification technique. Section 4 presents our experimental results. Section 5 reviews the similar work and discusses the contributions and limitations of the present study. And the conclusion has been made in Section 6.

Section snippets

Experiment design

We use the database, contributed to PhysioNet by Healey & Picard, which collected a set of multiple physiological recordings from healthy subjects when they were driving along with a specified route in and around Boston.

The prescribed route was designed to make the drivers experience the situations which might produce different degrees of stress and also restore the stressful events encountered in daily commute. Fig. 1 shows the whole driving task which is mainly divided into six sections:

Methodology

The general block diagram of the proposed system is shown in Fig. 2. Physiological signals are acquired by multi-sensors. After the filtering and time segmentation, features extraction is executed including the wavelet decomposition, the time and the spectral analysis. The sparse Bayesian learning (SBL) is employed to select the optimal feature sets. To further reduce the feature dimensionality, principal component analysis (PCA) is utilized. Finally, two kernel-based classifiers, i.e., Support

Feature analysis with ROC

Receiver operating characteristic (ROC) curve is usually applied as a visual approach to demonstrate the relationship between true positive rate and false positive rate along with the change of a threshold parameter. Three driving conditions, i.e., rest, city and highway, were divided into three 2-class cases, i.e., rest to others, city to others, and highway to others. Fig. 6(a–c) are ROC curves for each bio-signals and the fusion of all signals; Fig. 6(d–f) are ROC curves for time, spectral

Comparisons with similar work

This research, presented an analysis of multimodal features of physiological signals, to identify drive-related stress symptom. Then, it was proposed an automatic detection system to distinguish three different stress levels and provide detailed stress assessment during continuous drive tasks. Three important conclusions could be drawn from this study: 1) different levels of driving stress could be characterized by specific set of physiological measures. 2) multimodal features are favorable for

Conclusions

Monitoring drivers’ internal status has great potential in declining the occurrence probability of traffic accidents. The present research developed an automatic system for detecting the driving related stress level based on multichannel physiological records. A variety of features were extracted using wavelet decomposition, time and spectral analysis. Sparse Bayesian Learning (SBL) and Principal Component Analysis (PCA) were combined and adopted to search for the optimal and compact feature

Acknowledgement

This work is partly supported by National Natural Science Foundation of China (Nos. 61201124, 51407078) and Fundamental Research Funds for the Central Universities WH1414022.

References (54)

  • W. Li et al.

    Evaluation of driver fatigue on two channels of EEG data

    Neuroscience Letters

    (2012)
  • LiuJ. et al.

    EEG-based estimation of mental fatigue by using KPCA–HMM and complexity parameters

    Biomedical Signal Processing and Control

    (2010)
  • D.M. Morris et al.

    Lane heading difference: An innovative model for drowsy driving detection using retrospective analysis around curves

    Accident Analysis & Prevention

    (2015)
  • A.M. Pimenta et al.

    Thermal discomfort and hypertension in bus drivers and chargers in the metropolitan region of Belo Horizonte, Brazil

    Applied Ergonomics

    (2015)
  • N. Sharma et al.

    Objective measures, sensors and computational techniques for stress recognition and classification: A survey

    Computer Methods and Programs in Biomedicine

    (2012)
  • R.R. Singh et al.

    A comparative evaluation of neural network classifiers for stress level analysis of automotive drivers using physiological signals

    Biomedical Signal Processing and Control

    (2013)
  • J. Smith et al.

    Driving performance and driver discomfort in an elevated and standard driving position during a driving simulation

    Applied Ergonomics

    (2015)
  • WangJ.S. et al.

    A k-nearest-neighbor classifier with heart rate variability feature-based transformation algorithm for driving stress recognition

    Neurocomputing

    (2013)
  • YangG. et al.

    A driver fatigue recognition model based on information fusion and dynamic Bayesian network

    Information Sciences

    (2010)
  • ZhaoC. et al.

    Electroencephalogram and electrocardiograph assessment of mental fatigue in a driving simulator

    Accident Analysis & Prevention

    (2012)
  • ZhaoC. et al.

    Multivariate autoregressive models and kernel learning algorithms for classifying driving mental fatigue based on electroencephalographic

    Expert Systems with Applications

    (2011)
  • T. Åkerstedt et al.

    Having to stop driving at night because of dangerous sleepiness–awareness, physiology and behaviour

    Journal of Sleep Research

    (2013)
  • J. Bakker et al.

    What's your current stress level? Detection of stress patterns from GSR sensor data

  • ChenH. et al.

    Robust fusion of uncertain information

    Systems, Man, and Cybernetics, Part B

    (2005)
  • H.S. Chiang

    Ecg-based mental stress assessment using fuzzy computing and associative petri net

    Journal of Medical and Biological Engineering

    (2015)
  • A.G. Correa et al.

    Automatic detection of drowsiness in EEG records based on multimodal analysis

    Medical Engineering & Physics

    (2014)
  • C. Cortes et al.

    Support-vector networks

    Machine Learning

    (1995)
  • Cited by (0)

    View full text