Classification of physiological signals for wheel loader operators using Multi-scale Entropy analysis and case-based reasoning

doi:10.1016/j.eswa.2013.05.068

Expert Systems with Applications

Volume 41, Issue 2, 1 February 2014, Pages 295-305

https://doi.org/10.1016/j.eswa.2013.05.068 Get rights and content

Abstract

Sensor signal fusion is becoming increasingly important in many areas including medical diagnosis and classification. Today, clinicians/experts often do the diagnosis of stress, sleepiness and tiredness on the basis of information collected from several physiological sensor signals. Since there are large individual variations when analyzing the sensor measurements and systems with single sensor, they could easily be vulnerable to uncertain noises/interferences in such domain; multiple sensors could provide more robust and reliable decision. Therefore, this paper presents a classification approach i.e. Multivariate Multiscale Entropy Analysis–Case-Based Reasoning (MMSE–CBR) that classifies physiological parameters of wheel loader operators by combining CBR approach with a data level fusion method named Multivariate Multiscale Entropy (MMSE). The MMSE algorithm supports complexity analysis of multivariate biological recordings by aggregating several sensor measurements e.g., Inter-beat-Interval (IBI) and Heart Rate (HR) from Electrocardiogram (ECG), Finger Temperature (FT), Skin Conductance (SC) and Respiration Rate (RR). Here, MMSE has been applied to extract features to formulate a case by fusing a number of physiological signals and the CBR approach is applied to classify the cases by retrieving most similar cases from the case library. Finally, the proposed approach i.e. MMSE–CBR has been evaluated with the data from professional drivers at Volvo Construction Equipment, Sweden. The results demonstrate that the proposed system that fuses information at data level could classify ‘stressed’ and ‘healthy’ subjects 83.33% correctly compare to an expert’s classification. Furthermore, with another data set the achieved accuracy (83.3%) indicates that it could also classify two different conditions ‘adapt’ (training) and ‘sharp’ (real-life driving) for the wheel loader operators. Thus, the new approach of MMSE–CBR could support in classification of operators and may be of interest to researchers developing systems based on information collected from different sensor sources.

Introduction

Operating working machines in construction, mining, agriculture and forestry requires much mental and physical effort. The efficiency of these machines depends on the performance of the human operators/drivers. Monitoring and diagnosing the operators when they are exhausted with mental/physical workload is important feedback for an operator, especially a professional operator where an accident could have large consequences both on lives and economical costs. However, identification of mental/physical state and generating alarm due to stress, sleepiness, fatigue etc. is difficult while driving and still a scientific challenge. Today, different sensors enable clinician to determine psychological status with high accuracy. However, since there are large individual variations, analyzing data from a single sensor source could deteriorate the classification result. Data that are collected from multiple sensor sources could provide us more reliable and robust information of these psychophysiological parameters. For instance, if one sensor measurement is influenced by a certain noise or interference other sensor measurements could still support the system. As human beings, we have the natural ability to fuse signals that are coming from different sources and supports in reliable and feature-rich judgment. Using multiple sensor signals to achieve more reliable assessment of diagnosis this is what (i.e., naturally performed multisensory data fusion) experts’ are doing in real life while diagnosing these psychophysiological parameters.

This paper investigates sensor signal fusion in a case-based classification scheme by means of MMSE algorithm (Ahmed and Mandic, 2011, Ahmed and Mandic, 2012). Here, five sensor measurements i.e., Inter-beat-Interval (IBI) and Heart Rate (HR) from Electrocardiogram (ECG), Finger Temperature (FT), Skin Conductance (SC) and Respiration Rate (RR) are combined at low-level fusion applying MMSE algorithm to classify physiological parameters i.e., stressed or healthy of wheel loader operators. The proposed system has been evaluated with the data collected from 18 wheel loader operators. The main goal is to investigate whether the proposed system MMSE–CBR is able to classify the operators despite large individual variations and noises/interferences of the environment.

The rest of the paper is organized as follows: Section 2 presents the background of physiological sensor signals fusion. It also discusses about CBR and entropy analysis. Section 3, describes related work. Section 4, illustrates the study design for the data collection. In Section 5, the methods are described in detail. The experimental work is presented in Section 6. Section 7 discusses the evaluation results. Finally, Section 8 ends with concluding remarks.

Section snippets

Sensor signal fusion

Sensor signal fusion is a method that gives us the resulting information while using multiple sensors. According to Wilfried (2001), “Sensor fusion is the combining of sensory data or data derived from sensory data such that the resulting information is in some sense better than would be possible when these sources were used individually”. Commonly, sensor signal fusion can be achieved either by combining multiple sensor data sources or data from a single source over a period of time could also

Related work

Sensor data fusion is becoming an emerging technology. In the health sciences, synergy between CBR and other techniques is common (Ahmed et al., 2012, Begum et al., 2011, Bichindaritz, 2007, Bichindaritz and Marling, 2006, Marling et al., 2011, Montani et al., 2006, Perner, 2006, Perner et al., 2006). Nevertheless, following are some of the few attempts that apply fusion techniques in the CBR systems. Case-based data fusion to improve clinical decision support in coronary heart disease is

Study design and data collection

Psychophysiological measurements of wheel loader operators have been collected in two phases; (1) during conducting a Psychophysiological Stress Profile (PSP) and (2) during operating a wheel loader. During the study each operator was given an exclusive 2.5 h session, starting with the Psychophysiological Stress Profile (PSP) described in Section 4.1. Afterwards, the machine testing was performed, with the traction force setting as the independent variable. In order to minimize the skewing

Methods

The methods are presented in two-folds, (1) presents the proposed approach i.e. MMSE–CBR and (2) other traditional CBR approaches.

Evaluation

The classification accuracy has been evaluated considering the 2 data sets as discussed earlier; (1) PSP data set and (2) training and testing wheel loader data set i.e., sharp and adapt. An evaluation of CBR system classification integrating sensor fusion (MMSE–CBR) has been conducted using each data set. Finally, the proposed approach is compared with other single source system and fusion methods i.e. (3) MMSE–CBR and other single source parameters and fusions. Note that, only PSP data have

Experimental results and discussions

As discussed in the previous section several experimental works have been carried out to evaluate the performance of the proposed system that employs sensor fusion method in a case-based classification system for the wheel loader operators. Since, in reality clinicians make decision based on effectively fusing the information collected from different physiological sensor sources the main goal was to investigate whether in such classification systems it could also classify the operators

Summary and conclusions

This paper presented a new approach for multi-sensor fusion in a CBR system for classification of wheel loader operators based on 5 sensor measurements i.e., IBI, HR, FT, SC and RR. Multivariate and multi-scale entropy was applied to perform a low level or data level sensor fusion in the system. In the proposed system, the features extracted from the entropy estimation were used to formulate cases in a CBR system. The result shows that the system could classify ‘stressed’ and ‘healthy’ subjects

Acknowledgments

The authors would like to acknowledge the Swedish Knowledge Foundation (KKs) and Volvo Construction Equipment AB, Sweden for their support of this research.

References (44)

S. Ataei et al.
Sensor fusion of a railway bridge load test using neural networks
Expert Systems with Applications
(2005)
I. Bichindaritz et al.
Case-based reasoning in the health sciences: What’s next?
Artificial Intelligence in Medicine
(2006)
F. Cremer et al.
A comparison of decision-level sensor-fusion methods for anti-personnel landmine detection
Information Fusion
(2001)
M. Grbovic et al.
Decentralized fault detection and diagnosis via sparse PCA based decomposition and maximum entropy decision fusion
Journal of Process Control
(2012)
J.L. Kolodner
Reconstructive memory: A computer model
Cognitive Science
(1983)
S. Montani et al.
Case-based retrieval to support the treatment of end stage renal failure patients
Artificial Intelligence in Medicine
(2006)
P. Perner et al.
Recognition of airborne fungi spores in digital microscopic images
International Journal of Artificial Intelligence in Medicine
(2006)
J. Rodger
Toward reducing failure risk in an integrated vehicle health maintenance system: A fuzzy multi-sensor data fusion Kalman filter approach for IVHMS
Expert Systems with Applications
(2012)
A. Aamodt et al.
Case-based reasoning: Foundational issues, methodological variations, and system approaches
AICom – Artificial Intelligence Communications
(1994)
Ahmed, M. U., Islam, A. M., & Loutfi, A. (2012). A case-based patient identification system using pulse oximeter and a...

M. Ahmed et al.

Multivariate multiscale entropy: A tool for complexity analysis of multichannel data

Physical Review E

(2011)

M.U. Ahmed et al.

Multivariate multiscale entropy analysis

IEEE Signal Processing Letters

(2012)

S. Begum et al.

Multi-scale entropy analysis and case-based reasoning to classify physiological sensor signals

S. Begum et al.

Case-based reasoning systems in the health sciences: A survey on recent trends and developments

International journal of IEEE Transactions on Systems, Man, and Cybernetics-Part C: Applications and Reviews

(2011)

S. Begum et al.

A case-based decision support system for individual stress diagnosis using fuzzy similarity matching

Computational Intelligence

(2009)

Begum, S., Ahmed, M., Funk, P., Xiong, N., & Scheele, B. V. (2006). Using calibration and fuzzification of cases for...

Begum, S., Islam, M., Ahmed, M., & Funk, P. (2011). K-NN based interpolation to handle artifacts for heart rate...

Begum, S., Ahmed, M. U., Funk, P., & Filla, R. (2012). Driver’s mental state monitoring system using cbr based on heart...

Begum, S., Ahmed, M., Funk, P. (2010). ECG Sensor Signal Analysis to Represent Cases in a Case-based Stress Diagnosis...

Ben, M. B., & Ranchin, T. (2006). Case based reasoning data fusion scheme: application to offshore wind energy resource...

I. Bichindaritz

Prototypical case mining from medical literature

Applied Intelligence

(2007)

M. Costa et al.

Multiscale entropy analysis of complex physiologic time series

Physical Review Letters

(2002)

Cited by (27)

Non-contact-based driver's cognitive load classification using physiological and vehicular parameters
2020, Biomedical Signal Processing and Control
Citation Excerpt :
These features are presented in Table 4. These HRV features are used for cognitive load classification [33,56,57] and stress classification [58–60]. A summary of all features are presented in Table 1.
Classification of cognitive load for vehicular drivers is a complex task due to underlying challenges of the dynamic driving environment. Many previous works have shown that physiological sensor signals or vehicular data could be a reliable source to quantify cognitive load. However, in driving situations, one of the biggest challenges is to use a sensor source that can provide accurate information without interrupting diverging tasks. In this paper, instead of traditional wire-based sensors, non-contact camera and vehicle data are used that have no physical contact with the driver and do not interrupt driving. Here, four machine learning algorithms, logistic regression (LR), support vector machine (SVM), linear discriminant analysis (LDA) and neural networks (NN), are investigated to classify the cognitive load using the collected data from a driving simulator study. In this paper, physiological parameters are extracted from facial video images, and vehicular parameters are collected from controller area networks (CAN). The data collection was performed in close collaboration with industrial partners in two separate studies, in which study-1 was designed with a 1-back task and study-2 was designed with both 1-back and 2-back task. The goal of the experiment is to investigate how accurately the machine learning algorithms can classify drivers’ cognitive load based on the extracted features in complex dynamic driving environments. According to the results, for the physiological parameters extracted from the facial videos, the LR model with logistic function outperforms the other three classification methods. Here, in study-1, the achieved average accuracy for the LR classifier is 94% and in study-2 the average accuracy is 82%.
In addition, the classification accuracy for the collected physiological parameters was compared with reference wire-sensor signals. It is observed that the classification accuracies between the sensor and the camera are very similar; however, better accuracy is achieved with the camera data due to having lower artefacts than the sensor data.
Automatic driver sleepiness detection using EEG, EOG and contextual information
2019, Expert Systems with Applications
Citation Excerpt :
Longer blink durations (Åkerstedt, Peters, Anund, & Kecklund, 2005; Häkkänen, LicPsych, Summala, Partinen, Tiihonen, & Silvo, 1999; Schleicher, Galley, Briest, & Galley, 2008) and slow eye movements (Kurt, Sezgin, Akin, Kirbas, & Bayram, 2009), measured via EOG, are other indictors of sleepiness. Also HRV contributes with information related to sleepiness (Vicente, Laguna, Bartra, & Bailón, 2016), but it is more commonly used to measure driver fatigue and stress (s Begum, Ahmed, Funk, & Filla, 2012; Begum, Barua, Filla, & Ahmed, 2014). Contextual information tries to account for some of the many confounding factors that affect sleep and sleepiness (Åkerstedt et al., 2004; Cooke & Ancoli-Israel, 2011; Menefee et al., 2000; Reyner, Wells, Mortlock, & Horne, 2012).
The many vehicle crashes that are caused by driver sleepiness each year advocates the development of automated driver sleepiness detection (ADSD) systems. This study proposes an automatic sleepiness classification scheme designed using data from 30 drivers who repeatedly drove in a high-fidelity driving simulator, both in alert and in sleep deprived conditions. Driver sleepiness classification was performed using four separate classifiers: k-nearest neighbours, support vector machines, case-based reasoning, and random forest, where physiological signals and contextual information were used as sleepiness indicators. The subjective Karolinska sleepiness scale (KSS) was used as target value. An extensive evaluation on multiclass and binary classifications was carried out using 10-fold cross-validation and leave-one-out validation. With 10-fold cross-validation, the support vector machine showed better performance than the other classifiers (79% accuracy for multiclass and 93% accuracy for binary classification). The effect of individual differences was also investigated, showing a 10% increase in accuracy when data from the individual being evaluated was included in the training dataset. Overall, the support vector machine was found to be the most stable classifier. The effect of adding contextual information to the physiological features improved the classification accuracy by 4% in multiclass classification and by and 5% in binary classification.
An intelligent fault diagnosis model for rotating machinery based on multi-scale higher order singular spectrum analysis and GA-VPMCD
2016, Measurement: Journal of the International Measurement Confederation
Citation Excerpt :
The simulation analysis results from Hénon map and Lorenz model show that HO-SSA is robust for embedding dimension and gives a higher tolerant to noise than SSA. Recently, since multi-scale analysis method [13] can extract more time mode information, it has been applied not only for physiological data analysis [13–16] but also for other fields, such as hydrodynamics [17] and machinery fault diagnosis [18–20]. These research results show multi-scale analysis can extract much more hidden information in different scales of signals.
Feature extraction and class discrimination are two key problems for fault diagnosis of rotating machinery. Firstly, multi-scale higher order singular spectrum analysis (MS-HO-SSA) method is presented and the multi-scale higher order singular spectrum entropy (MSHOSSE) is defined as feature to reveal the non-Gaussian and nonlinear characteristic for the vibration signals from rotating machinery with local faults. Secondly, GA-VPMCD method is presented by combination genetic algorithm (GA) with conventional variable predictive model based class discriminate (VPMCD) approach. Lastly, an intelligent fault diagnosis model based on MS-HO-SSA and GA-VPMCD is put forward and utilized for rotor fault diagnosis. The experimental results show that MS-HO-SSA method is more effective for feature extraction and the GA-VPMCD provides better performance than conventional VPMCD and LSSVM.
Improving biomedical signal search results in big data case-based reasoning environments
2016, Pervasive and Mobile Computing
Citation Excerpt :
One major drawback of CBR systems is that the knowledge base must contain relevant cases to correctly make decisions on a current case [3,4]. This is especially difficult for high dimensional measured signals, such as Electrocardiogram (ECG) [5] and accelerometers which exhibit high variability due to noise, sensor displacement, misuse, and other factors that are difficult to control [6]. Systems hope to use such information to accurately classify signals and patients for an accurate medical diagnosis [7], dealing with the complexities as well as the potential issues of missing data [8].
Time series subsequence matching has importance in a variety of areas in healthcare informatics. These include case-based diagnosis and treatment as well as discovery of trends among patients. However, few medical systems employ subsequence matching due to high computational and memory complexities. This paper proposes a randomized Monte Carlo sampling method to broaden search criteria with minimal increases in computational and memory complexities over $R$ -NN indexing. Information gain improves while producing result sets that approximate the theoretical result space, query results increase by several orders of magnitude, and recall is improved with no significant degradation to precision over $R$ -NN matching.
Driving Pattern Classification for Wheel Loaders in Different Material Handling Using Machine Learning
2023, IEEE Conference on Intelligent Transportation Systems, Proceedings, ITSC
Driver Cognitive Loads Based on Multi-dimensional Information Feature Analysis
2021, Zhongguo Gonglu Xuebao/China Journal of Highway and Transport

View all citing articles on Scopus

View full text

Classification of physiological signals for wheel loader operators using Multi-scale Entropy analysis and case-based reasoning

Abstract

Introduction

Section snippets

Sensor signal fusion

Related work

Study design and data collection

Methods

Evaluation

Experimental results and discussions

Summary and conclusions

Acknowledgments

Expert Systems with Applications

Artificial Intelligence in Medicine

Information Fusion

Journal of Process Control

Cognitive Science

Artificial Intelligence in Medicine

International Journal of Artificial Intelligence in Medicine

Expert Systems with Applications

Case-based reasoning: Foundational issues, methodological variations, and system approaches

AICom – Artificial Intelligence Communications

Multivariate multiscale entropy: A tool for complexity analysis of multichannel data

Physical Review E

Multivariate multiscale entropy analysis

IEEE Signal Processing Letters

Multi-scale entropy analysis and case-based reasoning to classify physiological sensor signals

Case-based reasoning systems in the health sciences: A survey on recent trends and developments

International journal of IEEE Transactions on Systems, Man, and Cybernetics-Part C: Applications and Reviews

A case-based decision support system for individual stress diagnosis using fuzzy similarity matching

Computational Intelligence

Prototypical case mining from medical literature

Applied Intelligence

Multiscale entropy analysis of complex physiologic time series

Physical Review Letters