Robust remote heart rate estimation from multiple asynchronous noisy channels using autoregressive model with Kalman filter
Introduction
Contactless physiological monitoring techniques open up many possibilities for easy and convenient continuous monitoring without hindering the daily-life routines of the monitored subjects. Traditional methods for physiological monitoring require sensors to be physically attached to the subject, such as photoplethysmography (PPG) finger-clip sensors, electrocardiogram (ECG) electrodes, and respiration (RSP) chest belts. Often they cause skin irritation and discomfort, in addition to hindering mobility in daily activities.
Many commercial mobile devices nowadays come with a built-in digital camera. The ubiquitousness of such devices allows ordinary commercial mobile devices such as smartphones, tablets, and laptops, to be employed for remote diagnosis, or telehealthcare. The main interest in developing remote diagnosis methods using commercial hardware is the low-cost due to not having to purchase specialized medical equipment and the convenience of having to use a readily available device. This is especially useful for telehealthcare in rural areas, where access to clinics or specialized medical equipment may be nonexistent. However, mobile devices such as smartphones can be used for self-diagnosis including heart rate monitoring [1], colorimetric tests [2], and diabetes management [3].
Identifying the underlying heart rate signal from a video recording of a subject is a challenging proposition. Under optimum conditions, the lighting and environment should ideally be static, and the video source should be recording using high-quality digital cameras. Practically, however, noise from video compression, body movements, and dynamic illumination introduces interference and may result in an inaccurate estimation of the heart rate signal [4]. Most computational estimation methods include frequency filtering to exclude signal frequencies that are above or below that of heart rate signals but are unable to account for noise which is very similar to the heart rate signal itself. In addition, identifying the underlying heart rate signal using computational estimation methods [5], [6] typically selects only a single estimated outcome to represent the closest fit to the actual heart rate signal.
In this research, we introduce a contactless heart rate estimation schema that improves the accuracy of the measurement, compares to other methods. Additionally, the algorithm solves the challenges of unwanted signal influences including illumination variation, and rigid or non-rigid facial or environmental motions by using illumination rectification, non-rigid motion elimination, and few temporal filters. Despite the methods which are used to eliminate the effect of unwanted sources, some undetectable sources may sabotage the physiological signal in a realistic situation. Consequently, a data fusion algorithm is employed to adjust real-time measures based on the multiple measured heart rates and previous estimation.
The method employes various techniques to minimize the effect of rigid and non-rigid motions on the estimation and uses the RADICAL technique to extract independent components which outperform other ICA algorithms. Moreover, we proposed to estimate multiple heart rate measures from three different regions of the human face as well as three independent components resulted from applying ICA. So multiple measurements have been used in a proposed regression model which avoid sudden motions and noises and reduce the effect of rigid and non-rigid motions. The proposed technique increases the accuracy of the algorithm enormously as shown in the experiments.
The algorithm starts with extracting regions of interest using facial points. The physiological signals are calculated using the extracted regions. Then, an independent component analysis technique is adapted to extract subcomponents. The subcomponents are processed separately to obtain multiple heart rate measures which are later used in the data fusion method. The resulted components are transformed to the frequency domain using Fourier transformation. Afterward, peak points are obtained from the frequency signals with minimum peak distance of predefined length. Later, the frequency with the highest magnitude is chosen as the frequency of the heart rate for each signal. Finally, the heart rate measures are fused using the data fusion technique proposed in this paper, to calculate more accurate results.
In Section 2, we present a number of related works. In Section 3, we present the proposed methodology for heart rate estimation, including a method for tracking regions of interest (ROIs) of the subject's face and an algorithm to compensate for motion. In Section 4, we present a proposed data fusion method to improve the accuracy of the heart rate estimation method using the current and previous heart rate measures. In Section 5, we present the results of three experiments that are designed to test and validate our methodology. In Section 6, we discuss the conclusions drawn from the experiment.
Section snippets
Related works
The innovation of using digital cameras in mobile devices for measuring heart rate (HR) is an extension of PPG technology. A pulse oximeter device for measuring PPG illuminates the skin and measures the variations in light absorption corresponding to a change in blood pressure from heartbeats. Early PPG sensors require LEDs to illuminate the skin and a photodiode at close range for measurement.
To extract the underlying physiological signals from the illumination of the skin, several methods can
Heart rate estimation
The proposed technique in this paper is composed of a number of steps as shown in Fig. 1. In the first step, ROIs on the subject's face is located by using Constrained Local Model (CLM) presented in [17]. CLM algorithm extracts facial landmarks and generates a mask based on extracted points. Kanade-Lucas-Tomasi (KLT) algorithm [18] then is employed to track the location of featured landmarks in further image frames. The facial space is divided into three regions including the forehead, left and
Channel fusion and regression model
The heart rate estimation algorithm may face many challenges in a real situation such as illumination variation, shadow changes, and facial movements which can disturb the physiological signals and may increase estimation error. Many of the environmental changes are hard or impossible to eliminate, for example, micro-changes, or vibrations that their frequency is close to heart rate frequency, are impossible to remove and have an enormous effect on the result. In this paper, a data fusion
Experiment
We evaluate our algorithm using three experiments. The first experiment demonstrates the performance of all components inside the algorithm. Moreover, it evaluates the algorithm in dynamic environments. We investigate how much facial expressions, head movements, and illumination variation can affect the results and discuss the efficiency of the employed methods to deal with the changes. The second experiment evaluates the algorithm using the DEAP benchmark database presented in [33]. The
Conclusion
Previous algorithms can measure heart rate from face videos with high accuracy under conditions with high restriction of motions and illumination variation. The methods are not able to deal with environmental changes in real situations which prevents them to be utilized in real life scenarios. We proposed a novel heart rate estimation algorithm that can reduce environmental interferences. A method based on Normalized Least Mean Square adaptive filter is employed to reduce the effect of rigid
Acknowledgment
This work is sponsored by the Faculty of Computer Science and Information Technology, University of Malaya, Malaysia. UM, grand challenge's project number is GC003A-14HTM.
References (34)
- et al.
Quantifying colorimetric tests using a smartphone app based on machine learning classifiers
Sens. Actuators B: Chem.
(2018) - et al.
Pca-based magnification method for revealing small signals in video
Signal Image Video Process.
(2018) - et al.
Non-contact heart rate and heart rate variability measurements: a review
Biomed. Signal Process. Control
(2014) - et al.
Efficient constrained local model fitting for non-rigid face alignment
Image Vis. Comput.
(2009) - et al.
Fast fourier transforms: a tutorial review and a state of the art
Signal Process.
(1990) - et al.
A highly efficient and reliable heart rate monitoring system using smartphone cameras
Multimed. Tools Appl.
(2017) - et al.
Use of a novel smartphone-based diabetes management system improved feelings of confidence and safety and reduced hypoglycemia fear among parents/caregivers of children/adolescents with type 1 diabetes
J. Diabetes Sci. Technol.
(2017) - et al.
Non-contact, automated cardiac pulse measurements using video imaging and blind source separation
Optics Express
(2010) - et al.
Detecting pulse from head motions in video
- et al.
Contactless multiple wavelength photoplethysmographic imaging: a first step toward “spo2 camera” technology
Ann. Biomed. Eng.
(2005)
Noncontact simultaneous dual wavelength photoplethysmography: a further step toward noncontact pulse oximetry
Rev. Sci. Instrum.
Feasibility of Imaging Photoplethysmography
Remote plethysmographic imaging using ambient light
Optics Express
Advancements in noncontact, multiparameter physiological measurements using a webcam
IEEE Trans. Biomed. Eng.
Improvements in remote cardiopulmonary measurement using a five band digital camera
IEEE Trans. Biomed. Eng.
Rapid object detection using a boosted cascade of simple features
An extended set of haar-like features for rapid object detection
Cited by (15)
Illumination variation-resistant video-based heart rate monitoring using LAB color space
2021, Optics and Lasers in EngineeringCitation Excerpt :Chen et al. only selected the gray level of the green channel to extract PPG signals, and introduced the ensemble empirical mode decomposition of the Hilbert- Huang transform to reduce the effect of ambient light changes [23]. In addition to the above methods, many new related frameworks and algorithms have also been proposed for improving the measurement accuracy of the rPPG method [24–39], and the detailed research and developments of this subject can be referred to the recent review papers [3,5,40]. In this paper, a new framework for measurement of heart rate in LAB color space is proposed.
An Integrated Framework for Assessing the Quality of Non-invasive Fetal Electrocardiography Signals
2024, Journal of Medical and Biological EngineeringRobots and Intelligent Medical Devices in the Intensive Care Unit: Vision, State of the Art, and Economic Analysis
2023, IEEE Transactions on Medical Robotics and BionicsMSDN: A Multistage Deep Network for Heart-Rate Estimation From Facial Videos
2023, IEEE Transactions on Instrumentation and MeasurementObtaining Neck Weak Pulse Signal Using Multi-Region Dominant Frequency Enhancement Method
2023, Zhongguo Jiguang/Chinese Journal of LasersObtaining pulse wave signal from multi-frame image sequence of neck
2022, Guangxue Jishu/Optical Technique