Imaging and fusing time series for wearable sensor-based human activity recognition
Introduction
In our data-driven and data-rich society (e.g., real-time video feeds from CCTVs, and other sensing data from different data sources), Body Sensor Networks (BSN) [1] have gained widespread attention, including in the academic literature, such as those of human-computer interaction and ubiquitous computing (e.g., user identification [2] and human activity recognition [3]). Constant advances in hardware and software (e.g., inexpensive mobile devices with embedded powerful sensors and wireless technology [4], [5]) have also eased human activity recognition using sensor data from BSN, and example applications include healthcare [6], [7], heart-rate-based emotion reactions [8], [9], activity monitoring [10], [11], and commercial applications such as fitness tracking [12] and signal processing in node environment [13]. For example, human activity recognition can leverage time series signal of mobile device sensor data, where representative data features are extracted for classification and discrimination using various algorithms.
Traditionally, human activity recognition using mobile device sensors has been defined as a multivariate time series classification problem. To solve the problem, a key step is feature extraction, for example relying on some statistical features of the raw signal (e.g., variance, mean, entropy, and correlation coefficients) [14], or including some cross-formal coding (e.g., signals with Fourier transform and wavelet transform). These heuristic features are widely used in analyzing time series data.
However, in the deep learning framework, we can build a multi-layer deep structure to automatically extract relevant features. A deep learning model can train data in both supervised and unsupervised manner, and it has significant effect in processing graphical data. Moreover, the representation of features of time series has recently attracted widespread attention. The most successful way is to describe features as visual cues [15]. Depending on supervisory and non-hyper-visual learning techniques in computer vision, time series can be re-coded into images to enable machines to perform image recognition. This technology has been applied in speech recognition [16], classification [17] and radio frequency identification [18], and has shown to be more effective.
Therefore, the proposed architecture in this work integrates a method that transforms sensor data into some visual images, and a framework that enables human activity recognition to be carried out by using deep residual networks in image recognition. Specifically, we summarize the key contributions of this paper to be as follows.
- •
A feature engineering method is developed to transfer sensor-based time series data into different images, by unifying the global and local features from time series.
- •
A fusion framework is proposed to automatically extract image features from the generated images and to recognize user behavior by distinguishing different image features.
Now, we will describe the layout for the remaining of this paper. In the next section, we will briefly review related work. Our proposed approach is described in Section 3. In Section 4, we evaluate the proposed framework on both HHAR [19] and MHEALTH datasets [20] and describe the findings. Specifically, the findings demonstrate that our proposed approach works well with most types of heterogeneous multi-dimensional time series measurements. Finally, we conclude the paper in the last section.
Section snippets
Related work
In this section, we will briefly review the related literature on imaging time series, image recognition search, and heterogeneous sources processing.
Imaging Time Series. Encoding time series as images plays an important role in many classification tasks. In [21], for example, the authors investigated the use of recurrence plots as data representation for time series classification. In their approach, texture features are extracted on recurrence patterns from time series by applying visual
Proposed architecture
In this section, we will first perform a preliminary investigation on GAF and ResNet. Then for our activity recognition challenge, we will explore the feature engineering in GAF Images and present the proposed fusion ResNet framework.
Evaluation
The following datasets are used in our evaluation.
HHAR. The Heterogeneity Human Activity Recognition (HHAR) dataset from smartphones and smartwatches is a dataset, which has been devised to benchmark human activity recognition algorithms (classification, automatic data segmentation, sensor fusion, feature extraction, etc.) in real-world contexts; specifically, the dataset is gathered with a variety of device models and use-scenarios, in order to reflect sensing heterogeneity to be expected in
Conclusion
In this paper, a deep learning network architecture was proposed for human activity recognition based on mobile sensor data. Specifically, we proposed a novel method to encode time series into GAF images by unifying global and local time series features. This new processing method can be trained in mainstream image recognition residual networks. We designed a number of experiments to verify the feasibility of imaging time series. And we proposed a fusion ResNet to solve the problem of
Acknowledgment
This work was supported in part by the National Natural Science Foundation of China (No. 61672135), the National Science Foundation of China - Guangdong Joint Foundation (No. U1401257), the Sichuan Science-Technology Support Plan Program (No. 2018GZ0236 and No. 2017FZ0004), and the Fundamental Research Funds for the Central Universities (No. 2672018ZYGX2018J057 and No. ZYGX2015KYQD136).
References (43)
- et al.
Multi-sensor fusion in body sensor networks: state-of-the-art and research challenges
Inform. Fusion
(2017) - et al.
Synergy of big data and 5g wireless networks: opportunities, approaches, and challenges
IEEE Wirel. Commun.
(2018) - et al.
Automatic methods for the detection of accelerative cardiac defense response
IEEE Trans. Affect. Comput.
(2016) - et al.
Human activity recognition using multi-features and multiple kernel learning
Pattern Recognit.
(2014) Long short-term memory
Neural Comput.
(1997)- et al.
Learning aided user identification using smartphone sensors for smart homes
IEEE Internet Thing. J.
(2019) - et al.
Optimizing multi-sensor deployment via ensemble pruning for wearable activity recognition
Inform. Fusion
(2018) - et al.
S2m: A lightweight acoustic fingerprints-based wireless device authentication protocol
IEEE Internet Thing. J.
(2017) - et al.
A framework for creating healthcare monitoring applications using wireless body sensor networks
Proceedings of the ICST 3rd international conference on Body area networks
(2008) - et al.
Gaitassist: A daily-life support and training system for parkinson’s disease patients with freezing of gait
ACM Conference on Human Factors in Computing Systems
(2014)