Keywords

1 Introduction

Nowadays, perceiving human activity is one of the vital areas of computer vision research. The goal of human activity recognition (HAR) is to distinguish and analyze activities based on data extracted from sensors such as wearables [17] or external sensing modalities. HAR can be used in smart-homes [18, 22], sports [16], as well as in health monitoring [19], assistance for the elderly [13] and in mental health care [8].

Recently, there has been an increase in the popularity of external sensors such as Kinect and RGB sensors over wearable sensors as they can be seen as less intrusive. In various computer vision applications, depth images have been used widely among researches. In [24], temporal motion energies were extracted from human activity depth images and a Hidden Markov Model (HMM) was applied to model the activities using depth image features [14]. In computer vision, 2D pose estimation is widely used and several algorithms have been proposed to localize body joints. Fujimori et al. developed a wearable suit to capture body motion with tactile sensors, using motion sensors to estimate the user’s orientation [10]. Liu et al. obtained static gestures from individual pictures by using skeletal tracking with a Kinect camera [15].

These conventional approaches of activity recognition, though suitable for a variety of tasks, are often slow and lack reliability and performance in complex environments. OpenPose [6, 23], an open source library developed at Carnegie Mellon University in 2017 achieved significant interest among researchers due to its computational performance for extraction of body joints. OpenPose can operate in real-time detecting facial expressions, body and hand joints by feeding RGB images through deep convolutional neural networks (CNNs) [20, 21].

In this work, we propose deep learning approach using OpenPose and recurrent neural nets (RNN) to facilitate activity recognition. The OpenPose library was used to detect 14 body joints. Using this data, we extract the changes in magnitude and angle between joints in consecutive frames. These robust features are then used as the input for a sequence classifier which uses an RNN with Long Short-term Memory cells (LSTM). As LSTMs have the ability to retain salient information over a sequence of time steps, it lends itself well to sequence classification tasks such as activity recognition. In this work, the LSTM is trained to recognize which activities are performed by learning the sequence of motion features associated with each activity. The main contributions of this work, therefore, are the extraction of robust motion features from the body joints acquired using OpenPose, combined with the use of LSTM-RNNs to recognize the activities and boost the performance compared to other conventional approaches such as SVM, Decision Trees, and Random Forests.

Fig. 1.
figure 1

Flow of work

Full size image

This paper is organized as follows: Sect. 2 describes our methodology, where we discuss the data set and the extraction of motion features, as well as give an overview of the LSTM architecture and the other classifiers compared with our experiments. Section 3 presents the experiments and results. The paper is concluded in Sect. 4 where we also discuss the limitations of our approach and possibilities for future work.

2 Methodology

In this section, we present the steps included in our proposed method of activity recognition. Specifically, Sect. 2.1 describes the extraction of key points using OpenPose and Sect. 2.2 defines the motion features. The data set and classifiers are described in Sects. 2.3 and 2.4, respectively. Figure 1 gives an overview of the general flow of our process.

2.1 Extracting Joints

OpenPose takes RGB images as input and generates 2-dimensional anatomical key-points for individual bodies detected in the image. The first stage in two-branched CNN predicts confidence maps, and the second stage predicts part affinity fields - a 2D vector which encodes the position and orientation of each limb [5].

Furthermore, both the confidence maps and affinity fields are parsed by greedy inference to visualize the 2D key-points of all individuals in the image [7, 23]. OpenPose generates the location of 18 body joints. These body joints can then be exported and used for applications such as gesture and activity recognition.

Fig. 2.
figure 2

OpenPose skeleton joints

Full size image

2.2 Motion Features

Fig. 3.
figure 3

Extracted body joints from several frames while performing jumping jacks

Full size image

In our work, we compute the two temporal features magnitude and angle from 14 body joints (L) between consecutive frames. Joints 14, 15, 16 and 17 pertain to the face and head as shown in Fig. 2. These key points are excluded from the data as they are not necessarily useful for activity recognition. Formally, we derive the magnitude M at a joint number N at time frame t as follows:

$$\begin{aligned} M_t^N= \sqrt{(L_{Nx(t+1)}-L_{Nx(t)})^2+(L_{Ny(t+1)}-L_{Ny(t)})^2} \end{aligned}$$
(1)

The angles of a body joint N for frame tth are computed as follows:

$$\begin{aligned} A_{N(x,y)t}= tan^{-1}(\frac{L_{Ny(t+1)}-L_{Ny(t)}}{L_{Nx(t+1)}-L_{Nx(t)}}) \end{aligned}$$
(2)

where \({L}_{Nx}\) and \({L}_{{Ny}}\) are the distances between the two-consecutive frames in x-axis and y-axis. For each example, we extract the body joints from a sequence of consecutive frames. Figure 3 shows the several frames of jumping jacks, while the dotted lines represent the joint-to-joint connection for motion features. Thus, the motion features (T) at time step t can be represented as:

$$\begin{aligned} T_t= [ M_t^N, A_{N(x,y)t} ] \end{aligned}$$
(3)
Fig. 4.
figure 4

Examples of images from the MHAD [1] database.

Full size image

2.3 Data Set

Our data set is a subset of the Berkeley Multimodal Human Action Database [1]. MHAD contains 11 actions, as listed in Table 1. These actions are performed by 7 male and 5 female subjects recorded using audio, video, accelerometers, motion capture, and kinect. For current work, we have chosen to use a subset containing the image sequences captured by 12 RGB cameras placed in clusters surrounding the participants, achieving views from the front and back as shown in Fig. 4. We use the images produced by the video recordings of all 12 subjects performing each action for approximately 5 s. The image sequences are captured by each of the four camera clusters as shown in Fig. 5. Cluster C1 and C2 contain four cameras. The remaining two clusters, C3 and C4 contain two cameras each. We include the images from all four clusters in an effort to make our system view-invariant, as each camera captures the video at a different angle.

The number of images in each recording varies from around 40 (approx. 3 s of video at 22 Hz) to 130 frames (approx. 10 s of video). Activities which involved less complex movements, such as standing up or sitting down, consist of fewer than 40 frames. To mediate this difference in sequence lengths and ensure consistency in the data-set, the longer sequences were clipped after 85 frames and the shorter sequences were extended by stacking them. Each camera captures 132 sequences. This results in a data-set of 1584 sequences of 85 images each consisting of 12 participants performing 11 actions captured by 12 cameras. Each sequence of images thereby represents the view captured by a single camera in one of the clusters and a single data point in training and test data. Finally, z-score standardization was applied before applying the classifiers to the data.

2.4 Classification

Recurrent Neural Networks (RNN). An RNN with LSTM cells was implemented in order to tackle the long term dependencies found in our data. RNNs and LSTMs have previously been shown to be effective in modeling temporal sequences such as those found in speech [11] and handwriting recognition [12] and also in music [9]. This is due to their ability to retain ‘memory’ over several time steps by allowing the states from preceding time steps to affect the RNNs current state. While this makes the architecture a good choice for modeling time series data, some limitations exist when dealing with longer time series. If the sequence length becomes too long, the RNN may suffer from vanishing or exploding gradients during back-propagation through time (BPTT). LSTMs mitigates this problem by adding multiple learnable parameters, or gates, which affect weight updates during BPTT enabling more control over what is retained in the internal state of the LSTM cell and what it ‘forgets’ at each time step. As illustrated in Fig. 6, the features which describe the variations of magnitude and angle of each joint between consecutive frames are fed to the network at each time step.

Table 1. Activity labels: 11 classes
Full size table
Fig. 5.
figure 5

Setup of cameras from Berkeley MHAD acquisition system [1]

Full size image

The model used in this work consist of two layers containing 256 LSTM cells with hyperbolic tangent activation function. The output layer is a dense layer with softmax activation and 11 units representing the different activities. Categorical cross entropy is used as the loss function during batch gradient descent and RMSprop with a learning rate of 0.001, is used as the optimizer. The model is trained for 300 epochs with a batch size of 256 samples and a 0.4 dropout rate. These hyperparameters were chosen empirically according to which values yielded the best results for the task.

Fig. 6.
figure 6

LSTM-RNN implementation for all activities

Full size image

Support Vector Machine (SVM). SVM has been widely used in HAR systems due to its high classification performance [2, 3]. SVM creates hyper-planes that maximize the margins between several classes, which enables maximum classification accuracy. The vectors are used to represent hyper-planes are called support vectors. By minimizing the cost function, an optimal solution can be obtained i.e. maximize the distance between hyper-plane and the nearest training point. Herein, a non-linear multi-class SVM with sigmoid kernel was used.

Decision Trees. A decision support tool that utilizes a model of decisions or tree-like graph and their possible consequences including utility, and chance event outcomes, called a decision tree. A decision tree is a well-known classifier in machine learning. Its structure is similar to the flowchart in which each internal node represents a test on an attribute; for instance, whether it would be heads or tails by flipping a coin. Each branch is responsible for the test’s outcome, and the class label would be represented by each leaf node. The decision would be taken after applying all features. The classification rules would be based on the paths from the root to the leaf [4].

Random Forests. Random Forests method used in both classification and regression problems. It generates multiple decision trees based on the random selection of variables and data, and recognizes the classes of dependent variables based on decision trees. In this work, 10 decision trees were used to explore the classes.

Fig. 7.
figure 7

Box plot showing the accuracies achieved by each model at each of the 5 folds

Full size image

3 Experiments and Results

In this section, the experiments performed to validate the performance of the proposed method are explained. The accuracy of each classifier is evaluated by performing stratified 5-fold cross-validation. The average accuracy of each classifier is listed in Table 2. Figure 7 shows the distribution of the results achieved by the different classifiers. Figure 8 and Table 3 displays confusion matrices and the precision and recall values generated from the predictions of each classifier from a single fold of the data. Moreover, Fig. 9 shows the confusion matrices of the number of samples classified as belong to each class.

Table 2. Average accuracy from 5-fold cross validation of each classifier
Full size table
Fig. 8.
figure 8

Confusion matrices for (a) SVM, (b) Decision Trees, (c) Random Forests and (d) LSTM calculated on a test set of 308 sequences. These show the percentage of predictions for each class

Full size image
Table 3. Precision and recall of Decision tree, SVM, Random Forest, and LSTM on a test set of 308 sequences
Full size table
Fig. 9.
figure 9

Confusion matrices showing the number of samples classified as belonging to each class

Full size image

As shown in Fig. 8(a), SVM was unable to distinguish the difference among waving one hand, waving both hands and clapping activities. Furthermore, Decision trees and Random forests showed better performance i.e. 0.66 and 0.78 as shown in Table 2. Moreover, both classifiers outperformed SVM in precision and recall, respectively. Still, both were lacking in their ability to differentiate all activities properly. The LSTM model outperformed the conventional approaches, achieving the highest average accuracy at 92,4%. Student’s t-test was applied to validate the statistically discriminant. The p values obtained LSTM versus other approaches were less than 0.05, thus proofing the statistical significance of the LSTM approach. It can be seen clearly from the accuracy achieved at each fold shown in the box plot in Fig. 8 that LSTM showed superior results compared to the other approaches, with a higher median and higher accuracy for all folds.

4 Conclusions

In this work, person independent and view-invariant activity recognition approach has been proposed. The OpenPose library was used to detect anatomical key points in the image sequences collected from the MHAD database. Afterward, temporal motion features were extracted from consecutive frames in each sequence. Lastly, different classifiers were applied to detect human activities. The classification accuracy of these models was compared to the proposed approach of using an LSTM-RNN. Our approach shows improved results compared to the conventional approaches, it is able to correctly classify activities performed by several different subjects and from various camera angles. Although OpenPose is able to detect several persons in a frame, our work will not be able to correctly classify the activity of several people at once. In future work, the proposed network would be implemented in a real-time system to detect different activities and gestures.