Sensor-based activity recognition of solitary elderly via stigmergy and two-layer framework

https://doi.org/10.1016/j.engappai.2020.103859Get rights and content

Abstract

With the acceleration of aging process of population structure, the single resident lifestyle is increasing on account of the high cost of care services and the privacy invasion concern. It is essential to monitor the activities of solitary elderly to find the emergency and lifestyle deviation, as independent life cannot be maintained due to physical or mental problems. The unobtrusive systems are the most preferred choice for the real-life long-term monitoring, while the camera and wearable devices based systems are not suitable due to the privacy and uncomfortableness, respectively. We propose a novel sensor-based activity recognition model based on the two-layer multi-granularity framework and the emergent paradigm with marker-based stigmergy. The stigmergy based marking subsystem builds features by aggregating the context-aware information and generating the two-dimensional activity pheromone trail. The two-layer framework consists of coarse-grained and fine-grained classification subsystems. The coarse-grained subsystem identifies whether the input completed activity segmented by the traditional method is easily-confused, and utilizes our generalized segmentation method to increase the inter-cluster distance. The fine-grained subsystem employs machine learning or deep learning classifiers to realize the activity recognition task. The proposed model is a data-driven model based on the information self-organization. It does not need sophisticated domain knowledge, and can fully mine the hidden feature structure containing semantically related information and spatio-temporal characteristics. The experimental results demonstrate the effectiveness of the proposed method.

Introduction

According to the demography, the number of elderly people is increasing worldwide. It is predicted that the life expectancy will increase from 71 years old in 2010–15 to 77 years old in 2045–50 (Nations, 2017). The increasing life expectancy at birth and the declining birth rate result in an aging population structure. Furthermore, elderly people prefer the aging-in-place and independent mode of life on account of the high expense of nursing care and the privacy invasion problem of living with a carer (Simoens et al., 2005). However, many of them are unable to maintain the independent life because of the physical and mental problems.

In case of insufficient medical professionals, it is of great significance and in line with the social needs to implement the intelligent home care systems with medical and assistive functions based on ambient intelligence by means of advanced sensing technologies (Pantelopoulos and Bourbakis, 2009) and potential machine learning algorithms. Maintaining and enhancing the life quality of the aged eases the burden of families and caregivers since the elderly people would be allowed to maintain self-sufficient as long as possible. Ambient Assisted Living (AAL) system that takes advantage of intelligent environment equipped with objects with sensing, actuating, computing and communication features to offer long-term care or establish age-friendly environments, is a promising solution to assist solitary elderly in performing activities of daily living (ADLs) and enjoying a healthy lifestyle (Sodemann et al., 2012).

The capability of monitoring the health situation of solitary elderly in their own homes is a critical service of AAL contexts (Palumbo et al., 2014). This capability minimizes the occurrence of emergencies, which has a significant impact on public and/or private health services. And age-related chronic diseases, such as dementia, depression and arthritis, can be dealt with a proactive and preventive approach (Chernbumroong et al., 2013). Multiple methods have been presented to gather and analyze the information on the ADLs of the elderly, for example, detecting emergencies (Tong et al., 2013) and identifying activity deviations (Dhiman and Vishwakarma, 2019, Sodemann et al., 2012). As thus, these efforts actually help to increase the attention of caregivers to keep essential and timely responses. In this setting, accurate, comprehensive and fault-tolerant activity recognition (AR) is first and foremost.

Generally, AR can be classified into two categories according to the type of sensor which is used to monitor activity (Chen et al., 2012). The first is vision-based AR which is based on the utilizing of visual sensing devices such as cameras (Candamo et al., 2009). Visual monitoring is intuitive and informative, but it is subjected to problems related to privacy invasiveness and ethics. The second is sensor-based AR, which is based on the employing of the technology of wireless sensor network (Dawadi et al., 2013, De Venuto et al., 2016). Different sensors can be divided into two categories: wearable sensors (De Venuto et al., 2016) and dense sensors (Dawadi et al., 2013). Wearable sensors are installed directly or indirectly on human body. Most wearable sensors require continuous running without manual operation, which may be difficult in practical application scenes. Moreover, it is subjected to the receptivity and willingness of using wearable sensors considering the uncomfortableness. It is clearly that wearable sensors are not appropriate for monitoring ADLs consisting of complex actions or many interactions with environment. In the situation of dense sensing monitoring (or device-free non-privacy invasive monitoring), plenty of sensors, generally low-cost low-power and miniaturized, are embedded in a series of objects or locations in the environment to monitor movements and behaviors. Therefore, the most promising method for the real-life long-term monitoring is the device-free non-privacy invasive system.

Extracting advisable features is significant for AR (Cornacchia et al., 2016). Generally, there are two main methods for extracting features from raw sensor data. One is to exploit rich prior knowledge in the domain of interest for hand-crafted features, the other is to use neural networks to automatically learn features (Sekma et al., 2015, Zhu et al., 2014). In general, hand-crafted features can be classified into time-domain and frequency-domain features. Time-domain features are acquired directly from sensor data, which are usually statistical measures, such as mean, median, variance and standard deviation (Mortazavi et al., 2015). Frequency-domain feature is used to describe the periodicity of sensor signal, and the commonly used features are spectral energy, entropy and dominant frequency (Laudanski et al., 2015, Suto et al., 2017). Hand-crafted features have been extensively studied and demonstrated to be of value for AR. How to fully mine the hidden and differentiated information and build effective features is a difficulty for recognizing ADLs.

In general, activities are context-aware and composed of a variety of sub-activities that are considered as atomic-level actions by researchers (Atallah et al., 2011). A person can complete an activity in different sequences of actions or perform different activities in a same sequence. In addition, one does not always accomplish an activity in a fixed sequence of actions. The diversity of action sequence for a particular category of activity leads to the formation of new cluster accidentally during the training process, and the ambiguity between two categories of activities makes the similarity between their clusters high. The large intra-cluster distance or small inter-cluster distance will increase the possibility of activity confusion. How to settle the issue of activity ambiguity is another difficulty of AR.

Easily-confused activities are defined as activities with same event-triggered binary sensors in the same area in this paper. Many previous researches on ADLs based on sensor data exhibit traditional processing approaches which usually utilize statistical measures. These models are effective to identify activities, but rough construction of feature vector results in some identifiable information missed from the activity ontology. We should mine hidden information/feature structures containing behavioral semantics-related information and spatio-temporal properties, and then build explicit feature vectors to best describe the essence of activities.

A new modeling mode is adopted, which can be implemented by taking advantage of a special design method: emergence (Vernon et al., 2007). With the emergent method, the attention is focused on the low level data processing: sensor data are boosted with behavior and aggregated autonomously, which allows an aggregation awareness in the environment (Barsocchi et al., 2015, Tan et al., 2018, Xu et al., 2018). The biologically-inspired emergent method is according to the mechanism of the self-organization of data (Avvenuti et al., 2013). We propose the emergent representation for ADLs with marker-based stigmergy, which does not need sophisticated domain knowledge. The collective characteristics and interactions of sensor data can be well represented with a domain-independent spatio-temporal logic by means of the emergent paradigm.

It is well known for decades that simple individual behaviors can result in a complicated emergent behavior on a system level in different fields. It has been observed that this genre of emergent aggregated behavior is an ideal characteristic in ubiquitous computing (Barron and Cahill, 2004). Through the use of the emergent paradigm with marker-based stigmergy, we concentrate on finding answers to questions in people’s emergent behaviors, rather than in the environment with complicated cognitive strategies.

We propose a novel two-layer framework based on the event-triggered binary sensor data and the emergent paradigm with marker-based stigmergy for AR of solitary elderly. Specifically, the emergent paradigm with marker-based stigmergy is adopted to construct features by aggregating the sensor data and generating the corresponding activity pheromone trail, i.e. activity pheromone matrix. And the two-layer framework is comprised of coarse-grained and fine-grained classification subsystems. The coarse-grained subsystem uses different activity segmentation methods according to whether the activity is easily-confused or easily-identified, which makes the framework competent to deal with the easily-confused activities. And the fine-grained subsystem combines the activity pheromone trail and machine learning algorithms to achieve the goal of AR. The proposed model is data-driven and based on the self-organization of information, it does not need sophisticated domain knowledge, and can fully mine the hidden feature structure containing semantically related information and spatio-temporal characteristics. We utilize the open dataset Aruba provided by Washington State University’s Center for Advanced Studies in Adaptive Systems (CASAS) project (Cook and Schmitter-Edgecombe, 2009, Cook et al., 2012), and the contributions of this study are summarized as follows:

  • Activity pheromone trail. We propose the feature extraction mode based on the emergent paradigm with marker-based stigmergy, which does require sophisticated domain knowledge. The mode builds feature vectors through aggregating the event-triggered binary sensor data at a low level and generating the corresponding activity pheromone trail. The aggregated trail (i.e. stigmergic mark) is a data form that can jointly characterize the temporal and spatial characteristics of ADLs, and it can be regarded as the short-term and small-size activity memory used as the basis of AR. The pheromone trail and classification algorithm together enable the recognition of activities at a fine-grained level.

  • Pheromone evaporation and diffusion mechanism. We define the heterogeneous pheromone evaporation rate and diffusion rate as normal distribution functions of the activity duration and the range of activity, respectively. The timeliness of the activity pheromone trail is maintained by periodically generating, volatilizing and diffusing pheromones. The heterogeneous mechanism ensures the desensitization of old context-aware information and the effective accumulation of the recent and important context-aware information.

  • Generalized activity segmentation method. In order to decrease the intra-cluster distance and increase the inter-cluster distance, and improve the discrimination of different categories of activities, the additional context-aware information including preparatory phase information (pre-activity information) and inertial phase information (post-activity information) shall be added for easily-confused activities on the basis of traditional completed activities to generate extension activities, which can improve the recognition performance.

  • Two-layer framework with multi-granularity. The two-layer framework comprises of a coarse-grained classification subsystem and a fine-grained classification subsystem. Specifically, the coarse-grained subsystem identifies whether the input completed activity segmented via the traditional method is easily-confused by calculating the activity similarity. If the activity is easily-confused, the generalized segmentation method is adopted. The fine-grained subsystem employs machine learning or deep learning classifiers to achieve the goal of accurately recognizing ADLs.

The rest of this paper is organized as follows: The related works are discussed in Section 2, followed by the description of the proposed model including the marking subsystem and the two-layer activity recognition framework. In Section 4, the experimental studies are represented. And Section 5 draws the final conclusions.

Section snippets

Related works

This section presents the related works relevant to AR. With the rapid development of sensing technologies, machine learning, computer vision and pervasive computing, extensive researches have been implemented in employing different sensing technologies and modeling methods for analyzing and understanding human activities. Generally, activity models can be constructed by means of data-driven approach or knowledge-driven approach. The data-driven approach builds the activity model from the

Activity recognition model based on stigmergy and two-layer framework

The proposed activity recognition model based on the emergent paradigm with marker-based stigmergy and two-layer multi-granularity framework is presented in this section. The flow diagram is shown in Fig. 1. Concretely, the AR model consists of the dataset preparation process and the activity recognition process that is equivalent to the fine-grained classification subsystem using machine learning or deep learning classifiers. And the data preprocessing subsystem, coarse-grained classification

Experiments

In this section, to evaluate our proposed model, we employ the 5-fold cross-validation method and the Aruba dataset provided by the CASAS project. We elaborate the data preparation, evaluation measures and experimental results.

Conclusions

We propose a novel two-layer multi-granularity activity recognition model based on the emergent paradigm with marker-based stigmergy for elderly people living alone and independently in their own homes. The emergent paradigm is employed to build features by aggregating the sensor data at the low level and generating the activity pheromone trails by the marking subsystem, which does not need sophisticated domain knowledge. The activity pheromone trail is a two-dimensional representation of the

CRediT authorship contribution statement

Zimin Xu: Methodology, Validation, Formal analysis, Data curation, Writing - original draft, Writing - review & editing. Guoli Wang: Conceptualization, Methodology, Validation, Resources, Writing - review & editing, Supervision. Xuemei Guo: Validation, Resources, Writing - review & editing.

Declaration of Competing Interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Acknowledgments

This work was supported in part by the National Natural Science Foundation of P.R. China under Grant Nos. 61772574, 61375080 and U1811462 and in part by the Key Program of the National Social Science Fund of P.R. China with Grant No. 18ZDA308.

References (64)

  • XinM. et al.

    ARCH: Adaptive recurrent-convolutional hybrid networks for long-term action recognition

    Neurocomputing

    (2016)
  • AgarwalM. et al.

    Activity recognition using conditional random field

  • AkdemirU. et al.

    An ontology based approach for activity recognition from video

  • Al MachotF. et al.

    Human activity recognition based on real life scenarios

  • AtallahL. et al.

    Sensor positioning for activity recognition using wearable accelerometers

    IEEE Trans. Biomed. Circuits Syst.

    (2011)
  • AvvenutiM. et al.

    MARS, a multi-agent system for assessing rowers coordination via motion-based stigmergy

    Sensors

    (2013)
  • BarronP. et al.

    Using stigmergy to co-ordinate pervasive computing environments

  • BouchardB. et al.

    A smart home agent for plan recognition of cognitively-impaired patients

    J. Comput.

    (2006)
  • CandamoJ. et al.

    Understanding transit scenes: A survey on human behavior-recognition algorithms

    IEEE Trans. Intell. Transp. Syst.

    (2009)
  • CarberryS.

    Techniques for plan recognition

    User Model. User-Adapt. Interact.

    (2001)
  • ChenL. et al.

    Sensor-based activity recognition

    IEEE Trans. Syst. Man Cybern. -Syst.

    (2012)
  • ChenD. et al.

    Towards automatic analysis of social interaction patterns in a nursing home environment from video

  • CookD.J. et al.

    CASAS: A smart home in a box

    Computer

    (2012)
  • CookD.J. et al.

    Assessing the quality of activities in a smart environment

    Methods Inf. Med.

    (2009)
  • CornacchiaM. et al.

    A survey on activity detection and classification using wearable sensors

    IEEE Sens. J.

    (2016)
  • DawadiP.N. et al.

    Automated cognitive health assessment using smart home monitoring of complex tasks

    IEEE Trans. Syst. Man Cybern. -Syst.

    (2013)
  • De VenutoD. et al.

    Remote neuro-cognitive impairment sensing based on P300 spatio-temporal monitoring

    IEEE Sens. J.

    (2016)
  • FahadL.G. et al.

    Activity recognition in smart homes using clustering based classification

  • GalataA. et al.

    Learning structured behaviour models using variable length Markov models

  • HammerlaN.Y. et al.

    Deep, convolutional, and recurrent models for human activity recognition using wearables

  • HanninkJ. et al.

    Sensor-based gait parameter extraction with deep convolutional neural networks

    IEEE J. Biomed. Health Inform.

    (2017)
  • KahnP. et al.

    The development of an adaptive upper-limb stroke rehabilitation robotic system

    J. NeuroEng. Rehabil.

    (2011)
  • Cited by (18)

    • Temporal segment graph convolutional networks for skeleton-based action recognition

      2022, Engineering Applications of Artificial Intelligence
      Citation Excerpt :

      Human action recognition, extensively applied in video understanding (Simonyan and Zisserman, 2014; Wang et al., 2016; Tran et al., 2015), intelligent video surveillance (Ming et al., 2021; Dhiman and Vishwakarma, 2019; Tan et al., 2018) and human–machine interaction (Xu et al., 2020; He et al., 2018), has become an active and significant research area in computer vision.

    View all citing articles on Scopus
    View full text