Keywords

1 Introduction and Background

1.1 Introduction

Designing and conducting user studies and pilot trials in Active and Healthy Ageing (AHA) remains a major challenge for the researchers [1, 2]. Recognizing the challenging character of this domain and taking into account the socio-economic impact of such studies, Europe shifted a noteworthy portion of funding to large scale pilots in ecologically valid environments or even better in seniors’ homes. It is luckily that this domain, consisting a new distinct research area, will be in the foreground for the next decades, demanding technological researchers with expertise on the AHA domain.

Over the last few years, a large number of projects have been working on technological innovations for seniors. Algorithms detecting behavioral changes [3, 4], monitoring and classifying daily indoor/outdoor activities, serious games through contemporary controllers [5] as well as tools assessing seniors’ physical and cognitive status [6, 7] have been developed. Small scale trials and case studies [2] were the common methodologies for evaluating these technologies. However, only recently researches started employing large scale trials for producing high accuracy and robust technologies that could be applied in real life settings. In that context, Big Data datasets have been emerged.

However, these datasets usually belong to the project partners and it is only recently that funding bodies have asked for data openness provisions (e.g. H2020 Data Management Plans). Such initiatives target to accelerate design and evaluation of technologies even from SMEs or individual researchers towards market ready innovation. In that context, the cost, the infrastructure and complexity of conducting pilot trials with real users are deterrent factors for designing, testing and evaluating new ideas and tools. On top of this, inexperience or limited knowledge about different stages of piloting with real users (e.g. recruitment, protocol design, ethical issues, technical details, etc.), as well as the need for trust development when seniors are involved, are time and cost consuming barriers.

Consequently, specially designed tools and infrastructures supporting Big Data collection and sharing, are more than essential now than ever before. This work presents the architecture and implementation of an infrastructure for streaming AHA datasets on demand. A large number of datasets (~1.5 TB) have been collected within the pilot trials of the USEFIL project carried in the Active and Healthy Ageing Living Lab, an ecologically valid environment resembling a real home environment [8]. A number of elderly volunteers were visiting this environment daily and performing a set of daily activities. The sources of the collected datasets are the Kinect (RGB video and skeleton), as well as, the Wii Balanceboard device (center of balance) and an IP video camera.

1.2 Background

To date, several annotated open source datasets have been produced and shared with the research community. Some of them were captured in the patient’s homesFootnote 1 [9]. The common ways of getting access to such datasets is to download them from a websiteFootnote 2 [10] or even worse to get them stored in physical means (external hard-disk) due to the very large size (>1 TB) or the institution’s policy not to provide such datasets through public accessible means. In the same context, although the emergence of tools to explore, publish and share public datasets led to online collections of such tools [11], there is no tool that reproduces a pilot experiment by synchronously playing back a number of different datasets. Nebeling et al. [12] presented Kinect Analysis, a system designed for interaction elicitation studies with support for record-and-replay, visualisation and analysis based on Kinect’s depth, audio and video streams. Kinect Analysis enables post-hoc analysis during playback and live analysis with real-time feedback while recording. Nebeling et al. presented that Kinect Analysis in combination with Kinect-Script is useful and effective for a range of analysis tasks, based on a guessability study with 25 users. Finally, they concluded that their implementation could enable researchers to more easily collect, study and share interaction proposals.

In the same context, a number of applications have already been designed for recording and playing back the Kinect channels (RGB, depth and skeleton). The Kinect Stream Saver Application [13], based on the KinectExplorer-D2D, allows users to display and store the Kinect streams. The recording class stores stream data (color, depth, skeleton) to output files in the disk. The Kinect ToolboxFootnote 3 includes a set of replay classes that allows recording and replaying of Kinect sessions. As the authors declare: “It is much easier to debug and develop when you don’t always have to stand up”. Lars Ivar HatledalFootnote 4 developed a system which records a 3D video of a person using the Kinect v2 and plays it back in a 3D environment running in the browser (WebGL and three.js). The recorded file are compressed and uploaded to the browser for playing it back. However, none of the latter implementations stream the recorded datasets through a publish/subscribe architecture where more than one clients could concurrently get access to.

2 Materials and Methods

2.1 CAC Playback Manager

The CAC Playback Manager presented in this paper, is a system composed of a number of streaming players (start, stop, pause, move time, etc.), delivering data streams to remote clients (developers’ app) on demand (Fig. 1). This manager simulates the output (events, video, skeleton streams, etc.) of devices that have been recorded during a pilot trial or experiment. The recorded outputs are reproduced (playback) synchronously through the CAC-framework communication channel (web socket, publish/subscribe based) as if the pilot/experiment was conducted at the moment of the playback session. The CAC-framework ensures that any client (device or application) connected to it gets the streamed information as if it was used during the pilot/experiment even if this client was not there initially. The CAC Playback Manager exposes its functionality through an API to facilitate researchers in utilizing it.

Fig. 1.
figure 1

Conceptual architecture representing the flow from the actual pilot trials on the left side to the subsequent playback of the collected datasets on the right.

Full size image

2.2 The CAC Framework API and Client Libraries

The Controller Application Communication (CAC) framework [8] (www.cac-framework.com) is a cross device, application independent service providing a number of compatible client libraries [14]. Based on the contemporary websockets communication protocol, the CAC- framework functions as an intermediary, facilitating the connection of a series of controllers and devices (Kinect, Wii Balanceboard, Emotiv, etc.) to any applications requesting their input (cf. Fig. 2). Given the number and type of these devices’ information (skeleton, video, etc.), the framework is already adapted to high throughput. The framework’s design principles align with the Internet of Things (IoT) paradigm following the publish/subscribe architecture witha JSON messages.

Fig. 2.
figure 2

Controller Application Communication (CAC) framework concept

Full size image

Despite the simple and well documentedFootnote 5 communication protocol of the API, a number of available client libraries can facilitate developers to incorporate the framework’s functionality to web applications exploiting the usage of contemporary controllers. Javascript, nodeJS, C# and Python are some of the supported development languages. Taking advantage of these libraries, any application can be easily configured to get access to the streamed information without significant effort or configuration on the device’s side.

2.3 CAC Playback Player

A CAC Playback Player is a module that communicates with a specific type of dataset (MongoDB, local file, triplestore, etc.) and streams/playbacks the dataset with respect to the acquisition frame rate. Each player utilizes the corresponding CAC client library to stream the recorded dataset through the CAC framework, simulating the source device’s functionality. For instance, if a pilot trial was captured by Kinect, the CAC playback player would reproduce the datasets as if the pilot was conducted now. The player’s synchronization (frames per second) relies on the datasets’ timestamp as well as the requested playback speed.

Each CAC Playback Player exposes a set of commands and properties in order to be managed by the CAC Playback Manager. The properties refer to the CAC server endpoint, streaming session, player status (playing, stopped, paused, cursor position, etc.), dataset name and type, image and video size. Apart from the properties, a CAC playback player exposes a number of methods pertaining to initializing the player, start/stop/pause streaming, set/move playback cursor position as well as some callback’s on streaming finished, connection lost events, etc.

2.4 CAC Playback Manager API

The CAC Playback Manager orchestrates and synchronizes a number of streaming players by exposing a number of functionalities in the form of an API, delivering data streams to remote client (developers’ app) on demand. It keeps a record of all the alive sessions alongside the corresponding playback players and properties. The manager is responsible for releasing any session resources in case of inactivity (keep-alive message has not been received for more than 15 min). The CAC Playback manager exposes its functionality through a restful API (Table 1). The playback session id, shared and defining a common session among all clients and playback players, is a common parameter for each method of the API.

Table 1. CAC Playback Manager API methods. Playback session (psid) parameter is shared among all the methods.
Full size table

2.5 CAC Playback Manager Web App

The CAC Playback Manager Web AppFootnote 6 is a web application providing access to the CAC Playback Manager API through an intuitive user interface for non-programmers. The user configures the session id and initiates a playback session. A listbox allows the user to select the datasets while the start, pause and stop buttons correspond to the playback functionalities of the dataset. The user can move the cursor forward/backward or even to a specific date and time. The web app incorporates also a client to illustrate the streamed dataset (Fig. 3: the streamed skeletons on the right and the RGB images on the left are depicted).

Fig. 3.
figure 3

The CAC Playback Manager Web App (http://www.cac-framework.com/app). Skeletons streaming (Right) and RGB Video streaming (Left) are handled by the integrated CAC client.

Full size image

3 Use Cases

The system has already been utilized to support researchers. More specifically it facilitated the development of an algorithm for calculating High Density Regions (HDR) of indoor localization [15]. It was also employed to support the implementation of an algorithm about seniors’ skeleton rotation and identification. The exergaming recordings (part of the datasets collection protocol) facilitated the iterative development of the webFitForAll exergaming algorithms [16], improving seniors’ experience. Finally, the CAC Playback Manager facilitated the datasets’ annotations.

3.1 High Density Regions of Indoor Localization

The indoor analytics client [15] is implemented in NodeJS and is based partially on the Node.js CAC-framework client described in [15]. By subscribing to the skeleton streaming channel, the indoor analytics client gets access to the skeletons’ center of mass position along with the timestamp, the skeleton id and the corresponding position of the joints, clustering the regions in a room that are more frequently visited by seniors in their home. These High Density Regions are then used for context aware gait analysis.

The CAC Playback Manager was utilized to fine tune the algorithm by feeding the indoor analytics client with datasets collected during pilots conducted in the ecologically valid Active and Healthy Ageing Living Lab in the Lab of Medical Physics in Thessaloniki. Moreover, the developers utilized the datasets to find workarounds on overcoming the computing power constraints (RAM and CPU usage) towards deploying the indoor analytics client in a Raspberry PI 2.

3.2 Skeleton Rotation and Identification

A dataset representing a person revolving around his body’s axis was captured by a Kinect. This dataset was then used to facilitate the development of an algorithm for coordinating and transforming the skeleton to be vertical to the Kinect’s axis. This relies on identifying the skeleton’s angle and position offset (Fig. 4).

Fig. 4.
figure 4

Top: Screenshot of a participant conducting an exergame activity (revolving around his body’s axis). Bottom: the skeleton dataset representing the same person.

Full size image

3.3 Exergaming Design and Development

The FitForAll exergaming platform, tailored to seniors, was developed years before, when the contemporary gaming controllers where restricted to the Wii devices [5]. The advent of Kinect made the FitForAll team consider it as a one way, incorporating it in the second version, webFitForAll [16]. The adoption of Kinect opened new ways of navigating throughout the exercises as well as the way these exercises are performed. The recordings of some exergaming sessions performed by 14 people within USEFIL’s [17] trials constitute the basis datasets for fine tuning the exercises experience as well as the monitoring assessment algorithms behind the exercises [7].

4 Discussion and Conclusions

Our work was motivated from challenges that emerged from pilot trials with elderly people and testing of technologies in the Active and Healthy Ageing domain. Developing and evaluating algorithms and tools by reproducing pilot trials on demand with a time and cost effective way remains a big challenge in the Big Data area.

The work presented in this paper goes beyond the existing implementations that simply record Kinect datasets locally in files replacing the Kinect by playing back these files. Stream Saver Application, Kinect Toolbox and Lars Ivar Hatledal’s work have already implemented different recorders for the Kinect’s channel. None of these works however uses websockets or other web communication channels for streaming the recorder datasets to more than one clients at the same time. On the other hand, Nebeling et al. [12] did something very similar to our work. However, their work was restricted to Kinect compatibility with no provision for other type of datasets. One step further, the CAC Playback Manager was built with the intention to reproduce ecologically valid pilot trials and experiments on demand by playing back all the available recorded datasets through a unified solution based on web sockets (CAC-framework [8]). For this purpose, a CAC Playback Player is designed for each dataset type, being synchronized by the CAC Playback Manager.

The CAC Playback Manager delivers an API for playing back collected datasets reproducing a pilot trial or experiment conditions. The value of replaying datasets, in a way of reproducing pilot trials as if they were conducted now without requiring seniors to go through the same activities in the same way which is not ecologically valid, is quite large. Needless to say that such a tool allows developers and companies to develop and tune their algorithms and tools 24 h per day, 7 days per week. Although contemporary technology allows such tools and systems to be built without outstanding effort in conjunction with the fact that sharing of recordings and reproduction of analyses is more than appealing in the days of Big Data, such systems have not emerged yet.

Consequently, the work presented in this paper could be considered as a testbed for unexploited ideas and research results (especially for SMEs and small research groups for which the cost for pilot trials is most of the times deterrent) with considerable innovation potential, against market entrance failure. From a different perspective, such a system could be part of the curriculum in engineering educational institutes creating experienced early engineering researchers in the Active and Healthy Ageing domain. Needless to say that the socio-economic benefits of this in conjunction with the acceleration of new ideas evaluation in real settings are more than significant. Finally, the reproduction of pilot trials, could contribute to the creation of an innovation ecosystem able to transform ideas and knowledge into socio-economic value, accelerating the market entrance.