Elsevier

Information Systems

Volume 80, February 2019, Pages 148-158
Information Systems

Searching for variable-speed motions in long sequences of motion capture data

https://doi.org/10.1016/j.is.2018.04.002Get rights and content

Highlights

  • Subsequence matching algorithm for motion capture data that searches for variable-speed motions.

  • A unique combination of elastic similarity measure and multi-level segmentation structure.

  • Real-time searching in months of motion capture recordings by using a bit-vector representation or disk-based index.

  • Effective and indexable 4096-dimensional motion features extracted using a neural network.

Abstract

Motion capture data digitally represent human movements by sequences of body configurations in time. Subsequence searching in long sequences of such spatio-temporal data is difficult as query-relevant motions can vary in execution speeds and styles and can occur anywhere in a very long data sequence. To deal with these problems, we employ a fast and effective similarity measure that is elastic. The property of elasticity enables matching of two overlapping but slightly misaligned subsequences with a high confidence. Based on the elasticity, the long data sequence is partitioned into overlapping segments that are organized in multiple levels. The number of levels and sizes of overlaps are optimized to generate a modest number of segments while being able to trace an arbitrary query. In a retrieval phase, a query is always represented as a single segment and fast matched against segments within a relevant level without any costly post-processing. Moreover, visiting adjacent levels makes possible subsequence searching of time-warped (i.e., faster or slower executed) queries. To efficiently search on a large scale, segment features can be binarized and segmentation levels independently indexed. We experimentally demonstrate effectiveness and efficiency of the proposed approach for subsequence searching on a real-life dataset.

Introduction

Current motion capturing technologies can accurately record a human motion at high spatial and temporal resolutions. The recorded motion is represented as an ordered sequence of poses that describe skeleton configurations in corresponding video frames. The skeleton configuration is represented by a set of 3D coordinates determining positions of the captured body joints in space. The recorded motion sequences are used in a variety of applications, e.g., in healthcare to recognize movement disorders, in sports to analyze performances of top athletes, or in computer animation to browse large databases of human motions for production of realistically looking games or movies. These applications require effective and efficient search operations to increase reusability and findability of expensively recorded data in the past.

A search operation is primarily specified by a query object that can be either selected as an existing example [1], or modeled by special interfaces such as hand-drawn sketches [2] or puppet models [3]. We especially focus on the query-by-example subsequence matching operation: Given a short query sequence and a very long data sequence, search the data sequence and locate its subsequences that are the most similar to the query sequence. For example, find occurrences of perfect backflip landings within hundreds of hours of exercise recordings. Locating query-relevant subsequences constitutes a hard task since their lengths and positions (i.e., beginnings and endings) are unknown. Moreover, the query cannot be anticipated in advance and need not correspond to any semantic or known action class, so textual-annotation-based retrieval cannot be applied. To deal with these problems, a fine segmentation technique along with an effective similarity measure are needed.

The contribution of this paper is an efficient subsequence matching approach that is schematically illustrated in Fig. 1. The retrieval process is based on a multi-level segmentation structure that produces a minimum number of segments with respect to an elasticity property of a similarity measure. The elasticity allows segments to be shifted much more than of a single frame, which increases overall search performance. We further binarize segment features and employ a disk-based index structure to index individual segmentation levels independently and thus access them in parallel. This enables real-time searching in long motion sequences, taking even dozens of days. A proposed speed-invariant retrieval algorithm additionally supports searching for query-relevant motions that are executed faster or slower. The whole multi-level segmentation structure is customizable to compromise between search performance and accuracy. It is also dynamic to support the addition of a new content.

Section snippets

Related work

Subsequence matching methods for motion capture data generally require a (1) segmentation technique to partition a data sequence into meaningfully-long data segments, (2) similarity measure to compare query and data segments, and (3) retrieval algorithm to efficiently localize query-similar subsequences by grouping the most relevant data segments.

Similarity of motion data

To compare any pair of rather short motion sequences, we adopt a similarity measure proposed in [29]. It uses very effective and fixed-size feature vectors extracted from motions of variable lengths and compares them by the efficient Euclidean distance. The feature extraction is based on visualizing the normalized joint trajectories into a 2D image, fine-tuning a deep convolutional neural network by the generated images, and extracting the 4096D feature vector of a high descriptive power from

Multi-level segmentation structure of data sequences

To search for query-relevant subsequences, the data sequence has to be partitioned into segments. Traditional methods [8] suggest partitioning the data sequence into disjoint (non-overlapping) segments, while the query sequence into overlapping segments using the sliding window principle (or vise-versa). Such partitioning facilitates locating relevant data segments that are similar to some query segments. Although the data segments can be indexed and efficiently retrieved, this concept has the

Subsequence retrieval in multi-level segmentation structure

Within the preprocessing phase a user specifies three compulsory parameters – covering factor cf, minimum lmin and maximum lmax length of query – so that the multi-level segmentation structure could be constructed for a long data sequence. The objective of the retrieval phase is to search the long data sequence and locate its subsequences that are the most similar to a short query sequence, which is bounded in length [lmin,lmax]. We approximate localization of such subsequences by traversing

Experimental evaluation

We experimentally evaluate both effectiveness and efficiency of proposed subsequence retrieval algorithms that combine the multi-level segmentation structure with the elastic similarity measure to evaluate query-to-segment similarity. We also compare the results against existing subsequence matching approaches.

Conclusions

We propose a new speed-invariant subsequence matching algorithm that uses a synergy of elastic similarity measure and multi-level segmentation. The search space comprises overlapping segments of various sizes that ensure the bounded coverage of arbitrary parts within very long motion sequences. The size of overlaps and the total number of generated segments are bounded to be formally minimum with respect to the covering factor parameter, which reflects the versatility and effectiveness of the

Acknowledgment

This research was supported by GBP103/12/G084.

References (43)

  • FaloutsosC. et al.

    Fast subsequence matching in time-series databases

    SIGMOD Rec.

    (1994)
  • LinJ.F.S. et al.

    Movement primitive segmentation for human motion modeling: A framework for analysis

    IEEE Trans. Hum. Mach. Syst.

    (2016)
  • BeecksC. et al.

    Spatiotemporal similarity search in 3d motion capture gesture streams

  • KrügerB. et al.

    Fast local and global similarity searches in large motion capture databases

  • ValcikJ. et al.

    Assessing similarity models for human-motion retrieval applications

    Comput. Animat. Virtual Worlds

    (2016)
  • C. Beecks, M. Hassani, F. Obeloer, T. Seidl, Efficient query processing in 3D motion capture databases via lower bound...
  • C. Ren, X. Lei, G. Zhang, Motion data retrieval from very large motion databases, in: International Conference on...
  • YeL. et al.

    Time series shapelets: A novel technique that allows accurate, interpretable and fast classification

    Data Min. Knowl. Discov.

    (2011)
  • ZhouF. et al.

    Hierarchical aligned cluster analysis for temporal clustering of human motion

    IEEE Trans. Pattern Anal. Mach. Intell.

    (2013)
  • chung FuT.

    A review on time series data mining

    Eng. Appl. Artif. Intell.

    (2011)
  • WuS. et al.

    Indexing and retrieval of human motion data by a hierarchical tree

  • Cited by (23)

    View all citing articles on Scopus
    View full text