Searching for variable-speed motions in long sequences of motion capture data
Introduction
Current motion capturing technologies can accurately record a human motion at high spatial and temporal resolutions. The recorded motion is represented as an ordered sequence of poses that describe skeleton configurations in corresponding video frames. The skeleton configuration is represented by a set of D coordinates determining positions of the captured body joints in space. The recorded motion sequences are used in a variety of applications, e.g., in healthcare to recognize movement disorders, in sports to analyze performances of top athletes, or in computer animation to browse large databases of human motions for production of realistically looking games or movies. These applications require effective and efficient search operations to increase reusability and findability of expensively recorded data in the past.
A search operation is primarily specified by a query object that can be either selected as an existing example [1], or modeled by special interfaces such as hand-drawn sketches [2] or puppet models [3]. We especially focus on the query-by-example subsequence matching operation: Given a short query sequence and a very long data sequence, search the data sequence and locate its subsequences that are the most similar to the query sequence. For example, find occurrences of perfect backflip landings within hundreds of hours of exercise recordings. Locating query-relevant subsequences constitutes a hard task since their lengths and positions (i.e., beginnings and endings) are unknown. Moreover, the query cannot be anticipated in advance and need not correspond to any semantic or known action class, so textual-annotation-based retrieval cannot be applied. To deal with these problems, a fine segmentation technique along with an effective similarity measure are needed.
The contribution of this paper is an efficient subsequence matching approach that is schematically illustrated in Fig. 1. The retrieval process is based on a multi-level segmentation structure that produces a minimum number of segments with respect to an elasticity property of a similarity measure. The elasticity allows segments to be shifted much more than of a single frame, which increases overall search performance. We further binarize segment features and employ a disk-based index structure to index individual segmentation levels independently and thus access them in parallel. This enables real-time searching in long motion sequences, taking even dozens of days. A proposed speed-invariant retrieval algorithm additionally supports searching for query-relevant motions that are executed faster or slower. The whole multi-level segmentation structure is customizable to compromise between search performance and accuracy. It is also dynamic to support the addition of a new content.
Section snippets
Related work
Subsequence matching methods for motion capture data generally require a (1) segmentation technique to partition a data sequence into meaningfully-long data segments, (2) similarity measure to compare query and data segments, and (3) retrieval algorithm to efficiently localize query-similar subsequences by grouping the most relevant data segments.
Similarity of motion data
To compare any pair of rather short motion sequences, we adopt a similarity measure proposed in [29]. It uses very effective and fixed-size feature vectors extracted from motions of variable lengths and compares them by the efficient Euclidean distance. The feature extraction is based on visualizing the normalized joint trajectories into a 2D image, fine-tuning a deep convolutional neural network by the generated images, and extracting the 4096D feature vector of a high descriptive power from
Multi-level segmentation structure of data sequences
To search for query-relevant subsequences, the data sequence has to be partitioned into segments. Traditional methods [8] suggest partitioning the data sequence into disjoint (non-overlapping) segments, while the query sequence into overlapping segments using the sliding window principle (or vise-versa). Such partitioning facilitates locating relevant data segments that are similar to some query segments. Although the data segments can be indexed and efficiently retrieved, this concept has the
Subsequence retrieval in multi-level segmentation structure
Within the preprocessing phase a user specifies three compulsory parameters – covering factor , minimum and maximum length of query – so that the multi-level segmentation structure could be constructed for a long data sequence. The objective of the retrieval phase is to search the long data sequence and locate its subsequences that are the most similar to a short query sequence, which is bounded in length . We approximate localization of such subsequences by traversing
Experimental evaluation
We experimentally evaluate both effectiveness and efficiency of proposed subsequence retrieval algorithms that combine the multi-level segmentation structure with the elastic similarity measure to evaluate query-to-segment similarity. We also compare the results against existing subsequence matching approaches.
Conclusions
We propose a new speed-invariant subsequence matching algorithm that uses a synergy of elastic similarity measure and multi-level segmentation. The search space comprises overlapping segments of various sizes that ensure the bounded coverage of arbitrary parts within very long motion sequences. The size of overlaps and the total number of generated segments are bounded to be formally minimum with respect to the covering factor parameter, which reflects the versatility and effectiveness of the
Acknowledgment
This research was supported by GBP103/12/G084.
References (43)
- et al.
Efficient time-series subsequence matching using duality in constructing windows
Inf. Syst.
(2001) - et al.
A filtering method for searching similar multidimensional sequences under the time-warping distance
Inf. Syst.
(2003) - et al.
Retrieval of logically relevant 3d human motions by adaptive feature selection with graded relevance feedback
Pattern Recognit. Lett.
(2012) - et al.
Efficient and robust annotation of motion capture data
- et al.
Human motion retrieval from hand-drawn sketch
IEEE Trans. Vis. Comput. Graphics
(2012) - et al.
A puppet interface for retrieval of motion capture data
- A. Vögele, B. Krüger, R. Klein, Efficient unsupervised temporal segmentation of human motion, in: ACM Symposium on...
- et al.
Segmenting motion capture data into distinct behaviors
- et al.
Semantic segmentation of motion capture using laban movement analysis
- et al.
Automated human motion segmentation via motion regularities
Vis. Comput.
(2015)
Fast subsequence matching in time-series databases
SIGMOD Rec.
Movement primitive segmentation for human motion modeling: A framework for analysis
IEEE Trans. Hum. Mach. Syst.
Spatiotemporal similarity search in 3d motion capture gesture streams
Fast local and global similarity searches in large motion capture databases
Assessing similarity models for human-motion retrieval applications
Comput. Animat. Virtual Worlds
Time series shapelets: A novel technique that allows accurate, interpretable and fast classification
Data Min. Knowl. Discov.
Hierarchical aligned cluster analysis for temporal clustering of human motion
IEEE Trans. Pattern Anal. Mach. Intell.
A review on time series data mining
Eng. Appl. Artif. Intell.
Indexing and retrieval of human motion data by a hierarchical tree
Cited by (23)
Tracking subjects and detecting relationships in crowded city videos
2024, Multimedia Tools and ApplicationsPESTA: An Elastic Motion Capture Data Retrieval Method
2023, Journal of Computer Science and TechnologySegmentCodeList: Unsupervised Representation Learning for Human Skeleton Data Retrieval
2023, Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)Towards Efficient Human Action Retrieval Based on Triplet-Loss Metric Learning
2022, Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)SPEED21: Speed climbing motion dataset
2021, MMSports 2021 - Proceedings of the 4th International Workshop on Multimedia Content Analysis in Sports, co-located with ACM MM 2021Efficient indexing of 3D human motions
2021, ICMR 2021 - Proceedings of the 2021 International Conference on Multimedia Retrieval