Recognizing Interactive Group Activities Using Temporal Interaction Matrices and Their Riemannian Statistics

Abstract

While video-based activity analysis and recognition has received much attention, a large body of existing work deals with activities of a single subject. Modeling and recognition of coordinated multi-subject activities, or group activities, present in a variety of applications such as surveillance, sports, and biological monitoring records, etc., is the main objective of this paper. Unlike earlier attempts which model the complex spatial temporal constraints among multiple subjects with a parametric Bayesian network, we propose a compact and discriminative descriptor referred to as the Temporal Interaction Matrix for representing a coordinated group motion pattern. Moreover, we characterize the space of the Temporal Interaction Matrices using the Discriminative Temporal Interaction Manifold (DTIM), and use it as a framework within which we develop a data-driven strategy to characterize the group motion pattern without employing specific domain knowledge. In particular, we establish probability densities on the DTIM for compactly describing the statistical properties of the coordinations and interactions among multiple subjects in a group activity. For each class of group activity, we learn a multi-modal density function on the DTIM. A Maximum a Posteriori (MAP) classifier on the manifold is then designed for recognizing new activities. In addition, we have extended this model to one with which we can explicitly distinguish the participants from non-participants. We demonstrate how the framework can be applied to motions represented by point trajectories as well as articulated human actions represented by images. Experiments on both cases show the effectiveness of the proposed approach.

Appendix

1.1 7.1 Derivation of (37)

By elementary calculus it is obvious that

(44)

Since ϵf is a local deviation from P ^J in Taylor’s expansion, here we may assume ϵf≪P ^J. Hence we ignore the terms of ϵf and have the approximation

(45)

Note that the best ϵ is determined after f is learned.

1.2 7.2 Derivation of the Expansion of (40) in Sect. 4

Consider exemplar set {z ₁,z ₂,…,z _M }, where the tuple $z_{i}\triangleq(X_{i}, Y_{i}, \bar{W}_{i}, \bar{X}_{i}, c_{i})$. With these exemplars and assuming uniform prior probabilities for the classes, we have

(46)

where the last equality comes from the fact that given an exemplar z, i.e., Y, the matching W between it and the testing interaction $\bar{Y}$ does not depend on the class label c anymore. Further more, note that according to the definition of z the component $\bar{W}$ in z has already encoded the matching between $\bar{Y}$ and Y. Therefore, we assume that the optimal matching W only depends on $\bar{W}$ given z, and consequently W and $\bar{Y}$ are conditionally independent given z. As a result, we have

$$ P(\bar{Y},W|z)=P(W|z)P(\bar{Y}|z)=P(W|\bar{W})P(\bar{Y}|z). $$

(47)

At the same time, note that given matching $\bar{W}$ in z, P subjects can be selected from $\bar{Y}$ which are then reduced to a P×P Temporal Interaction Matrix $\bar{X}$. As a result, we have

$$ P(\bar{Y}|z)=P(\bar{X}|z)=P(\bar{X}|X). $$

(48)

Now, the first integrand in the rightmost hand side of (46) can be written as

$$ P(\bar{Y},W|z)=P(W|\bar{W})P(\bar{X}|X), $$

(49)

and the second integrand can also be straightforwardly simplified as

$$ P(z|c)=P(X|c). $$

(50)

Eventually, (46) can be expressed as

(51)

and the objective (40) simply becomes

(52)

1.3 7.3 Solution to (42) or (43)

A unified representation for the optimization problems (42) and (43) can be written as

(53)

where w is the PQ×1 vectorization of the matrix W by stacking the columns of W. The constant vectors or matrices c,H,E,e,F,f encode the other numbers in the objective or constraints in the original optimization problem (42) or (43).

Note that though the objective is quadratic in w, it is not necessarily convex or concave, and the elements of w only allow 0 or 1. Instead of directly tackling it, we solve the following optimization.

(54)

where $\hat{\mathbf{c}}=[\sigma_{1},\sigma_{2},\ldots,\sigma_{PQ}]^{T}$, $\hat{H}=\operatorname{diag}\{-\sigma_{1},-\sigma_{2},\ldots,\allowbreak-\sigma_{PQ}\}$, and σ _i is a sufficiently large number satisfying $\sigma_{i}>\sum^{PQ}_{j=1,j\neq i}|H_{ij}|+H_{ii}$.

Note that $\hat{H}$ defined in this way imposes a negative strictly dominant diagonal to H and the quadratic term $\hat{H}+H$ is strictly negative definite. Therefore, (54) is a concave programming problem in the convex unit hypercube [0,1]^P×Q and will achieve its minimum at one of the feasible vertices satisfying the linear equality and inequality constraints. The feasible vertices, meanwhile, are exactly the feasible solutions of (53), and at these vertices, the values of the objective of (54) are equal to those of (53) due to the cancellation brought by $\hat{\mathbf{c}}$. It is therefore implied that by solving the much more efficient problem in (54) we obtain the exact solution for the original problem in (53). To solve (54), we simply employ the optimization software CVX (Grant and Boyd 2011).