Progressive semantic learning for unsupervised skeleton-based action recognition

Abstract

Traditional contrastive learning frameworks for skeleton-based action recognition use data augmentation and memory bank techniques to obtain positive/negative samples required for training, but this instance-level pseudo-label generation mechanism does not take full advantage of the rich cluster-level semantic information contained in human skeleton sequences. In this paper, we propose a Progressive Semantic Learning method (ProSL), which gradually optimizes the pseudo-label generation mechanism in self-supervised contrastive learning through an iterative framework, so that representation learning can effectively capture action semantic information. Specifically, the existing contrastive learning methods can output an initial skeleton encoder. Then, on the basis of this encoder, clustering methods can be applied to generate a Codebook containing the semantic information of human actions, which is further used to improve the pseudo-label generation mechanism. Finally, based on the above two-step iterations, we achieve progressive semantic learning and obtain a more reasonable skeleton encoder. Extensive experiments on four datasets demonstrate that our proposed method achieves SOTA on multiple downstream tasks.

Appendix A Rationality of the training process

Iterative algorithms have proven their effectiveness in many fields (Dempster et al., 1977; MacQueen et al., 1967; Wu et al., 2008). Here, we analyze ProSL based on the Expectation-Maximization algorithm. Suppose in the t-th iteration, the parameters of the network are denoted as $\theta ^t$, and the parameters of SCoBo are denoted as $\varphi ^t$. For n samples, the objective of self-supervised skeleton sequence representation learning is to maximize the log-likelihood function:

$$\begin{aligned} \theta ^{t+1}=argmax\sum \limits _{i=1}^{n}log p(x_i; \theta ^t) \end{aligned}$$

(13)

After introducing ScoBo, we maximize the log-likelihood function of the model distribution as follows:

$$\begin{aligned} \theta ^{t+1}=argmax\sum \limits _{i=1}^{n}log\sum _{\varphi _i^t}p(x_i, \varphi _i^t; \theta ^t) \end{aligned}$$

(14)

Equation 14 is derived based on the marginal probability of $x_i$, and we cannot directly calculate $\theta ^{t+1}$. To address this, introduce an unknown new distribution $Q_i(\varphi _i^t)$ and scale Eq. 14 using Jensen’s inequality as follows:

$$\begin{aligned} \begin{aligned} \sum _{i=1}^{n} \log \sum _{\varphi _{i}^t} p\left( x_{i}, \varphi _{i}^t ; \theta ^t\right) \ge \sum _{i=1}^{n} \sum _{\varphi _{i}^t} Q_{i}\left( \varphi _{i}^t\right) \log \frac{p\left( x_{i}, \varphi _{i}^t ; \theta ^t\right) }{Q_{i}\left( \varphi _{i}^t\right) } \end{aligned} \vspace{-6pt} \end{aligned}$$

(15)

If we want to satisfy the equality of Jensen’s inequality, then:

$$\begin{aligned} & \frac{p(x_i, \varphi _i^t; \theta ^t)}{Q(\varphi _i^t)} = c = \sum _{\varphi _i^t}p(x_i, \varphi _i^t; \theta ^t) \end{aligned}$$

(16)

$$\begin{aligned} & Q_i(\varphi _i^t) = \frac{p(x_i, \varphi _i^t; \theta ^t)}{\sum _{\varphi _i^t}p(x_i, \varphi _i^t; \theta ^t)} =\frac{p(x_i, \varphi _i^t; \theta ^t)}{\sum _{\varphi _i^t}p(x_i; \theta ^t)} =p(\varphi _i^t|x_i;\theta ^t) \end{aligned}$$

(17)

Equation 15 provides a lower bound for the log-likelihood function. Therefore, in the Maximization step, i.e., the representation learning process, our objective is:

$$\begin{aligned} \theta ^{t+1} = argmax \sum _{i=1}^{n} \sum _{\varphi _{i}} Q_{i}\left( \varphi _{i}^t\right) \log \frac{p\left( x_{i}, \varphi _{i}^t ; \theta ^t\right) }{Q_{i}\left( \varphi _{i}^t\right) } \end{aligned}$$

(18)

For the Expectation step, which is the action semantic learning process in ProSL, we need to estimate $p(\varphi _i^{t+1};x_i, \theta ^{t+1})$ to obtain $SCoBo^{t+1}$.

The iterative process has been proven to converge (Dempster et al., 1977), however, the final iteration results are highly influenced by the initialization, and model parameters may get stuck in local optimal points. In ProSL, to ensure the quality of $\varphi _{i}^0$, we choose to first perform several rounds of representation learning before starting the iterative training. Figure 4b in the main text illustrates the necessity of this strategy.