Spatio-temporal compression for semi-supervised video object segmentation

Abstract

In this paper, we explore the spatial–temporal redundancy in video object segmentation (VOS) under semi-supervised context with the purpose to improve the computational efficiency. Recently, memory-based methods have attracted great attention for their excellent performance. These methods involve first constructing an external memory to store the target object information in the history frames and then selecting the information that is beneficial for modeling the target object by memory reading. However, such methods are inefficient and unable to achieve both high accuracy and high efficiency, due to the large amount of redundant information in memory. Moreover, they periodically sample historical frames and add them to memory; this operation may lose important information from dynamic frames with incremental object changing or aggravate temporal redundancy from static frames with no object changing. To address these problems, we propose an efficient semi-supervised VOS approach via spatio-temporal compression (termed as STCVOS). Specifically, we first adopt a temporally varying sensor to adaptively filter static frames with no target objects evolutions and trigger memory to update with frames containing noticeable variations. Furthermore, we propose a spatially compressed memory to absorb features with varied pixels and remove outdated features, which considerably reduces information redundancy. More importantly, we introduce an efficient memory reader to perform memory reading with less footprint and computational overhead. Experimental results indicate that STCVOS performs well against state-of-the-art methods on the DAVIS 2017 and YouTube-VOS datasets, with a ${\mathcal {J}} \& {\mathcal {F}}$ overall score of 82.0% and 79.7%, respectively. Meanwhile, STCVOS achieves a high inference speed of approximately 30 $\mathcal {FPS}$.

Appendix A Equivalent derivation of the formula

In this supplementary material, we provide a mathematical derivation of EMR discussed in Sect. 3.3. This is an efficient attention mechanism which is mathematically equivalent to dot-product attention but substantially faster in terms of memory reading.

Given the memory $M_{t}$ with the embedded key $K_{M}=\left\{ K_{M}\left( j\right) \right\} \in R^{k\times C/8}$ and query key $K_{Q}=\left\{ K_{Q}\left( i\right) \right\} \in R^{H\times W\times C/8}$, under the condition of using softmax used in STM [23] as a normalization function, we can write the result of the dot product between i and j as follows:

$$\begin{aligned} \hat{V_{Q}} \left( i\right) =\sum _{j} \frac{e^{K_{M}\left( j\right) K_{Q}\left( i\right) ^{T} }}{\sum \nolimits _{j} e^{K_{M}\left( j\right) K_{Q}\left( i\right) ^{T} }} V_{M}\left( j\right) , \end{aligned}$$

(A1)

where H, W, and C are the height, width, and channel size of ResNet-50 [40], respectively. Also, k represents the number of pixel features in $M_{t}$, and $V_{M}=\left\{ V_{M}\left( j\right) \right\} \in R^{k\times C/2}$ denotes the embedded value of the memory. Then, we can rewrite Eq.A1 generally by using any normalization function, formulated as sim, as follows:

$$\begin{aligned} \begin{gathered} \hat{V_{Q}} \left( i\right) =\sum _{j} \frac{sim\left( K_{M}\left( j\right) ,K_{Q}\left( i\right) \right) }{\sum \nolimits _{j} sim\left( K_{M}\left( j\right) ,K_{Q}\left( i\right) \right) } V_{M}\left( j\right) ,\\ sim\left( K_{M}\left( j\right) ,K_{Q}\left( i\right) \right) \ge 0.\end{gathered}. \end{aligned}$$

(A2)

We then propose a linear attention mechanism using a first-order approximation of the Taylor expansion on Eq.A1:

$$\begin{aligned} e^{K_{M}\left( j\right) K_{Q}\left( i\right) ^{T} }\approx 1+K_{M}\left( j\right) K_{Q}\left( i\right) ^{T}. \end{aligned}$$

(A3)

However, the above approximation cannot guarantee the non-negativity. Therefore, we normalize $K_{M}\left( j\right)$ and $K_{Q}\left( i\right)$ by $l_{2}$ norm to ensure $K_{M}\left( j\right) K_{Q}\left( i\right) ^{T} \ge -1$. We can write Eq.A3 as:

$$\begin{aligned} sim\left( K_{M}\left( j\right) ,K_{Q}\left( i\right) \right) =1+l_{M}\left( j\right) l_{Q}\left( i\right) ^{T}, \end{aligned}$$

(A4)

where $l_{M}\left( j\right) =\frac{K_{M}\left( j\right) }{\parallel K_{M}\left( j\right) \parallel _{2} }$ and $l_{Q}\left( i\right) =\frac{K_{Q}\left( i\right) }{\parallel K_{Q}\left( i\right) \parallel _{2} }$. Then, Eq.A2 can be rewritten as:

$$\begin{aligned} \hat{V_{Q}} \left( i\right) =\frac{\sum _{j} \left( 1+l_{M}\left( j\right) l_{Q}\left( i\right) ^{T} \right) V_{M}\left( j\right) }{\sum \nolimits _{j} \left( 1+l_{M}\left( j\right) l_{Q}\left( i\right) ^{T} \right) }, \end{aligned}$$

(A5)

and simplified as:

$$\begin{aligned} \hat{V_{Q}} \left( i\right) =\frac{\sum _{j} V_{M}\left( j\right) +l_{Q}\left( i\right) ^{T} \left( \sum _{j} l_{M}\left( j\right) \left( V_{M}\left( j\right) \right) ^{T} \right) }{N+l_{Q}\left( i\right) ^{T} \sum \nolimits _{j} l_{M}\left( j\right) }.\nonumber \\ \end{aligned}$$

(A6)

We find that the computational and memory complexity of Eq.A6 is $O\left( N\right)$ that greatly improves efficiency, but also has the same effect as the memory reader of STM.