Matching and Localizing: A Simple yet Effective Framework for Human-Centric Spatio-Temporal Video Grounding

Abstract

Human-Centric Spatio-Temporal Video Grounding (HC-STVG) is a recently emerging task that aims to localize the spatio-temporal locations of the target person depicted in a natural language query. To tackle this task, we propose a simple yet effective two-stage framework, which is based on a Matching and Localizing paradigm. Under our framework, we decompose HC-STVG into two stages. In the first stage, we conduct cross-modal matching between the query and candidate moments to determine the temporal boundaries. Specifically, we develop an Augmented 2D Temporal Adjacent Network (Aug-2D-TAN) as our temporal matching module. In this module, we improve 2D-TAN [7] from two aspects: 1), A Temporal-Aware Context Aggregation module (TACA) to jointly aggregate the past contexts in forward direction and the future contexts in backward direction, which helps to learn more discriminative moment representations for cross-modal matching. 2), A Random Concatenation Augmentation (RCA) mechanism to combat overfitting and reduce the risk of unreasonably learning query-independent saliency prior, which is mistakenly provided by the training videos that only contain a single salient event. In the second stage, we utilize the pretrained MDETR [4] model to associate the language query with meaningful bounding boxes. Then, we conduct a query-based denoising procedure on the language-aware bounding boxes to obtain the frame-wise prediction for spatial localization. Experiments show that our simple yet effective framework can achieve a promising performance for the challenging HC-STVG task.