Egocentric Action Prediction via Knowledge Distillation and Subject-Action Relevance

Abstract

Egocentric action prediction aims to predict the future actions of the performer wearing the camera, given a partial video segment. The challenge in egocentric action prediction tasks (over action recognition) is the lack of context. The definition of an egocentric action as a noun-verb combination helps to obtain the context information from the video. However, the lack of training data to train two sequential deep-learning models to detect nouns and verbs remains a challenge. We propose a model involving two sequential GAN architectures where the first one determines the noun (based on the frames), and the next determines the verb (based on the optical flow). Each of these GAN architectures consists of a teacher and a student architecture. The teacher architectures are much deeper CNNs (pre-trained on a large dataset) compared to the student architectures. We further reduce the search space for the verb by applying the Reduced Verb Space Generator (RVSGen) algorithm between the noun prediction and verb prediction processes. RVSGen helps in proposing the most suitable verbs corresponding to the noun obtained by the noun predictor. Experimentation carried out on a benchmark dataset shows the efficacy of the proposed model compared to the state-of-the-art.