Multi-action Prediction Using an Iterative Masking Approach with Class Activation Mapping

Abstract

While prediction techniques for multiple objects in images have become increasingly sophisticated, predicting multiple actions in videos remains challenging. Since most video training datasets only labeled a single action per clip, the trained three-dimensional convolutional neural network (3D CNN) model was limited to predicting a single action. To overcome this limitation, we propose an iterative method that combines a 3D CNN model with class activation mapping (CAM), which can achieve multi-object and multi-action prediction in videos. In each iteration, the action class with the highest score is output first. Then, the selected CAM method is applied to detect the primary action region. After masking this region in the input video, the masked video is re-input to the CNN model to predict actions occurring in other regions. In the experimental section, we used a video dataset of a single mouse with a single action label to train a 3D CNN model and tested the prediction performance using another set of composite videos of multiple mice with the same or different actions. The results demonstrate that the proposed method combined with Grad-CAM can correctly predict the individual actions of multiple mice in the videos. Moreover, we also analyzed a few of human action videos to illustrate the feasibility of this approach.