Spatio-temporal Manifold Embedding for Nearly-Repetitive Contents in a Video Stream

Abstract

This paper presents a framework to identify and align nearly-repetitive contents in a video stream using spatio-temporal manifold embedding. The similarities observed in frame sequences are captured by defining two types of correlation graphs: an intra-correlation graph in the spatial domain and an inter-correlation graph in the temporal domain. The presented work is novel in that it does not utilise any prior information such as the length and contents of the repetitive scenes. No template is required, and no learning process is involved in the approach. Instead it analyses the video contents using the spatio-temporal extension of SIFT combined with a coding technique. The underlying structure is then reconstructed using manifold embedding. Experiments using a TRECVID rushes video proved that the framework was able to improve embedding of repetitive sequences over the conventional methods, thus was able to identify the repetitive contents from complex scenes.