A Graph Based Approach to Speaker Retrieval in Talk  Show Videos with Transcript-Based Supervision

Abstract

This paper proposes a graph based strategy to retrieve frames containing the queried speakers in talk show videos. Based on who is speaking and when information from the audio transcript, an initial audio-based step, that restricts the queried person to frames corresponding to when he/she is speaking, with a second step that analyzes visual features of shots is combined. Specifically, based on the production property of talk show video, (1) Shot based graph is constructed first. Then the densest sub-graph is returned as the final result. But instead of direct search (DS) of the densest part, (2) We model the intra node connection and inter node connection by a frame layer degree map to take into account the duration information within each shot node; (3)A graph partition strategy without restriction on the shape and the number of sub-graphs is proposed, in which shots containing the same person are more similar to each other. Experiments on one episode of the French talk show “Le Grand Echiquier” show more than 10% improvement to audio only method and more than 7.5% improvement to DS method on average.

This work is supported in part by the National 973 Project under Project No. 2007CB311002, and the National 863 Program under Project No. 2009AA01Z409. This material is based upon work funded by European K-Space Project and French Infom@gic Project.