VisualCor system: search actor correlations in TV series

In content-based video understanding, actor association can usually capture the high-level semantics, which is also the focus of user's browsing and searching attention. Mining actor correlations is of great importance but left unexplored until recently. However, due to the complex actor relationships in videos, building and quantifying character correlations to reflect significant concurrent scenarios meets great challenges. In this paper, we report our work on actor correlation search and mining, in which we present a context-based actor correlations graph search framework for TV series. Our framework serves as a first attempt for effective actor association concurrence search. Firstly, we leverage face detection and tracking to locate actors and generate multi-pose face sets, and adopt 2D-PCA detector and centered nearest neighbor to cluster faces. We then achieve actor indexing with manually adjustment to ensure extreme precision. Secondly, to measure the actor association into a unified graph, we propose a context-based correlation analysis strategy in shot sequence. Considering video structure cues, a hierarchical concurrence measurement is proposed to further train the actor association graph. We have deployed our proposed framework onto an online actor correlation search system, VisualCor, which could be viewed as a visualized, content-based simulation of the Renlifang search engine. It contains an actor correlation graph interface to facilitate character association search and browsing in "Friends" soap operas (containing over 20 hours videos) with excellent ranking accuracy, efficiency, and the ability to reveal human perception.