With the explosive growth of the social multimedia sharing, copyright protection and search result refinement are always the critical issues for the service operators. To resolve the problems, content-based near-duplicate video retrieval is developed in recent years. In this paper, we construct a condensed Pattern-based Prefix tree (PP-tree) to index the patterns of reference videos for fast retrieval. To calculate how likely a query video and a reference video are near-duplicates, a novel algorithm for discovering the temporal relations among patterns is proposed. Comprehensive experiments on public datasets are conducted to verify the effectiveness and efficiency of the proposed method. Experimental results show that the proposed near-duplicate video retrieval approach outperforms the state-of-the-art approaches in terms of precision, recall, and execution time.