With the exponential growth of the web multimedia contents, illegal video copies are widespread and easy to be obtained from search engines and video sharing websites. For video copyright protection, near-duplicate video retrieval becomes more and more important. In this paper, we proposed a Pattern Set based Dynamic Programming (PSDP) algorithm to retrieve near-duplicate videos efficiently and effectively. In addition, the precise positions of the near-duplicate segments in videos can also be located. To better deal with the slow/fast motion and the dropped frame problem, the Time-shift Pattern set Similarity (TPS) is then applied. Two datasets are used to evaluate the effectiveness and efficiency of the proposed method, and the experimental results show that the proposed method outperforms the compared approaches in terms of precision and execution time in both two datasets.