Scene is an important semantic unit for video analysis, retrieval and browsing. However, due to the lack of a generic algorithm, many studies focus on specific methods for certain video genes, e.g., news, sports, etc. In this paper, we propose a general framework for scene segmentation. First, we construct a graph, in which the elements encode the shot-to-shot coherent characteristics of a video clip based on visual similarity and temporal relation between shots. In this step, we only exploit the inherent property of video itself and it is independent of video genres. Second, spectral method is applied on the graph to group shots into scenes. The proposed method is not only simple but also effective to deal with organized features. Experimental results validate the robustness of our method on different kinds of videos.