This paper illustrates three levels of video units for video segmentation from coarse to fine granularity, which include highlights, attentive visual change, and game status change. The boundaries between video shots are commonly known as scene change and the action of segmenting a video sequence into multiple shots is called scene change detection. However, different applications are suitable for different fine granularities of the shot boundary. In this paper, we take the content semantic into consideration to segment the video. In performance evaluation, we choose the sports programs as the testing data, because the sports video entertains large population of audiences and is very popular. The experimental results show the efficiency of the proposed method for sports programs.