A novel sports genre categorization algorithm based on representative shot extraction and geometry visual phrase(GVP) is presented in this paper. Performance of sports classification can be observably improved by generating reduced image set containing representative information and encoding spatial information into bag-of-words (BOW) model. Firstly, Shots containing significant information of videos are chosen by key-frame clustering. Secondly, GVP are searched by the co-occurrence of visual words in a spatial layout based on scale invariant feature transform (SIFT). Then visual words and GVP are concatenated to form enhanced histograms before SVM based classifying procedure. Compared with most existing methods, our algorithm is domain knowledge free as well as fully automatic and thus provides better extensibility. Experiments on a database of 10 sport genres with over 10257 minutes of videos from different sources achieved an average accuracy of 87.3%, which validates the robustness of our proposed algorithm over large-scale database.