We present a two-layer generative model for sport video mining that is composed of a two-layer observation model. The first layer is the Gaussian mixture model (GMM) using frame-wise camera motion for intra-shot analysis and the second layer is the hidden Markov model (HMM) involving the GMM as the mid-level observation for inter-shot analysis. A recursive model estimation method is developed for statistical inference which combines two Expectation Maximization (EM) algorithms. Specifically, the proposed generative model is used for American football play analysis where each play shot is classified into one of four classes, i.e., short plays, long plays, kicks and field goals. The experimental results show promising classification performance around 80%.