This paper presents a new approach named primitive-based coupled-HMM for human natural complex action recognition. First, the system proposes a hybrid human model and employs 2-order B-spline function to detect the two shoulder joints in the silhouette image to obtain the basic motion features including the elbow angles, motion parameters of the face and two hands. Then, primitive-based coupled hidden Markov model (PCHMM) is presented for natural context-dependent action recognition. Lastly, comparison experiments show that PCHMM is better than the conventional HMM and coupled HMM.