Sign Language Recognition (SLR) systems are mostly based on Hidden Markov Model (HMM) and have achieved excellent results. However, the assumption of frame independence in HMM makes it inconsistent with the characteristic of strong temporal correlation in sign language signals. Polynomial Segment Model (PSM) explicitly represents the temporal evolution of sign language features as a Gaussian process with time-varying parameters. In this paper PSM is first introduced to SLR framework to solve the temporal correlation problem. Considering the correlation among the coefficients of polynomial trajectory’s different orders, Mahalanobis distance is used as the classification criterion to evaluate the likelihood of test data. Experimental results show that our method outperform the conventional HMM methods by 6.81% in recognition accuracy.