Sign language recognition systems suffer from the problem of signer dependence. In this letter, we propose a novel method that adapts the original model set to a specific signer with his/her small amount of training data. First, affinity propagation is used to extract the exemplars of signer independent hidden Markov models; then the adaptive training vocabulary can be automatically formed. Based on the collected sign gestures of the new vocabulary, the combination of maximum a posteriori and iterative vector field smoothing is utilized to generate signer-adapted models. Experimental results on six signers demonstrate that the proposed method can reduce the amount of the adaptation data and still can achieve high recognition performance.