In this paper, we propose a novel face authentication scheme using the active appearance model (AAM) and the hidden Markov model (HMM). The proposed face authentication system can be divided into two parts. First, the AAM is used to extract the low-dimensional feature vectors including combined texture and shape information of individual face images. The extracted feature vectors are further classified into several clusters using vector quantization. The clustered feature vectors are then characterized using HMMs to make full use of the temporal information across the face images. After all parameters in the HMMs are calculated, we can dynamically determine the thresholds for face authentication. An iterative algorithm is also proposed to automatically determine a suitable number of HMM states and a suitable number of observation classes to achieve good authentication accuracy. The experimental results show the efficacy of the proposed method