The objective of this work is to provide a simple and efficient method to detect human faces in the compressed domain. We separate skin regions from non-skin regions using the Bayesian decision rule with a Gaussian mixture model (GMM). Next, we detect the location of a face using a deformable template that considers shape and orientation of the face. Then, face candidates are verified by using the second moments computed directly from the DCT coefficients in the face region. In particular, the coefficients are manipulated differently according to the orientation of the face. Good results have been obtained for a large variety of video sequences. This algorithm can be applied to JPEG images as well, without any modification.