Gender recognition is a hot research topic in recent years. Human-machine interfaces or video surveillance can be greatly improved if human gender can be recognized automatically. In this study, an embedded hidden Markov model is used for gender recognition. Video, which is recorded in different angles of view, is utilized to sample properties of each gender. Ten consecutive gait frames are segmented and organized as a composite image, which is used to establish EHMM. For video in each angle of view, two EHMMs are built and trained. The gender of the subject of a testing composite image is decided by the EHMM whose likelihood is most similar to the testing EHMM. We test the proposed approach using the CASIA Gait Database (Dataset B) in this study. Experimental results show that the proposed system can identify the gender of human accurately.