In this paper, we present a novel method for recognizing human actions in videos. The method applies the human pose based features to describe actions and models the conditional probability relationship between feature sequences and actions using hidden conditional random field (HCRF). Given a video, limb masks are extracted by clustering image features in human region. Limb masks are helpful to reduce the interference from background and partially address the “double-counting” problem during the pose estimation. Then, the extracted pose sequence is smoothed using the kalman smooth to remove the noise and make the pose sequence consistent. Multiple kinds of feature sequences based on poses are extracted to describe the information of actions from different views. We train one HCRF for each feature sequence and combine the confidence from different HCRFs to improve the recognition accuracy. Experiments on the benchmark dataset show different features have their own advantages in action recognition and combine them can reach a good result.