Recognition of sequential human activities, such as “sitting down” and “standing up”, is a common but challenging problem in human-robot interaction, which requires modeling their underlying temporal patterns. Although previous sequence modeling methods, such as Hidden Conditional Random Fields (HCRFs), demonstrated satisfactory recognition accuracy, they do not explicitly model the uncertainty in underlying temporal patterns, which can provide valuable information to characterize sequential activities. To address this problem, we introduce a novel Minimum Uncertainty HCRF (MU, or μHCRF). Different from traditional HCRF-based techniques that only utilize the negative log-likelihood of the categories' conditional probability as the loss function, the proposed μ-HCRF also introduces a regularization term to model the underlying temporal pattern of the latent variables. As another theoretical contribution, we provide a derivation to show that the formulated problem has a closed-form solution, and prove that inference of the proposed μHCRF is tractable. Extensive empirical study is performed to evaluate our approach, using four public benchmark datasets. Experimental results have shown that our μHCRFs outperform previous techniques and achieve state-of-the-art performance on human activity recognition, especially on sequential activities.