This paper presents a new method for recognizing trajectory-based human activities. We use a discriminative latent variable model in our proposed method, which considers that human trajectories are made up of some specific motion regimes, and different activities have different switching patterns among the motion regimes. We model the trajectories using Hidden Conditional Random Fields (HCRFs) and the motion regimes act as sub-structures in the model. Experiments using both synthetic and real data sets demonstrate the superiority of our model in comparison with other methods, including Hidden Markov Models (HMM) and Conditional Random Fields (CRFs).