In this paper, we propose a novel technique for model-based recognition of complex object motion trajectories using hidden Markov models (HMM). We build our models on principal component analysis (PCA)-based representation of trajectories after segmenting them into small units of perceptually similar pieces of motions. These subtrajectories are then grouped using spectral clustering to decide on the number of states for each HMM representing a class of object motion. The hidden states of the HMMs are represented by Gaussian mixtures (GM's). This way the HMM topology as well as the parameters are automatically derived from the training data in a fully unsupervised sense. Experiments are performed on two data sets; the ASL data set (from UCI's KDD archives) consists of 207 trajectories depicting signs for three words, from Australian Sign Language (ASL); the HJSL data set contains 108 trajectories from sports videos. Our experiments yield an accuracy of 90+% performing much better than existing approaches.