Human activity recognition has many valuable applications in computer vision. Unlike existing works, the challenging problem of the similarity degree of skeleton-based human postures is addressed. In this paper, the Relation Matrix of 3D Rigid Bodies (RMRB3D), which is a compact representation of postures, makes a powerful way to compute the similarity degree between postures. Then representative postures are built through Spectral Clustering (SC) on sample data and action sequences of discrete symbols will be generated according to a global linear eigenfunction constructed by Spectral Embedding (SE). Finally, action classifier can be modeled as temporal order by using Dynamic Time Warping (DTW) and Hidden Markov Model (HMM). The experimental evaluations of the proposed method on challenging 3D action datasets show that our approach achieves promising results.