Point set registration (PSR) is an important problem in computer vision, robotics and biomedical engineering communities. Usually, only positional information at each point is adopted in a registration. In this paper, the orientational vector (or normal vector) associated with each point is also utilized. Generalized point set registration is formulated and solved under the Expectation-Maximization (EM) framework. In the E-step, the posterior probabilities representing the correspondence probabilities are computed. In the Mstep, rigid transformation parameters including the rotation matrix, the translation vector are updated. The proposed algorithm stops when it converges to the optimal solution or a maximum number of iterations is achieved. The observed position set and normal vector set are assumed to follow Gaussian Mixture Models (GMMs) and Fisher distribution Mixture Models (FMMs), respectively. To further improve our algorithm's robustness, the hybrid mixture models (HMMs) are assumed to be inhomogeneous. Experimental results on the surface points extracted from a human femur' CT model show that our algorithm can achieve lower registration error, is more robust to noise and outliers than the state-of-the-art registration methods.