In this paper, a novel approach of hand gesture recognition is proposed. The spatial-temporal trajectory of hand gesture is first tracked by the mean shift embedded particle filter (MSEPF), and then represented in a static image using temporal template. Hand gestures are recognized by a two-layer classifier, which is based on statistical shape and orientation analysis of such temporal template based trajectories (TTBT). Experimental results show that our algorithm has high recognition rate.