In this paper, we propose to recognize manipulative hand gestures by incorporating symbolic constraints in a particle filtering approach used for trajectory-based activity recognition. To this end, the notion of situational and spatial context of a gesture is introduced. This scene context is incorporated during the analysis of the trajectory data. A first evaluation in an office environment demonstrates the suitability of our approach. Different from purely trajectory-based approaches, our method recognizes manipulative gestures including the information which objects were manipulated.