In this paper, we propose a framework to recognize complex human interactions. First, we adopt trajectories to represent human motion in a video. Then, the extracted trajectories are clustered into different groups (named as local motion patterns) using the coherent filtering algorithm. As trajectories within the same group exhibit similar motion properties (i.e., velocity, direction), we adopt the histogram of large-displacement optical flow (denoted as HO-LDOF) as the group motion feature vector. Thus, each video can be briefly represented by a collection of local motion patterns that are described by the HO-LDOF. Finally, classification is achieved using the citation-KNN, which is a typical multiple-instance-learning algorithm. Experimental results on the TV human interaction dataset and the UT human interaction dataset demonstrate the applicability of our method.