In this paper we propose a novel method to recognize different types of two-person interactions in video sequences. After extracting the spatio-temporal interest points (STIPs) from the visual scene through the 3D Harris detector, K-means clustering is applied to construct the visual codebook. We adopt a new feature selection procedure, called knowledge gain, based on the rough set theory to identify the most meaningful visual words in the codebook. For each video sequence, the histogram of selected visual words is used to train a multi-class SVM classifier. The algorithm is tested on two different datasets in order to demonstrate the applicability of the technique in different environmental configurations. Experimental results show that knowledge gain can improve the classification performance.