Recognition and detection of interactive actions performed by multiple persons have a wide range of real-world applications. Existing studies on the human activity analysis focus mainly on classifying video clips of simple actions performed by a single person, whereas the problem of understanding complex human activities with causal relationships between two people has not been sufficiently addressed yet. In this paper, we employ systematically organized skeleton features enhanced with directional features, and utilize sparse-group lasso to automatically choose discriminative factors that help in dealing with interactive action recognition and real-time detection tasks. Experiments on two person interaction datasets demonstrate the superiority of our approach to the state-of-the-art methods.