Building accurate human behavior models is necessary for ambient intelligence. However, human activity recognition (HAR) in continuously monitored physical space meets many challenges to achieve a good performance when using only simple computing resources. In this work, we model HAR as an edge classification problem for a collaborative event graph of context entities in a sequential bipartite graph form. We design a semantic learning framework, called KGAR, to perform HAR by mining, encoding, and exploiting deep semantic knowledge of activities in an end-to-end fashion. KGAR has three components: preprocessor, KGEncoder, and predictor. The preprocessor builds offline a tiny knowledge graph of activities, to model and capture multidimensional semantic relationships between activities and core context entities. KGEncoder encodes the knowledge graph of activities using improved graph neural networks (GNNs) models, to handle confusing context patterns. The predictor may be deployed using some lightweight deep neural network to produce real-time labels. Experimental results show that using KGEncoder in KGAR improves the performance of original deep neural networks by 25% - 439% on five datasets. The time of labeling each sensor event during testing with event streams is less than 0.5 ms. We have also conducted extensive experimental study to show that KGAR outperforms different types of models in more complex activity scenarios. We believe KGAR can be used for real-time HAR in real life with its high prediction performance and a low computing requirement.