Oriented object detection based on deep learning has received extensive attention while marking the Oriented Bounding Box (OBB) is time-consuming and laborious. For the oriented object detection task, this paper proposes a point-based weakly semi-supervised training strategy, which only requires the training set with an extremely small number (10%) of fully labeled images with OBB and the other with points. Specifically, following the common self-training pipeline, we propose Point to OBB Network (P2ONet) as the teacher model to generate the high-quality pseudo OBB for each point-annotated object. Inspired by channel attention, we introduce Group Attention to P2ONet to better tackle the opposite assignment for similar proposals in the training task leading by the point label assignment strategy. Furthermore, by exploring the constraint in normal self-training pipeline, we propose Confidence-Aware Loss to alleviate the impact of inaccurate pseudo-boxes. Experiments on the DOTA dataset show the close performance between our method and normal oriented object detection training methods with remarkably lower labeling costs.