Accurate segmentation of high-resolution remote sensing images is increasingly demanded, yet poses significant challenges for algorithm efficiency. Most current approaches pursue accuracy by employing global context information to enhance the overall consistency or utilize multi-scale features or attention mechanisms to optimize object details, without considering the network complexity uniformly. In this paper, we propose a light-weight semantic segmentation network for HRRS images by way of explicitly supervising the objects' body and edge features to optimize the overall consistency and object details of semantic segmentation at the same time. Furthermore, we introduce a score-based feature fusion module to establish the long-range dependency between pixels in the final stage of feature fusion (to combine the body and edge features) effectively. Experiments on ISPRS Vaihingen dataset show an obvious advantage of the proposed approach compared with the existing approaches. Specifically, it achieves 89.60% overall accuracy with only 2.83M parameters and 2.37GFLOPs computation costs.