RGB-Infrared person Re-identification (RGB-IR Re-ID) task aims at using RGB (infrared) query image to matching person in infrared (RGB) gallery image, the large modality gap between two modalities makes the task very challenging. Different modality image’s feature lacked modality-shared information essentially, leading to large cross-modality discrepancy, existing method are not using modality relations to handle the discrepancy. In this paper, we proposed a novel Graph-based Modality-aware Relation Network (GMRN) to solve this problem, our method contains two parts: 1) fine-granularity multi-modality feature aggregation module to incorporate cross-modality information, 2) modality-aware graph convolution network utilize intra-modality relations to further learn discriminative features under two different modalities. Extensive experiments have made on two public RGB-IR Re-ID dataset SYSU-MMOI and RegDB, experiment results show that our method outperforms current state-of-the-art methods by a large margin. Our code is available at https://github.com/clsrsun/GMRN-ReID