In the face recognition application scenario, we need to process facial images captured in various conditions, such as at night by near-infrared (NIR) surveillance cameras. The illumination difference between NIR and visible-light (VIS) images causes a domain gap, and the variations in pose and emotion also make facial matching more difficult. Since heterogeneous face recognition (HFR) has difficulties in domain discrepancy, many studies have focused on extracting domain-invariant features, such as facial part relational information. However, when pose variation occurs, the facial component position changes and a different part relation is extracted. In this paper, we propose a part relation attention module that crops facial parts obtained through a semantic mask and performs relational modeling using each of these representative features. Furthermore, we suggest component adaptive triplet loss using adaptive weights for each part to reduce the intra-class distance regardless of the domain as well as pose. Finally, our method exhibits a performance improvement in the CASIA NIR-VIS 2.0 [1] and achieves superior results in the BUAA-VisNir [2] with large pose and emotion variations.