Recently, cross-spectral image patch matching methods based on feature difference aggregation have achieved excellent performance, but they introduce a large number of parameters, limit matching speed, and have poor scalability. At the same time, only using feature difference learning (FD) to extract differential features will lead to the loss of consistent features between cross-spectral image patches. Therefore, we construct a novel four-branch efficient feature relation learning network (EFR-Net) without feature difference aggregation. Specifically, a new four-branch feature relation learning strategy is proposed, which reasonably combines multiple feature relation learning to comprehensively and effectively extract the differential features and consistent features between image patches. At the same time, we construct an efficient local attention (ELA) module with negligible parameters, which can learn some global context information, enhance the interaction of local information, and promote the extraction of discriminative features. In addition, a combined metric network is introduced to facilitate network optimization and improve network generalization. Furthermore, a public optical and synthetic aperture radar (SAR) image patch matching dataset with a patch size of $64\times64$ pixels is constructed based on the OS dataset, which is called the OS patch dataset. We also establish an experimental benchmark on this new dataset. Extensive experimental results show that the proposed EFR-Net achieves excellent performance on cross-spectral image patch matching (OS patch dataset, VIS–NIR patch dataset) and single spectral image patch matching (Brown dataset).