We propose a semantic mapping-based remote sensing image matching method, which aims to obtain the matching positions of candidate patches containing key points directly on the reference image, avoiding the use of cost-volume search pixel by pixel. First, a global context-fusing attention structure is created to fuse global semantic information for candidate patches with the entire sensed image. Then, a self-attention layer with semantic dependencies is proposed to extract the semantic dependencies on the reference image for cross-modal representation. The global receptive field provided by self-attention enables the proposed method to obtain the semantic mapping of candidate patches on the reference image. The experimental results show that the proposed method is insensitive to image distortion and achieves cross-modal matching of SAR-optical images with high accuracy, while still running several orders of magnitude faster. This ensures increased speed in remote sensing image analysis and pipeline processing while promoting new directions in learning-based registration.