Automatic optical-to-synthetic aperture radar (SAR) image matching is still a challenging task due to the existence of severe nonlinear radiometric differences between the images and the presence of strong speckles in the SAR images. To address this problem, we propose a novel feature descriptor called rank-based local self-similarity (RLSS) for optical-to-SAR image template matching. The RLSS descriptor is an improved version of the local self-similarity (LSS) descriptor, inspired by Spearman's rank correlation coefficient in statistics. It can describe the local shape properties of an image in a discriminable manner. To further improve the discriminability, a dense RLSS (DRLSS) descriptor is formed with a dense scheme by integrating the RLSS descriptors for multiple local regions into a dense sampling grid. Experimental results conducted based on the optical and SAR image pairs demonstrated that the proposed descriptor was robust to nonlinear radiometric differences and it outperformed two state-of-the-art descriptors [dense LSS (DLSS) and histogram of orientated phase congruency (HOPC)].