Detecting dense small objects (DSOs) of varying scales still remains a challenging research problem in remote sensing imagery (RSI). Due to their weak feature extraction capabilities for small objects, most existing detection approaches struggle to handle the high proportion of DSO in RSI, thereby increasing the likelihood of missed detections. In addition, the close proximity and overlap of multiscale objects further complicate detection due to occlusion between bounding boxes. In this study, we systematically propose a novel loss function, remote sensing dense small target detection (RSDS), for detecting DSO in RSI, which contains three main components. The first is Gaussian reassignment loss (GRL), which adaptively redistributes sample weights to prevent any single sample (such as positive, negative, easy, and hard samples) from dominating the overall loss. To solve the zero-loss issue in traditional intersection over union (IoU) and intersection over ground truth (IoG) metrics when object boxes do not intersect, we design the Gaussian Wasserstein distance (WD) penalty loss, which models the eligible 2-D detection boxes as Gaussian distributions and calculates the similarity between them. The final one, which we called the occlusion box interaction loss, explains the attraction between DSO and the repulsion from their surroundings. Deploying RSDS not only significantly reduces the probability of missing DSO in RSI, but also enhances the detection accuracy in other similar computer vision tasks. Experiments on the HRSID, NWPU-10, and SSDD datasets show that some general models incorporating RSDS achieve precision improvements of 6.68%, 11.52%, and 5.26%, and 8.71%, 5.17%, and 9.34% in $\text {mAP}_{0.5}$ and $\text {mAP}_{0.5\text {:}0.95}$ , respectively, compared with other baselines. The code will be found at https://github.com/CCC0090/RDSD-Loss.