Transmission line (TL) fasteners play the role of connecting components in smart-grid transmission processes with abnormal TL fastener states, seriously impacting the power supply. Therefore, regular detection of TL fasteners is significant. However, the images taken by unmanned aerial vehicle have problems, such as small size and complex background, which bring great challenges to the existing object detection models. Based on this, this article proposes a multilevel spatial refinement network (MSRN), including an attention-guided receptive field enhanced feature pyramid network (ARFE-FPN) and a double refinement head (DR-Head). For the small target problem, ARFE-FPN first uses dilated convolution to expand the receptive field, and uses global average pooling to extract background activation values. Then, it performs channel weighting on TL fastener features under different fields of view. For the problem of complex background, DR-Head first constructs a semantic prediction task to realize the preseparation of foreground and background, and then combines the high-resolution feature map to further highlight the fastener features in the low-resolution feature map. Experiments on the TL fastener dataset show that MSRN has the best detection accuracy, and its AP can reach 92$\%$.