Small infrared target detection is a key and challenging issue in object detection and tracking systems. Existing algorithms can be mainly categorized into nonlocal-based or local-based methods. However, the detection performance degrades rapidly when facing highly heterogeneous backgrounds. This is mainly due to that they exploit only one kind of information (e.g., local or nonlocal) while sacrificing the other. Thus, an effective small target detection method is proposed to combine local and nonlocal priors. The former is obtained by a sliding dual window while the latter is realized by low-rank and sparse decomposition. Experimental results on three real datasets validate the effectiveness of the proposed framework, which is more stable and robust compared with several state-of-the-art methods, especially for the image scenes with heavy background clutters.