To enhance the accuracy of remote sensing (RS) data analysis, cloud detection from the complex ground environment is crucial. We refer to clouds that are easily confused with similar background as weak targets clouds, including thin clouds, tiny clouds, cloud boundaries, clouds with snow’s existence or highlighted background’s existence. This article proposes a coarse-to-fine cloud detection network for weak target recognition. The network consists of two subnetworks: the scalable weak target feature extraction subnetwork (SWTFES) and the cascade weak target refinement subnetwork (CWTRS). SWTFES incorporates a multiscale feature extraction module (MFEM) with different scale receptive field branches and an attention-based cross-layer fusion module (ACFM) to characterize cloud at various scales. The improved reverse attention operation and the cascade group reverse attention module (CGRAM) serve as the guiding principles in CWTRS, driving the network to progressively add and refine the weak target’s details to distinguish it from the complex background surface. We evaluate our methodology on four cloud datasets with various resolutions, varying from 0.5 to 16 m, and different satellites (including Gaofen-1 wide field of view (WFV), Sentinel-2, Gaofen-2, and WorldView-2). The experimental results demonstrate that WodNet has achieved excellent results in cloud detection in a variety of complex scenarios, compared to other models, performing state-of-the-art (SOTA) in four challenging datasets.