Detecting ships in synthetic aperture radar (SAR) imagery is a pivotal task for marine surveillance and security. Although many deep learning (DL) methods have been proposed for SAR ship detection, they still lack the ability to explore intrinsic scattering features, and their ship target detection capabilities necessitate further enhancements in complex labile environments, especially for small ships. For this reason, this letter proposes a dual branch scattering feature fusion network (SFFNet). First, scattering center feature maps are reconstructed, and then, we design a scattering feature attention fusion module (SFAFM) in view of reconstructed feature maps, which can enhance the prominent feature extraction ability of the network. Moreover, the backbone feature extraction architecture incorporates a dense depthwise block (DDWB) aimed at more effectively fostering information interactions for scattering features and improving the efficiency of the network. To validate the efficacy of the SFFNet, comprehensive experiments were conducted on two public datasets, namely, HRSID and LS-SSDD-v1.0, and experimental results indicated that the detection accuracy reached 98.3%, and the false detection rate decreased to 0.21%. The proposed method can achieve superior performance when benchmarked against other state-of-the-art detection methods.