Optical and synthetic aperture radar (SAR) remote sensing are crucial for ship detection. Integrating SAR’s all-weather imaging with optical data’s shape recognition enhances downstream applications. However, current cross-domain methods often use unsupervised or semi-supervised techniques for single-source detection, limiting their practical use in cross-domain ship detection. Inspired by human visual cortex mechanisms, this article proposes OptiSAR-Net, an end-to-end cross-domain multisource ship detection network. Specifically, OptiSAR-Net features dual adaptive attention (DAA) for extracting standard features from SAR and optical images, and bilevel routing deformable spatial pyramid pooling-fast (BSPPF) for adapting to multiscale changes. To mitigate SAR noise, we employ VoV-GSCSP with spatial shuffling attention (VSSA) in the neck. OptiSAR-Net achieved state-of-the-art average precisions (APs) of 88.6% and 91.3% on the optical datasets DOTA and HRSC2016, respectively, and showed strong performance on the SAR datasets HRSID and SSDD. On the cross-domain heterogeneous dataset (CDHD), OptiSAR-Net differentiated ship targets effectively with only 2.7 million parameters and 11.7 GFLOPs, achieving an inference speed of 89 FPS on an NVIDIA RTX 3090. These results demonstrate that cross-domain multisource detection significantly enhances performance and application potential compared to single-source detection. Code is available at https://github.com/SCNU-RISLAB/OptiSAR-Net.