The current SAR ship detection datasets follow the traditional paradigm in computer vision field, that is, the image data and the corresponding annotations are provided. However, due to the special imaging mechanism of SAR, the amplitude or intensity data is limited and cannot offer sufficient electromagnetic information of target. Although the single-look complex SAR data is informative, it requires a large number of storage space and is difficult to interpret visually. To this end, a new benchmark for SAR ship detection is proposed in this paper, namely PhySAR-Det. It is constructed based on different levels of SAR products of Gaofen-3 satellite, including L1A complex-valued image and the geo-coded L2 product. We propose another microwave visual characteristic (MVC) product derived from L1A data and projected to L2 coordinates to characterize the scattering mechanisms. The L2 images with high visual quality attached with the corresponding MVC products form the proposed PhySAR-Det dataset. The experiments show that the MVC product improves the performance compared with only image data available. It will be available at https://github.com/XAI4SAR/PhySARDet.