The approach of applying deep learning-based object detectors to wideband spectrograms for signal detection, classification, and localization has garnered increasing interest. However, the diversity of signal bandwidths and durations results in significant variations in the scales and aspect ratios of signal bounding boxes within spectrograms. These characteristics pose the anchor mismatch problem for the anchor-based detectors in existing methods, leading to inaccurate time-frequency localization and suboptimal detection performance. This letter proposes a novel signal detector that employs a concise anchor-free paradigm instead of anchors to detect signals. Furthermore, a coarse-grained classification to fine-grained regression strategy rather than direct regression is adopted to achieve more accurate time-frequency localization information. Experimental results demonstrate that the proposed detector outperforms the deep learning-based baselines.