Highway pavement health-condition and maintenance is crucial for traffic safety. Through our investigation, Prevalent methods still can not provide satisfactory results in highway pavement defect detection due to the diversity and complexity of the defects. In order to address the diversity and complexity of defects and enhance the performance and efficiency of highway defect detection, we propose a novel pavement defects detection framework termed HPDD (Highway Pavement Defect Detection)-Net. The proposed model adopts the Swin transformer as the backbone, FPN (Feature Pyramid Networks)as the neck, and the TOOD module as the Bbox_head. HPDD-Net integrates candidate region generation and object classification tasks through task alignment. Compared to prevalent two-staged detection models, our approach offers significant improvements in terms of speed and precision. Experimental results has again proved the effectiveness of our implementation by achieving excellent image classification performance and computational efficiency.