Traffic object detection is the foundation for autonomous driving algorithm; its main challenge is to accurately and quickly detect the traffic objects. Current state-of-the-art methods typically employ anchor-free methods to improve the efficiency and accuracy, with few mentioning the impact of traffic rules on pedestrians and vehicles. That is, in a congested road sections, the same types of traffic objects gather together. To this end, we propose a novel traffic object detection method, Position Association and Semantic Fusion Learning Network (PASFLN), which can effectively leverage semantic information and position relationships of traffic objects to improve performance of object detection. In PASFLN, (1) we design the BiEfficientnet backbone network, which has two branches, one is the semantic branch for extracting strong semantic information, and the other is the positional branch for extracting strong location information; (2) to fuse these two branches, we design the MCIA-FFM channel attention fusion module; (3) to address the semantic gap between different feature layers, we design the Balanced-FPN, which can alleviate the semantic discontinuity problem of the feature pyramid and balance the proportion of semantic and positional information. We have conducted experimental studies on two publicly available datasets, BDD100K and Pascal VOC. Experimental results show that our PASFLN outperforms current state-of-the-art methods by an average of 0.4-2.1% in terms of performance.