Accurate and robust object detection for surveillance videos hold immense potential applications in the field of public security. However, because of the artifacts in the surveillance video, such as motion blur, noise, low illumination, etc., relying solely on single-frame object detection algorithms may not guarantee the trustiness of surveillance video data analysis. Nevertheless, surveillance videos also exhibit characteristics of stable background and high inter-frame correlation. Therefore, in this paper, we introduce a novel model for surveillance object detection based on vision transformer with inter-frame multiscale probabilistic cross-attention. This model leverages Inter-Frame Semantic Cross-Attention (IFSC) to capture dynamic spatio-temporal features, thereby improving detection performance in low-quality frames. Additionally, it employs Inter-Frame Probabilistic Sparse Cross-Attention (IFPSC) to highlight salient features and suppress background features, enhancing the robustness of surveillance object detection. The experimental results on the UA-DETRAC dataset have demonstrated that the proposed surveillance object detector outperforms other SOTA models and achieves an optimal balance between speed and accuracy.