In view of the difficulty and low accuracy of small object detection in remote sensing images, this paper proposes a bidirectional cross-scale connection feature fusion network with an information direct connection layer and a shallow information fusion layer. Aiming at the problem that the detection targets in remote sensing images are mainly small and medium-sized targets, we fuse the shallow feature maps with rich spatial information in the bidirectional cross-scale connection feature fusion network instead of directly using the shallow feature maps for regression and classification. While ensuring the model inference speed, the detection accuracy of small objects is improved. At the same time, we use the information direct connection layer to perform feature fusion with the initial information in each iteration of the bidirectional cross-scale connection feature fusion pyramid to prevent the loss of small object information. Experimental results show that the algorithm proposed in this paper can obtain good accuracy and real-time performance on the NWPU VHR-10 dataset.