In recent years, a number of detectors have been proposed to improve the accuracy and speed of object detection tasks. However, poor detection performances for small objects and difficulties in optimizing deep networks remain critical challenges for object detection. We try to tackle these problems in two ways. First, we propose an innovative cross-fusion block (CFB) module that can enhance the representational power of features for instances of small objects. In CFBs, high-level features with rich semantic information and low-level features from different layers at the same scale are cross-fused together. Second, we propose a periodic oscillation attenuation learning rate (POA_lr) that can effectively skip some purely locally optimal solutions in the training process to obtain better detection accuracy. Extensive experiments on PASCAL VOC and MS COCO datasets show that CFB and POA_lr can achieve higher detection accuracy while maintaining real-time processing speeds. The code will be made publicly available.