The object detection of high-resolution optical remote sensing images is an important part of remote sensing technology. It has significant application value, but the influence of various imaging factors usually causes changes in the object features which greatly increases the difficulty of object detection. This paper proposes a depth regression-based CNN object detection algorithm combined with transfer learning to overcome the difficulty. The experiments have shown that the algorithm effectively improves the speed and precision of remote sensing image object detection, which outperforms other state-of-the-art detection methods. Our algorithm achieves 90.70% average precision (AP) in the test set and the average detection time of each image block is about 0.021s, which has lower time overhead and better robustness to rotation and illumination changes of the object.