Fast and efficient pedestrian detection and vehicle detection has become an increasingly important task in the autonomous driving technology. In this paper, we propose a new pedestrian detection and vehicle detection algorithm based on the YOLOv2 with optimized feature extraction. We adopt the priori experience about the feature box sizes, instead of K-mean clustering algorithm in the original YOLOv2 algorithm. We first conduct statistical analysis on the dataset with a label of pedestrian label and vehicles, and then we design the initial value of the pre-selection box that is more in line with the characteristics of pedestrian and vehicle. Together with hard negative mining, multi-scale training, and model pretraining, the proposed algorithm not only improves the detection accuracy but also keeps the good detection efficiency. Experimental results on traffic benchmark record demonstrate that the optimized algorithm satisfies the real-time capability and the accuracy requirement of the lowspeed autonomous driving.