Two important factors that decide the applicability of an object detection system are accuracy and speed. Modern convolutional object detection methods have achieved the accuracy level acceptable for a wide range of applications, but the most accurate ones are computationally expensive, requiring powerful hardware to achieve real-time detection. There is often a tradeoff between accuracy and complexity, which depends on the system configuration, e.g. the choice of certain parameters. The optimal configuration changes from application to application and cannot be decided beforehand. In this work, we explore various ways to optimize the speed by exploiting the characteristics of surveillance videos. We show that, when the number of classes is small, it is possible to exploit the characteristics of videos to automate the calibration of some key parameters of Faster R-CNN [15], which yields speed improvement at the minimum loss of accuracy. We experimentally evaluated the proposed method for detecting cars from a traffic surveillance video dataset. The results are promising: the system achieved comparable accuracy in terms of mAP while speeding up the whole detection process by a factor of two.