With the popularization of drones, drone detection is more and more important for industrial and agricultural management. The video surveillance is an important way for drone detection. However, it is difficult to achieving a balance between performance and computational burden for some old video surveillance platforms with low computation capability. In this paper a drone detection method based on color-edge joint information and convolution neural network is proposed. Firstly, the background is eliminated by frame difference based method to reduce environment disturbances. Secondly, the frame difference result is processed by edge abstraction and morphology operation to find the candidate targets. Finally, the color information and abstracted edge information of candidate targets are jointly supplied to a convolutional neural network to detect the drone target. Due to the effective way to select candidate targets and concise neural network architecture, the proposed method can achieve satisfying detection performance on the platforms with low computing capability. The experiment results demonstrate that the proposed method is effective for drone detection.