A lightweight multi-scale aggregated model for detecting aerial images captured by UAVs

Highlights

• The Multi-scale detection, detecting objects of various scales in aerial images.

• The multi-scale fusion of the feature maps obtained by attention, enhance details.

• Redistribute the weight of the loss to balance the accuracy between categories.

• Use pruning technology to compress models to accelerate detection efficiency.

Abstract

Detecting the objects of interesting from aerial images captured by UAVs is one of the core modules in the UAV-based applications. However, it is very difficult to detection objects from aerial images. The reason is that the scale of objects in the aerial images captured by UAVs varies greatly and needs to meet certain real-time performance in detection. To deal with these challenges, we proposed a lightweight model named DSYolov3. We made the following improvements to the Yolov3 model: 1) multiple scale-aware decision discrimination network to detect objects in different scales, 2) a multi-scale fusion-based channel attention model to exploit the channel-wise information complementation, 3) a sparsity-based channel pruning to compress the model. Extensive experimental evaluation has demonstrated the effectiveness and efficiency of our approach. By the proposed approach, we could not only achieve better performance than most existing detectors but also ensure the models practicable on the UAVs.

Introduction

Unmanned Aerial Vehicle(UAV) is an aircraft that carries no human pilot or passengers. It is guided autonomously, or by remote control [1]. Equipped with high definition camera, UAVs can be remarkably efficient on surveillance tasks. Compared with surveillance cameras with fixed position and angle, the camera assembled in UAVs has its own unique advantages, such as varying shooting angles and positions. Due to the particularities of UAVs cameras, UAVs have been widely used in many social applications, such as object detection [2], target tracking [3], and path planning [4], etc.

Detecting the objects of interesting from images captured by UAVs is one of the core modules in the UAV-based applications, which has received increasing attention from multiple research communities in recent years. In research, visual object detection to determine the location of a target (e.g., pedestrians or vehicles) in a natural image, which has been extensively studied in computer vision in past years, and is also becoming increasing helpful in satellite and aviation monitoring [5], smart city construction [6], and construction of ecological civilization [7]. The UAV is an ideal platform for object detection since the aerial imagery system in UAVs can capture the scene in the view from top. However, compared with object detection in ordinary images, there is a significant difference to detect objects from the aerial image taken in UAVs: firstly, the angles of objects in aerial images are mostly viewed from the top and oblique, and the objects in aerial image are more difficult to recognize; secondly, though the size of the object changes with the shooting angle, the researchers are gratified that due to the shooting angle of view, there are fewer objects that overlap each other, so there is no need to pay too much attention to the overlap objects. Furthermore, traditional machine learning detection algorithms for UAVs [8] can not obtain good detection performance because the hand-crafted features are not very robust to changes in diversity. With the rapid development of deep learning frameworks, visual object detection has achieved great success for general object detection [9], [10]. Despite the great progress of generic object detection, they usually are just suitable for images captured from ground-based cameras and could not be directly transferred to UAV images. It is mainly due to two facts:

• To fulfill the detection task, UAVs are required to process the captured data efficiently. However, the traditional deep object detectors, such as Faster-RCNN [11], usually incur high computational cost since large numbers of convolutional operations are involved during the detection. And the lightweight and flexibility of UAVs limits its ability to carry sufficient resource to perform computing-intensive tasks online. Although some lightweight object detectors, such as SSD [9] and Yolov3 [10] could run fast and achieve promising performance, they are mainly designed for generic objects, not ideal for UAV images.

• Traditional deep object detectors usually require high-resolution images as input for high detection accuracy. However, the images captured by UAVs have varying object scales and usually contain many very small objects with low resolution, which are very difficult to detect. In order to cope with this problem, some improved algorithms are proposed for small object detection, such as SNIP [12] and Cascade-RCNN [13]. These algorithms have greatly improved the detection accuracy of small targets, but most of them rely on RPN [11] network and pyramid structure, which results in slow inference speed and the weak versatility of models. They are more suitable for offline detection tasks with low real-time requirement. Recently, Zhang et al. proposed a SlimYolov3 [14] detector which can perform the detection in real-time and is suitable for deploying on UAVs but the accuracy needs to be improved.

Despite some pioneering efforts for the object detection in UAV images, it still remains a under-explored task due to the above-mentioned challenges. To fill the research gap, this paper contributes a deep convolutional network-based fast object detector, named Deep Slight Yolov3 (DSYolov3) based on the classical Yolov3 [10] framework, to make the widely used Yolo series models be better applied to the task of detecting objects of different proportions in the images captured by the UAVs. Compared with the original Yolov3 framework, we propose a network structure with two additional components to make the model work better for scale-varying object detection. First, we propose to utilize multiple detection headers connected to different layers in the backbone network to detect objects at different scale. Then, we design a Multi-scale Fusion of Channel Attention Model (MFCAM) to exploit the channel-wise information complementation. Besides, a practical object detector for the UAVs has strict limits on the model parameters for high efficiency. To reduce the model redundancy in the manually-designed object detectors [15], [16], we design a simple and effective model pruning strategy to compress the proposed DSYolov3 framework by discarding the unimportant components without significantly affecting the performance for running efficiently in UAVs applications.

Compared to most existing small object detectors, our method is a one-stage detector and could obtain real-time performance while achieving better accuracy. After model pruning, the models are suitable for deploying on the UAVs. Our contributions are summarized as follows.

• To fully combine the feature from different layers of a CNN backbone and detect object with different sizes, we optimize the Yolov3 model by modifying original three-level decision discrimination network to five-level. Detecting objects at different scales can locate small objects and classify them more accurately.

• We propose a module named MFCAM (Multi-scale Fusion of Channel Attention Model) which combine the channel-wise attention mechanism and spatial pyramid pooling, to fuse the channel-wise feature at different scales for the scale-varying object detection tasks.

• We adopt a simple model pruning strategy [15], [16] to balance the accuracy and the computational complexity by discarding the unimportant channels in the model, so that the model could be able to run efficiently in UAVs applications.

Extensive experimental evaluation on VisDrone2018-Det benchmark dataset and UAVDT dataset shows that it could achieve promising performance in small object detection task and the pruned model is applicable in UAVs applications. The rest of this paper is organized as follows: Section 2 briefly reviews the related works on object detection algorithms in deep learning, attention mechanisms and model compression algorithms. In Section 3, we introduce our proposed framework and adopted approaches, followed by the extensive experimental details in Section 4. Finally we analyze our approach in Section 5 and conclude the work in Section 6.