Visual saliency detection via a recurrent residual convolutional neural network based on densely aggregated features

Highlights

• An aggregation module is designed to aggregate densely connected features with different resolutions, which enables fully communication and fusion between different network layers.

• An improved recurrent residual refinement mechanism is proposed, in which the residuals are learned recurrently under deep supervision to achieve continuous optimization of the saliency map.

• Extensive experiments prove the advantages of DAF-RRN over the state-of-the-arts.

Abstract

Current visual saliency detection algorithms based on deep learning suffer from reduced detection effect in complex scenes owing to ineffective feature expression and poor generalization. The present study addresses this issue by proposing a recurrent residual network based on dense aggregated features. Firstly, different levels of dense convolutional features are extracted from the ResNeXt101 network. Then, the features of all layers are aggregated under an Atrous spatial pyramid pooling operation, which makes comprehensive use of all possible saliency cues. Finally, the residuals are learned recurrently under a deep supervision mechanism to achieve continuous optimization of the saliency map. Application of the proposed algorithm to publicly available datasets demonstrates that the dense aggregation of features not only enhances the aggregation of effective information within a single layer, but also enhances external interactions between information at different feature levels. As a result, the proposed algorithm provides better detection ability than that of current state-of-the-art algorithms.

Introduction

Salient object detection is an important field of study in image information processing that seeks to recognize and capture the areas of images that are of particular relevance to the overall scene in a similar manner to which humans recognize salient objects in scenes. Accordingly, salient object detection represents a machine learning process that plays a key role in various computer vision tasks, such as pedestrian reidentification, target tracking, and semantic segmentation. Numerous advancements in recent years, such as the development of deep learning technology and the enhancement of feature expression ability, have resulted in significant and continuous progress in the performance of salient object detection algorithms [1], [2].

Salient object detection algorithms have benefited greatly from the ability of deep neural networks to extract effective features from unprocessed images automatically, and the rich semantic information generated can reflect differences in saliency between image regions. For example, Li et al. [3] constructed a multiscale deep features (MDF) model directly from three levels of features  obtained  from a convolutional neural network (CNN) [4], and the MDF model was applied for training super pixel-level classifiers to achieve salient object detection. The generated semantic information was demonstrated to enhance the smoothness of the salient region boundaries predicted by the image-level features, while the region-level depth features enhanced the accuracy the selected salient object regions. Therefore, Li et al. [5] further proposed a deep contrast learning (DCL) model that fused image-level and region-level convolutional features to improve the detection accuracy. Furthermore, the statistical information contained in conventional color, texture, and center priors was demonstrated to have a complementary effect on the detection capability associated with the deep features. A saliency detection model was proposed by Liu et al. [6] that combined a global model with local optimization using conventional features, where the recurrent fully convolutional network (RFCN) algorithm was employed to combine the salient prior map in the network structure. Both methods have improved the salient object detection performance of the algorithm.

In recent years, with the continuous development of deep learning technology, more models for saliency detection have been developed. Ning et al. [7] proposed a feature selection network that combines global and local fine-grained features to realize person reidentification. Ning et al. [8] proposed a model that has joint weak saliency and attention aware is proposed, which can obtain more complete global features by weakening saliency features. The model then obtains diversified saliency features via attention diversity to improve the performance of the model. Ning et al. [9] proposed a lightweight encoding and decoding network (EDNet). A perfect balance of network prediction accuracy and time consumption was achieved by using feature fusion and efficient deconvolution in the feature decoding stage. Gao et al. [10] proposed a salient object detection framework for surveillance applications based on the intelligent network with hierarchical cloud computing and the scene-specific edge computing. It can make the saliency detection model in the edge server suitable for its own specific scene, and performs well in other environments. Gao et al. [11] proposed a multi-stage context perception scheme to efficiently extract the contextual information corresponding to different-size receptive fields in the single image; and proposed the stage-wise refinement to allocate the label information to different parts of the network for helping the network to learn the enriched semantically common knowledge. Zhang et al. [12] proposed a region-proposal based optical flow strategy to suppress the saliency enhancement of non-salient regions due to the moving background. Besides, it develops the bidirectional Bayesian state transition strategy to model the motion uncertainty for refining the spatiotemporal saliency feature. These models have obvious breakthroughs in preserving detail information and improving operation efficiency, but the performances would decrease when dealing with complex scenes.

The present work addresses this issue by proposing an algorithm that uses densely aggregated features in conjunction with a recurrent residual network (DAF-RRN) to detect salient objects in images. Firstly, different levels of dense convolutional features are extracted from a basic network, and then the features of all layers are aggregated to make comprehensive use of all possible saliency cues. Finally, the residuals are learned recurrently under a deep supervision mechanism to achieve continuous optimization of the saliency map. The proposed algorithm is applied to publicly available datasets, and the results demonstrate that the dense aggregation of features not only enhances the aggregation of effective information within a single layer, but also enhances external interactions between information at different feature levels, which increases the detection ability of the proposed algorithm relative to that of current state-of-the-art visual saliency detection algorithms.

The contributions can be summarized as:

(1) The proposed model uses dense connection to aggregate effective information, which combines the technologies of feature dimension reduction, feature reuse and feature cascade. Compared with the existing methods, the feature information is more comprehensive.

(2) All levels of dense features with different resolutions are aggregated via the proposed aggregated module, which enables fully communication and fusion between different network layers. It enriches the obtained dense features with strong expression ability and abundant saliency cues.

(3) An improved Recurrent Residual Refinement Aggregating Network (R3ANet) is designed to generate initial saliency maps from the densely aggregated features, with which the residuals are learned recurrently under a deep supervision mechanism to achieve continuous optimization of the saliency map.

(4) Extensive experiments have been carried out on 3 datasets, which proves the advantages of DAF-RRN over the state-of-the-arts by quantitative and qualitative evaluation.