The reflection problem often occurs when imaging through a semitransparent material such as glass. It degrades the image quality and affects the subsequent analyses on the image. Traditional single-image based reflection removal methods assume the reflection is blurry. Deep neural networks (DNNs) are, then, used to identify the blurry reflection and remove it. However, it is often that the blurry reflection still contains strong edges. They will be treated as the background and kept in the image. In this letter, we propose a novel two-stage DNN based reflection removal algorithm. In the first stage, we include a new feature reduction term in the loss function when training the network. Due to its strong reflection suppression ability, the reflection components in the image can be more effectively suppressed. However, it will also attenuate the gradient values of the background image. For recovering the background, in the second stage, we first estimate a reflection gradient confidence map based on the initial estimation result and use it to identify the strong background gradients. Then, we use a generative adversarial network to reconstruct the background image from its gradients. Experimental results show that the proposed two-stage approach can give a superior performance compared with the state-of-the-art DNN based methods.