Image deraining is an important task for the outdoor computer vision system in the rain days. Deep learning with attention mechanism has shown promising performance for rain removal. However, most of existing attention mechanisms select the locations of attention module empirically, and the receptive field of network is small and fixed. Motivated by the Attention in Attention (AiA) and multi-stage strategy, we propose a multi-stage deep neural network for image deraining, which is equipped with the refined AiA (RAiA) module. Specifically, RAiA extends the original AiA by exploiting the joint dependencies of channel and spatial information to generate the dynamic allocated weights. Moreover, the proposed network is carried out by a multi-stage U-Net fashion, which can extract features progressively, enlarge the receptive field of network, and improve the ability of image structures representation. We demonstrate the superiority of proposed method on the public synthetic datasets and real rainy images through comparing to seven state-of-the-art deep learning based methods.