Significant advances have been made in image deraining due to the use of kinds of deep neural networks. However, existing deep neural network-based methods usually contain significant abundant network parameters and thus leads to expensive computation cost, which limits the application of deraining technology in high-level vision tasks. In this paper, we propose a compact and flexible Tree-structured Channel-wise Refinement Block (TCRB) for efficient image deraining, which contains augmentation, refinement, and aggregation modules to better explore features. Specifically, the refinement module can progressively extract groups of more discriminative features from the channel augmented inputs, and then the aggregation module adaptively fuses features from the refinement module to preserve image details by leveraging the Enhanced Channel Attention (ECA) method. Moreover, we present a Tree-structured Channel-wise Refinement Network (TCRN) by stacking multiple TCRBs, which could achieve competitive performance as the complicated networks. We embed the TCRB into a Multi-scale Tree-structured Channel-wise Refinement Network (MTCRN) based on an encoder and decoder network architecture and show that it performs favorably against state-of-the-art deraining algorithms on both synthetic datasets and real-world rainy images, while reaching a better trade-off in terms of model parameters and inference time.