RGB-D object recognition is an essential but challenging task in computer vision. Most existing methods often exploit popular neural network architectures and various multi-modal fusion manners, which fail to explore the relation between low-level and high-level feature representations. To address this issue, we propose a convolutional recurrent fusion method for RGB-D object recognition. Firstly, a novel regularization strategy, called CutResize, is utilized to generate more reliable multi-scale RGB-D fusion input. Secondly, a convolutional attention gate recurrent unit (ConvAGRU) is proposed to fuse multi-modal features, which can capture the relation between low-level and high-level feature representations. Finally, the network recognizes RGB-D objects in an end-to-end manner. Especially, the CutResize uses the scaled image from another patch to mix with the input image, which avoids the mixed image only contains background and improves the localization ability of the model for small-scale features. The proposed ConvAGRU enhances the context correlation of the network and produces more compact and robust multi-modal fusion features. The proposed method is evaluated on several public datasets (including CIFAR-10, CIFAR-100, JHUIT-50, and OCID datasets). The experiment results show that the proposed method obtains better recognition performance than existing methods and achieves state-of-the-art results.