Visual tracking has seen considerable success with fully convolutional Siamese networks (SiamFC). However, the features extracted by SiamFC are redundant in practical applications. Most of the feature channels are ineffective or even serve as noise, which adversely affects tracking performance. The convolutional kernels tend to converge during the training phase of SiamFC, thereby the output features are typically redundant or irrelevant. In this paper, we propose a group residual convolutional Siamese network (GRSiamFC) to address this problem. Specifically, GRSiamFC divides the convolutional layers used for feature extraction into groups and trains each group in turn. Then the residuals between the output of previous groups and the true labels are used as losses to guide the current group to learn the residuals that were not fitted by the previous group, thereby improving the discriminative ability of the model. Extensive experiments on OTB50, OTB100, UAV123, and UAV20L show that the proposed GRSiamFC significantly improves the performance of SiamFC while keeping the complexity unchanged.