Crop growth models and vegetation index (VI) based methods have been commonly used to estimate rice grain yield. However, the complicated model calibration procedure and the narrow time window limit the application of these two methods, respectively. The convolutional neural network (CNN) performs better than VI-based approaches on yield estimation at the ripening stage, but the generalization of CNN still needs to be improved. The objective of this study is to improve the generality of CNN in estimating plot-scale rice grain yield using high-resolution UAV-based RGB images. A new deep learning architecture with deep features decomposition is proposed. The results showed that the proposed network is more robust than the network without deep features decomposition when the phenological stage of the test set is different from the training set. The results indicate that the time-invariant features which only relate to rice yield can be decomposed by the proposed network, and demonstrate the stable performance of proposed CNN in a wider time window for rice grain yield forecasting.