Image registration and the estimation of spatial and spectral blur kernels are essential steps before fusing panchromatic image (PAN) and multispectral image (MSI). Usually, these basic steps are performed separately, which will lead to error accumulation and ultimately affect the fusion performance. In this paper, we propose a novel deep learning (DL) based framework which can jointly register images and learn the blur kernels of original PAN and MSI. Specifically, we first construct a convolutional neural network (CNN) to learn the offsets between PAN and MSI, and the offsets are utilized to align the two images. Then, we analyze the relationship between the registered PAN and MSI, and design a tiny network for blur kernel learning. After solving the gradient derivation problems, we can combine the two networks and train them end-to-end. Experimental results on GF-2 satellite images demonstrate that the proposed method can significantly improve the fusion performance of some popular pansharpening methods.