Objective: Registration between phases in 4D cardiac MRI is essential for reconstructing high-quality images and appreciating the dynamics. Complex motion and limited image quality make it challenging to design regularization functionals. We propose to introduce a motion representation model (MRM) into a registration network to impose customized, site-specific, and spatially variant prior for cardiac motion. Methods: We propose a novel approach to regularize deep registration with a deformation vextor field (DVF) representation model using computed tomography angiography (CTA). In the form of a convolutional auto-encoder, the MRM was trained to capture the spatially variant pattern of feasible DVF Jacobian. The CTA-derived MRM was then incorporated into an unsupervised network to facilitate MRI registration. In the experiment, 10 CTAs were used to derive the MRM. The method was tested on 10 0.35 T scans in long-axis view with manual segmentation and 15 3 T scans in short-axis view with tagging-based landmarks. Results: Introducing the MRM improved registration accuracy and achieved 2.23, 7.21, and 4.42 mm 80% Hausdorff distance on left ventricle, right ventricle, and pulmonary artery, respectively, and 2.23 mm landmark registration error. The results were comparable to carefully tuned SimpleElastix, but reduced the registration time from 40 s to 0.02 s. The MRM presented good robustness to different DVF sample generation methods. Conclusion: The model enjoys high accuracy as meticulously tuned optimization model and the efficiency of deep networks. Significance: The method enables model to go beyond the quality limitation of MRI. The robustness to training DVF generation scheme makes the method attractive to adapting to the available data and software resources in various clinics.