Pansharpening is widely used to increase the spatial resolution of a multispectral (MS) image by fusing with a panchromatic (PAN) image that has high-spatial resolution and the same scene. In this paper, the similarities of MS and PAN images in cartoon-texture space are exploited. The cartoon and texture components of an image always contain the global structure information and the locally-patterned information, respectively. Therefore, the global and local spatial details (i.e., high-order information) could be preserved well in the fused high-spatial resolution MS image after leveraging the similarities of these images. Pansharpening is formulated as the optimization problem with respect to the cartoon-texture similarities between the MS and the PAN images in this work based on the aforementioned observation. Specifically, cartoon similarity is determined through gradient sparsity and formulated as a total variation term, whereas texture similarity is described according to the low-rank property. The alternative direction multiplier method is used to solve the optimization problem. In the experiment, the Gaofen-1 satellite dataset is used to compare the proposed method with other classical pansharpening methods. Experimental results demonstrate that our method outperforms the comparison methods in terms of visual and quantitative qualities.