Two-directional two-dimensional canonical correlation analysis ((2D)$^2$CCA) directly seeks linear relationship between different image data sets without reshaping images into vectors. However, it fails in finding the nonlinear correlation. In this letter, a novel method named as two-directional two-dimensional kernel canonical correlation analysis is proposed, which is a nonlinear version of (2D)$^2$CCA and is able to find the nonlinear relationship between different image data sets. Experimental results in different expressions, illumination conditions and poses show the effectiveness of the proposed method.