In this letter, a dual difference regression classification (DDRC) based face recognition method is proposed to reduce the effect of outliers in regression methods. DDRC consists of two parts: pixel difference binary pattern (PDBP) and difference regression classification (DRC). PDBP calculates the pixel differences in image domain, and then the pixel differences are binarized and converted to decimal values. DRC calculates the vector differences in feature domain to estimate an optimal predictor matrix. Experimental results on noiseless and noisy images of Yale B, Extended Yale B, JAFFE, AT&T, and GT databases with different image sizes show the effectiveness of the proposed face recognition method in terms of the recognition accuracy.