For an image, texture unit (TU) is a small complete unit which characterizes the local texture of a given pixel and its neighborhood. Recently, TU-based approaches have been widely used in face recognition. This paper proposes a novel face representation and recognition approach based on TU. We make three major contributions: (1) we introduce a novel TU feature, Local Gabor Quarternary Pattern (LGQP), which incorporates both Gabor magnitude and phase information in a single TU code; (2) similarity measure of two TU images is treated as a tracking problem between two images, and we present a novel point-to-point matching (PPM) approach for TU similarity measure; (3) based on an integral histogram technique, the PPM similarity can be computed very rapidly. Experimental results on CMU-PIE and FERET data sets show that our method is able to reach very promising results.