In this paper, we propose to represent an image as a local descriptor tensor and use a multilinear supervised neighborhood embedding (MSNE) for discriminant feature extraction, which is able to be used for subject or scene recognition. The contributions of this paper include: (1) a novel feature extraction approach denoted as the histogram of orientation weighted with a normalized gradient (NHOG) for local region representation, which is robust to large illumination variation in an image; (2) an image representation framework denoted as the local descriptor tensor, which can effectively combine a moderate amount of local features together for image representation and be more efficient than the popular existing bag-of-feature model; and (3) an MSNE analysis algorithm, which can directly deal with the local descriptor tensor for extracting discriminant and compact features and, at the same time, preserve neighborhood structure in tensor-feature space for subject/scene recognition. We demonstrate the performance advantages of our proposed approach over existing techniques on different types of benchmark database such as a scene data set (i.e., OT8), face data sets (i.e., YALE and PIE), and view-based object data sets (COIL-100 and ETH-80).