Fisher's linear discriminant (FLD) is one of the most widely used linear feature extraction method, especially in many visual computation tasks. Based on the analysis on several limitations of the traditional FLD, this paper attempts to propose a new computational paradigm for discriminative linear feature extraction, named “classifiability-based discriminatory projection pursuit” (CDPP), which is different from the traditional FLD and its variants. There are two steps in the proposed CDPP: one is the construction of a candidate projection set (CPS), and the other is the pursuit of discriminatory projections. Specifically, in the former step, candidate projections are generated by using the nearest between-class boundary samples, while the latter is efficiently achieved by classifiability-based AdaBoost learning from the CPS. We show that the new “projection pursuit” paradigm not only does not suffer from the limitations of the traditional FLD but also inherits good generalizability from the boundary attribute of candidate projections. Extensive experiments on both synthetic and real datasets validate the effectiveness of CDPP for discriminative linear feature extraction.