In this study, transductive learning and binary hierarchical decision trees are used together to find discriminative embedding (projection) directions. The projection directions returned by the proposed methodology are used for dimensionality reduction and the accuracy of nearest neighbor classification is significantly improved. We choose random classes and samples to create multiple hierarchical trees, and transductive support vector machine (TSVM) classifier is used to separate the data samples at each node of the binary hierarchical trees. The normals of the separating hyperplanes returned by the TSVM are used for dimensionality reduction. Different strategies are used to combine the projection directions coming from different hierarchical trees. In all experiments significant improvement are obtained over the nearest neighbor using full dimensionality of the input space. Since dimensionality is reduced significantly, speed of classifier has also been improved.