Two novel unsupervised dimensionality reduction techniques, termed sparse distance preserving embedding (SDPE) and sparse proximity preserving embedding (SPPE), are proposed for feature extraction and classification. SDPE and SPPE perform in the clean data space recovered by sparse representation and enhanced Euclidean distances over noise removed data are employed to measure pairwise similarities of points. In extracting informative features, SDPE and SPPE aim at preserving pairwise similarities between data points in addition to preserving the sparse characteristics. This paper calculates the sparsest representation of all vectors jointly by a convex optimization. The sparsest codes enable certain local information of data to be preserved, and can endow SDPE and SPPE a natural discriminating power, adaptive neighborhood and robust characteristic against noise and errors in delivering low-dimensional embeddings. We also mathematically show SDPE and SPPE can be effectively extended for discriminant learning in a supervised manner. The validity of SDPE and SPPE is examined by extensive simulations. Comparison with other related state-of-the-art unsupervised algorithms show that promising results are delivered by our techniques.