Graph clustering aims to group the data into clusters according to a similarity graph, and has received sufficient attention in computer vision. As the basis of clustering, the quality of graph affects the results directly. In this paper, a Robust Adaptive Sparse Learning (RASL) method is proposed to improve the graph quality. The contributions made in this paper are three fold: (1) the sparse representation technique is employed to enforce the graph sparsity, and the l2,1 norm is introduced to improve the robustness; (2) the intrinsic manifold structure is captured by investigating the local relationship of data points; (3) an efficient optimization algorithm is designed to solve the proposed problem. Experimental results on various real-world benchmark datasets demonstrate the promising results of the proposed graph-based clustering method.