Spectral clustering (SC) has been applied to analyze varieties of data structures over the past few decades owing to its outstanding breakthrough in graph learning. However, the time-consuming eigenvalue decomposition (EVD) and information loss during relaxation and discretization impact the efficiency and accuracy especially for large-scale data. To address above issues, this brief proposes a simple and fast method named efficient discrete clustering with anchor graph (EDCAG) to circumvent postprocessing by binary label optimization. First of all, sparse anchors are adopted to accelerate graph construction and obtain a parameter-free anchor similarity matrix. Subsequently, inspired by intraclass similarity maximization in SC, we design an intraclass similarity maximization model between anchor–sample layer to cope with anchor graph cut problem and exploit more explicit data structures. Meanwhile, a fast coordinate rising (CR) algorithm is employed to alternatively optimize discrete labels of samples and anchors in designed model. Experimental results show excellent rapidity and competitive clustering effect of EDCAG.