Compressed wideband spectrum sensing has attracted much interest in recent years, which enables flexible spectrum sharing to improve the efficiency of scarce frequency resource. Despite the great potential for sub-Nyquist-rate sampling, existing high-accurate compression sensing (CS) methods unfortunately incur the extremely high computational complexity, e.g., in recovering the sparse signal or estimating the a priori information on sparsity. This creates a serious challenge in deploying real-time wideband sensing in the resource constraint platforms. In this work, we develop a fast compressed spectrum sensing method, which achieves accurate performance but also greatly reduces the computational complexity. Our new method jointly exploits the low-rank and sparse properties of a sub-Nyquist measurement matrix. We first design a low-complexity sparsity estimator, by approximating a large covariance matrix with multiple small matrices. To recover the sparse spectrum, we then formulate one low-dimensional non-convex optimization problem via random orthogonal projection, which makes the CS method more computationally efficient. As demonstrated on real datasets, our method reduces the computational complexity of wideband spectrum sensing by $\sim\! 10\times$; moreover, it achieves highly accurate results without compromising the reconstruction/sensing performance. Thus, it has great promise for real-time sub-Nyquist sensing on low-complexity platforms.