Channel covariance matrix, or large-scale channel state information, is usually assumed to be perfectly known, which is not practical due to the quickly changing environment. This letter proposes a highly accurate channel covariance matrix estimation method based on fingerprint-based localization. In initial pilot assignment stage, we eliminate the impact of inference and noise by introducing random phase shift to estimate channel covariance matrices, which are regarded as fingerprints for estimating locations. Users are clustered based on locations and pilots are reassigned accordingly to improve estimation accuracy. Simulation results show that the proposed method achieves higher covariance matrix estimation accuracy.