Label Distribution Learning (LDL) is an effective paradigm for solving label ambiguity issues. When applying LDL, the specific label distribution dataset is often necessary. Unfortunately, most existing datasets include only logical label and obtaining label distribution is costly. To address this problem, Label Enhancement (LE) is used to recover label distribution from logical label. The existing LE methods only focus on mining label relevant information while neglecting the label irrelevant information. The learned label irrelevant information will confuse the prediction of model and reduce the performance and robustness. To address this limitation, we propose a novel LE method called Debiased Contrastive Learning for Label Enhancement (LE-DCL). Specifically, LE-DCL simultaneously learns label relevant information and removes label irrelevant information, which can improve the recovery performance and reduce the negative impact of irrelevant information. To help model identify relevant and irrelevant information accurately and prevent model overfitting, we also introduce a contrastive loss. Finally, we recover the label distribution based on the learned latent representation. Experiments on real-world datasets validate the effectiveness of our method.