Speaker recognition plays an important role in speech processing and classification. In this paper, we propose a features extraction method using Perception Auditory Factor to improve the performance of speaker recognition in noisy environment. After the speech enhancement based on auditory perception characteristic and the 2-dimension enhancement for spectrogram, speech distribution is obtained from noisy speech, which is combined with perception speech content to extract Perception Auditory Factor. Speech features are extracted by using Perception Auditory Factor for speaker recognition. The experiments prove that, in four different noisy environments, the proposed method improves the robustness to the compared method in a wide range of SNR from −10 dB to 20 dB. Thanks to the proposed method, the average recognition correctness rates increase 25.2%, 19.1% and 9.7% to the three compared methods respectively. The proposed method is more suitable for speaker recognition in noisy environment.