Video person re-identification (V-ReID) can leverage rich spatial-temporal information embedded in sequence data to achieve better accuracy. However, it is vulnerable to the interference of inaccurate and redundant noisy frames in each sequence as well as the background clutter and person-irrelevant pixels in each frame. To solve the above issues, this paper presents an adaptive interference removal framework (IRF) to learn discriminative feature representations by removing various interference. Our IRF mainly consists of two modules including an attention-guided adaptive interference frame removal module (IFRM) and an attention-guided adaptive interference pixel removal module (IPRM). IFRM and IPRM are designed to locate task-relevant keyframes and key pixels, respectively. IFRM adopts the attention mechanism to predict frame-wise scores to characterize the contribution of each frame to the final identification task. IPRM collaboratively utilizes camera identity classification loss, person identity classification loss, target attention loss, and person mask adversarial loss for extracting pure pedestrian representations. A progressive mask augmentation strategy is designed to restrain the data distribution of the generated person masks to further guide model training. Extensive experiments demonstrate that our models outperform state-of-the-art accuracy on seven person ReID datasets.