Video-based person re-identification (Re-ID) aims at matching the video snippets of the same person across multiple cameras. The ubiquitous appearance misalignment is a critical challenge in video person re-identification. Existing alignment-based methods rely on off-the-shelf human parsing models and cannot handle anomalous appearance information (e.g., obstacles and pedestrian interference) in video sequences. In this paper, we propose Anomaly-Aware Semantic Self-Alignment (ASSA), a novel video-based person Re-ID framework that seeks out body parts without prior human topology information and learns part-based feature representations against anomalous information. The proposed ASSA performs part classifier training and part-aligned representation learning alternately. For the classifier training, we design a Salient Region Extraction module to segment the entire foreground from the background in each input frame. Furthermore, a novel Anomaly-Aware Refinement module is proposed to suppress the influence of anomalous interference. Extensive experiments on three prevalent benchmarks demonstrate the effectiveness and superiority of the proposed framework.