Reducing the inter-modality gap has been the core of visible-infrared person re-identification (VI-ReID). Most of the existing methods directly constrain the features to reduce the gap between the different modalities. However, the consistency of channel semantic information across different modalities is ignored. In this letter, channel alignment along with severe semantic constraints is proposed to alleviate the significant distribution differences between modalities. To obtain the optimum channel matching mode, a novel channel alignment mechanism termed Channel Instance Level Alignment (CILA) is proposed at the shallow layer, and matched channel features are constrained by the proposed Channel Instance Alignment (CIA) Loss. In the middle layer, we divide the features into multiple hierarchically-aware channel clusters and align the channel clusters by Channel Cluster Alignment (CCA) Loss. The proposed method is validated on SYSU-MM01 and RegDB, and extensive experiments show that the proposed method achieves competitive performance compared with the state of the arts.