Hyperspectral imaging can provide richer spectral information and can benefit cholangiocarcinoma histopathological image segmentation. However, deep-learning segmentation model designed for RGB image will disrupt the spectral structure in the first convolutional layer. One solution is treating the spectral dimension as an additional spatial dimension and using 3D convolution, but spectral dimension and spatial dimension cannot be simply equivalent. Another solution is treating the spectral dimension as sequence and using recurrent networks to extract spectral feature. This paper proposed a Swin-Spectral Transformer network. It follows the latter solution and proposed Spectral Multi-head Self-Attention (Spectral-MSA) in the spectral dimension. Then Spectral-MSA is combined with Shifted Window-based MSA (SW-MSA), named the Swin-Spectral Transformer, to acquire effective spectral and spatial feature representation. Also, this paper proposed spectral aggregation token for effective dimensional reduction to get 2D segmentation result. Finally, experiment shows the proposed method outperforms other competing methods and obtains aAcc of 90.87%, mIoU of 75.47% and mDice of 85.29% on the refined cholangiocarcinoma segmentation dataset.