Deep learning has emerged as a prominent technique in the field of holographic imaging, owing to its rapidity and high performance. Prevailing deep neural networks employed for holographic image reconstruction predominantly rely on convolutional neural networks (CNNs). While CNNs have yielded impressive results, their intrinsic limitations, characterized by a constrained local receptive field and uniform representation, pose challenges in harnessing spatial texture similarities inherent in holographic images. To address this issue, we propose a novel hierarchical framework based on self-attention mechanism for digital holographic reconstruction, termed HoloFormer. Specifically, we adopt a window-based transformer block as the backbone, significantly reducing computational costs. In the encoder, a pyramid-like hierarchical structure enables the learning of feature map representations at different scales. In the decoder, a dual-branch design ensures that the real and imaginary parts of the complex amplitude do not exhibit cross-talk with each other. During the training phase, we incorporate contrastive regularization to maximize the utilization of mutual information. Overall, our experiments demonstrate that HoloFormer achieves superior reconstruction results compared to previous CNN-based architectures. This progress further propels the development of deep learning-based holographic imaging, particularly in lensless microscopy applications.