Many real-world images (e.g., hyperspectral images (HSIs) and color videos) are usually partially observed and contaminated by Poisson noise, which hinder their subsequent applications. Recently, the tensor singular value decomposition (t-SVD)-based model was suggested for image recovery with Poisson observation. However, the classic t-SVD usually fails to capture the complex nonlinear structure of real-world images. To address this problem, we suggest an attention-guided low-rank tensor factorization (AGLRTF)-based model for image recovery with Poisson observation. More concretely, we consider a self-attention network as the transform in the t-SVD framework, which can treat the frontal slices unequally, allowing us to enhance the low rankness of the transformed frontal slices. Also, the self-attention block is learned unsupervised from the given data. Extensive experiments on HSIs demonstrate that our method achieves approximately a 2-dB higher PSNR metric compared with state-of-the-art methods.