Typical infrared polarization image fusion aims to integrate background details in the infrared intensity and salient target in the degree of linear polarization (DoLP). Many fusion methods show advanced network architecture, but few works can form effective feature representations for the differences in prior distributions of the infrared intensity and DoLP, and the interference of DoLP with noise makes fusion more challenging. This article uses a learned low-rank decomposition model to extract low-rank representations containing background details in infrared intensity and sparse features with salient targets in DoLP. To reduce noise interference, we design a fusion module based on an attention-guided filter, where the infrared intensity serves as a guide map to suppress the background in DoLP. Moreover, a novel loss constraint is proposed to improve the fusion performance. Specifically, the fusion network is trained by reconstructing polarized images in different directions from the fused image. Quantitative and qualitative experimental results validate the effectiveness of our approach. In comparison to existing methods, our fusion model can better preserve the polarization salient target and suppress the background interference with fewer parameters. The source code is available at https://github.com/lkyahpu/PIPFNet.