Lossy compression will inevitably introduce image artifacts in the decoded image and degrade the image quality. In recent years, convolutional neural network (CNN) has been exploited for removing compression artifacts with great success. However, most existing CNN-based methods only utilize image's local prior without considering the global prior on the training of their networks. In this letter, a novel CNN, called the local and global priors network (LGPNet), is proposed that simultaneously learns both the local and the global priors for removing compression image artifacts. To achieve this goal, a dual-attention unit (DAU) is developed and incorporated into the well-known U-Net architecture for learning a better local prior. Meanwhile, the global prior is also learned from the entire image via our proposed global prior network. Extensive experimental results have clearly demonstrated that our proposed LGPNet is able to effectively remove image artifacts and greatly improve the image quality of JPEG-compressed images.