Co-salient object detection is a challenging task, which aims to segment the co-occurring salient objects in multiple images at the same time. To address this task, we propose an end-to-end Enhancement Location-Refinement Network (ELR-Net) to capture both salient and repetitive visual patterns from multiple images. For various scenarios, common objects in different images only have the same semantic information, so we first propose a deep co-salient method based on channel and spatial attention module (CASM), which combines the attention mechanism to enhance the common semantic information. Subsequently, we employ a Co-attention and Refinement Module (CARM) to capture the common attributes of co-salient objects by learning the features consensus representation from a group of images using our group affinity module (GAM) and then we develop a self-correlation module (SCM) to further fine-grained information on the co-salient regions. Specifically, SCM can maintain the feature independence upon semantic categories and further help our model to distinguish pixels with similar but different categories. Moreover, single image saliency maps (SISMs) are predicted to extract intra-saliency cues, and then a correlation fusion module (CFM) is employed to extract inter-saliency cues. The proposed ELR-Net is evaluated on three challenging benchmarks, i.e., CoSal2015, CoSOD3k, and CoCA, demonstrate that our ELR-Net outperforms 9 cutting-edge models and achieves state-of-the-art performance.