Due to hardware restrictions, Light Field (LF) images are often captured with heavy noise, which seriously obstructs the subsequent LF applications. In this paper, we propose a novel symmetrical refocusing strategy to construct the focal stack for every view in LF images and design a simple learning-based framework for LF denoising. Specifically, we first select views that are symmetrically arranged around a target view in LF images. Then we shift and average the selected views to calculate the focal stack, in which all refocused images are aligned with the target view and the noises are effectively suppressed. Then, a Fusion Network is designed to fuse the sharp regions in the focal stack to obtain the denoised target view with sharp details. We further exploit more angular and spatial detail information in LF images and combine the fusion outputs to obtain the final denoised LF images. We evaluate our method in various noise levels and kinds of noisy LF images with different disparity ranges. The experiments show that our method achieves the highest quality in both qualitative and quantitative evaluation than state-of-the-art methods. The proposed symmetrical refocusing strategy is also verified to highly improve the denoising performances.