The real-world image degradation in the super-resolution task is recently considered as a combination of Gaussian blur, down-sampling, and additional white Gaussian noise. To han-dle this degradation, previous methods estimate the Gaussian blur kernel or model the degradation based on a randomly selected image patch. However, these methods cannot han-dle degradations with high-level noise well as they ignore the spatial variability or even the existence of noise. Moreover, using image denoising networks to preprocess low-resolution images also fails due to the loss of important high-frequency information. In this paper, we propose a framework called EASE to flexibly handle real-world degradations. Specifi-cally, we develop a lightweight module to erase noise and blur simultaneously by learning from an image denoising and an image restoration network, which adapts to existing net-works that focus on handling bicubic down-sampling. Exten-sive experiments prove the superiority of our method, espe-cially when handling degradations with high-level noise.