Residual Hybrid Attention Network (RHAN) can restore images of arbitrary compression quality through flexibly fusing high-frequency features in the spatial and frequency domains based on the input quality factor. Specifically, to remove the compression artifacts, we propose a hybrid attention block (HAB) to adaptively restore the loss of high-frequency components, which parallelly predicts attention maps along two separate dimensions spatial and frequency spectra. To recover the compressed image flexibly and controllably, we further design a modulation decompression block (MDB), which employs a prior factor to learn a pair of modulation parameters and performs adaptively affine transformation on the obtained high-frequency features, thereby achieving high-quality image restoration at arbitrary compression levels. The quantitative and qualitative experiments on various public data sets show that the RHAN achieves the best performance and optimal visual perceptual quality in the JPEG image restoration with arbitrary compression levels.