14 September 2021 Temporal–spatial feature compensation combines with multi-feature discriminators for video super-resolution perceptual quality improvement
Xuan Zhu, Xin Liu, Lin Wang, Zhenpeng Guo, Jun Wang, Rongzhi Wang, Yifei Sun
Author Affiliations +
Abstract

Generative adversarial network (GAN) for super-resolution (SR) has attracted enormous interest in recent years. It has been widely used to solve the single-image super-resolution (SISR) task and made superior performance. However, GAN is rarely used for video super-resolution (VSR). VSR aims to improve video resolution by exploiting the temporal continuity and spatial similarity of video sequence frames. We design a GAN with multi-feature discriminators and combine it with optical flow estimation compensation to construct an end-to-end VSR framework OFC-MFGAN. Optical flow estimation compensation makes use of temporal continuity and spatial similarity features of adjacent frames to provide rich detailed information for GAN. Multi-feature discriminators based on visual attention mechanism include the pixel discriminator, edge discriminator, gray discriminator, and color discriminator. GAN with multi-feature discriminators makes the data distribution and visually sensitive features (edge, texture, and color) of SR frames similar to high-resolution frames. OFC-MFGAN effectively integrates the time, space, and visually sensitive features of videos. Extensive experiments on public video datasets and surveillance videos show the effectiveness and robustness of the proposed method. Compared with several state-of-the-art VSR methods and SISR methods, the proposed method can not only recover prominent edges, clear textures, and realistic colors but also make a pleasant visual feeling and competitive perceptual index.

© 2021 SPIE and IS&T 1017-9909/2021/$28.00 © 2021 SPIE and IS&T
Xuan Zhu, Xin Liu, Lin Wang, Zhenpeng Guo, Jun Wang, Rongzhi Wang, and Yifei Sun "Temporal–spatial feature compensation combines with multi-feature discriminators for video super-resolution perceptual quality improvement," Journal of Electronic Imaging 30(5), 053005 (14 September 2021). https://doi.org/10.1117/1.JEI.30.5.053005
Received: 27 January 2021; Accepted: 24 August 2021; Published: 14 September 2021
Advertisement
Advertisement
RIGHTS & PERMISSIONS
Get copyright permission  Get copyright permission on Copyright Marketplace
KEYWORDS
Video

Lawrencium

Video surveillance

Optical flow

Visualization

Gallium nitride

Super resolution

Back to Top