Multiframe interpolation for video using phase features

Lunan Zhou; Yaowu Chen; Xiang Tian; Rongxin Jiang

doi:10.1117/1.JEI.28.4.043002

3 July 2019 Multiframe interpolation for video using phase features

Lunan Zhou, Yaowu Chen, Xiang Tian, Rongxin Jiang

Author Affiliations +

Journal of Electronic Imaging, Vol. 28, Issue 4, 043002 (July 2019). https://doi.org/10.1117/1.JEI.28.4.043002

Abstract

Traditional frame interpolation algorithms typically find dense correspondences to synthesize an in-between frame. Finding correspondences is often sensitive to occlusion, disocclusion, and changes in color or luminance. We present a phase-feature-aided multiframe interpolation network that aims to estimate multiple in-between frames in one pass and handle challenging scenarios such as extreme light changes and occlusion. We first model the relation between multiple in-between frames together to enhance the temporal consistency. Two candidate optical flow fields are produced for a given in-between frame, one predicted from our network and the other estimated from those of neighboring frames using a flow fusion map. We also employ an image fusion map to combat occlusion problems in the warping processes, producing two candidate interpolated images that are fed to a shallow network with a residual structure to obtain the final interpolated image. To handle challenging scenarios, we apply a set of Gabor filters to extract phase variations in the feature domain with a multiscale phase subnetwork. Our entire neural network is end-to-end trainable. Our experiments show that this method outperforms the state-of-the-art approaches and achieves marked visual improvement in challenging scenarios.

Citation Download Citation

Lunan Zhou, Yaowu Chen, Xiang Tian, and Rongxin Jiang "Multiframe interpolation for video using phase features," Journal of Electronic Imaging 28(4), 043002 (3 July 2019). https://doi.org/10.1117/1.JEI.28.4.043002

Received: 26 November 2018; Accepted: 10 June 2019; Published: 3 July 2019

ACCESS THE FULL ARTICLE

INSTITUTIONAL
Select your institution to access the SPIE Digital Library.

SELECT YOUR INSTITUTION

PERSONAL
Sign in with your SPIE account to access your personal subscriptions or to use specific features such as save to my library, sign up for alerts, save searches, etc.

PERSONAL SIGN IN

No SPIE Account? Create one

PURCHASE THIS CONTENT

SUBSCRIBE TO DIGITAL LIBRARY

50 downloads per 1-year subscription

Members: $195

Non-members: $335 ADD TO CART

25 downloads per 1 - year subscription

Members: $145

Non-members: $250 ADD TO CART

PURCHASE SINGLE ARTICLE

Includes PDF, HTML & Video, when available

Members: $24.00

Non-members: $28.00 ADD TO CART

JOURNAL ARTICLE
14 PAGES

DOWNLOAD PAPER SAVE TO MY LIBRARY

GET CITATION

RIGHTS & PERMISSIONS

Get copyright permission Get copyright permission on Copyright Marketplace

KEYWORDS

Optical flow

Video

Image filtering

Image fusion

Image processing

Motion estimation

Visualization

Show All Keywords

Keywords/Phrases

Search In:

Publication Years