research-article

Spatio-temporal Saliency-based Motion Vector Refinement for Frame Rate Up-conversion

Authors:

Xiangling DingAuthors Info & Claims

ACM Transactions on Multimedia Computing, Communications, and Applications (TOMM), Volume 16, Issue 2

Article No.: 55, Pages 1 - 18

https://doi.org/10.1145/3382506

Published: 22 May 2020 Publication History

Abstract

A spatio-temporal saliency-based frame rate up-conversion (FRUC) approach is proposed, which achieves better quality of interpolated frames and invalidates existing texture variation-based FRUC detectors. A spatio-temporal saliency model is designed to select salient frames. After obtaining initial motion vector field by texture- and color-based bilateral motion estimation, two motion vector refining (MVR) schemes are adopted for high and low saliency frames to hierarchically refine the motion vectors, respectively. To produce high-quality interpolated frames, image enhancement are performed for salient frames after frame interpolation. Due to distinct MVR schemes, there are different degrees of texture information in interpolated frames. Some edge and texture information is supplemented into salient frames as post-processing, which can invalidate existing texture variation-based FRUC detectors. Experimental results show that the proposed approach outperforms state-of-the-art works in both objective and subjective qualities of interpolated frames, and achieves the purpose of FRUC anti-forensics.

References

[1]

Khosro Bahrami and Alex C. Kot. 2014. A fast approach for no-reference image sharpness assessment based on maximum local variation. IEEE Signal Processing Letters 21, 6 (2014), 751--755. https://ieeexplore.ieee.xilesou.top/abstract/document/6780989/

[2]

Wenbo Bao, Xiaoyun Zhang, Li Chen, Lianghui Ding, and Zhiyong Gao. 2018. High-order model and dynamic filtering for frame rate up-conversion. IEEE Transactions on Image Processing 27, 8 (2018), 3813--3826. https://ieeexplore.ieee.xilesou.top/abstract/document/8334253/

[3]

Bryce E. Bayer. 1976. Color imaging array. https://patents.glgoo.top/patent/US3971065/en US Patent 3,971,065.

[4]

Jenny Benois-Pineau and Henri Nicolas. 2002. A new method for region-based depth ordering in a video sequence: application to frame interpolation. Journal of Visual Communication and Image Representation 13, 3 (2002), 363--385. https://sciencedirect.xilesou.top/science/article/pii/S1047320301904900

[5]

Byeong-Doo Choi, Jong-Woo Han, Chang-Su Kim, and Sung-Jea Ko. 2007. Motion-compensated frame interpolation using bilateral motion estimation and adaptive overlapped block motion compensation. IEEE Transactions on Circuits and Systems for Video Technology 17, 4 (2007), 407--416. https://ieeexplore.ieee.xilesou.top/abstract/document/4162523/

Digital Library

[6]

Dooseop Choi, Wonseok Song, Hyuk Choi, and Taejeong Kim. 2016. MAP-based motion refinement algorithm for block-based motion-compensated frame interpolation. IEEE Transactions on Circuits and Systems for Video Technology 26, 10 (2016), 1789--1804. https://ieeexplore.ieee.xilesou.top/abstract/document/7225154

Digital Library

[7]

Xiangling Ding, Yue Li, Ming Xia, Jiale He, and Gaobo Yang. 2019. Detection of motion compensated frame interpolation via motion-aligned temporal difference. Multimedia Tools and Applications 78, 6 (2019), 7453--7477. https://link.springer.xilesou.top/article/10.1007/s11042-018-6504-5

Digital Library

[8]

Xiangling Ding, Gaobo Yang, Ran Li, Lebing Zhang, Yue Li, and Xingming Sun. 2018. Identification of motion-compensated frame rate up-conversion based on residual signals. IEEE Transactions on Circuits and Systems for Video Technology 28, 7 (2018), 1497--1512. https://ieeexplore.ieee.xilesou.top/abstract/document/7869361/

[9]

Xiangling Ding, Ningbo Zhu, Leida Li, Yue Li, and Gaobo Yang. 2018. Robust localization of interpolated frames by motion-compensated frame-interpolation based on artifact indicated map and Tchebichef moments. IEEE Transactions on Circuits and Systems for Video Technology 29, 7 (2018), 1893--1906. https://ieeexplore.ieee.xilesou.top/abstract/document/8403308/

[10]

Dashan Gao, Vijay Mahadevan, and Nuno Vasconcelos. 2008. On the plausibility of the discriminant center-surround hypothesis for visual saliency. Journal of Vision 8, 7 (2008), 13--30. https://jov.arvojournals.org/article.aspx?articleid=2193585

[11]

Yong Guo, Li Chen, Zhiyong Gao, and Xiaoyun Zhang. 2016. Frame rate up-conversion using linear quadratic motion estimation and trilateral filtering motion smoothing. Journal of Display Technology 12, 1 (2016), 89--98. https://www.osapublishing.org/abstract.cfm?uri=jdt-12-1-89

[12]

Jiale He, Gaobo Yang, Jingyu Song, Xiangling Ding, and Ran Li. 2018. Hierarchical prediction-based motion vector refinement for video frame-rate up-conversion. Journal of Real-Time Image Processing (2018), 1--15. https://link.springer.xilesou.top/article/10.1007/s11554-018-0767-y

[13]

Seong-Gyun Jeong, Chul Lee, and Chang-Su Kim. 2012. Exemplar-based frame rate up-conversion with congruent segmentation. In 2012 19th IEEE International Conference on Image Processing. IEEE, 845--848. https://ieeexplore.ieee.xilesou.top/abstract/document/6466992/

[14]

Seong-Gyun Jeong, Chul Lee, and Chang-Su Kim. 2013. Motion-compensated frame interpolation based on multihypothesis motion estimation and texture optimization. IEEE Transactions on Image Processing 22, 11 (2013), 4497--4509. https://ieeexplore.ieee.xilesou.top/abstract/document/6567908/

Digital Library

[15]

Suk-Ju Kang, Kyoung-Rok Cho, and Young Hwan Kim. 2007. Motion compensated frame rate up-conversion using extended bilateral motion estimation. IEEE Transactions on Consumer Electronics 53, 4 (2007), 1759--1767. https://ieeexplore.ieee.xilesou.top/abstract/document/4429281/

Digital Library

[16]

Suk-Ju Kang, Sungjoo Yoo, and Young Hwan Kim. 2010. Dual motion estimation for frame rate up-conversion. IEEE Transactions on Circuits and Systems for Video Technology 20, 12 (2010), 1909--1914. https://ieeexplore.ieee.xilesou.top/abstract/document/5604667/

Digital Library

[17]

Xiangui Kang, Jingxian Liu, Hongmei Liu, and Z. Jane Wang. 2016. Forensics and counter anti-forensics of video inter-frame forgery. Multimedia Tools and Applications 75, 21 (2016), 13833--13853. https://link.springer.xilesou.top/article/10.1007/s11042-015-2762-7

Digital Library

[18]

Un Seob Kim and Myung Hoon Sunwoo. 2014. New frame rate up-conversion algorithms with low computational complexity. IEEE Transactions on Circuits and Systems for Video Technology 24, 3 (2014), 384--393. https://ieeexplore.ieee.xilesou.top/abstract/document/6578124/

Digital Library

[19]

Won Hee Lee, Kyuha Choi, and Jong Beom Ra. 2014. Frame rate up conversion based on variational image fusion. IEEE Transactions on Image Processing 23, 1 (2014), 399--412. https://ieeexplore.ieee.xilesou.top/abstract/document/6651823/

Digital Library

[20]

Ran Li, Hongbing Liu, Jie Chen, and Zongliang Gan. 2016. Wavelet pyramid based multi-resolution bilateral motion estimation for frame rate up-conversion. IEICE Transactions on Information and Systems 99, 1 (2016), 208--218. https://www.jstage.jst.go.jp/article/transinf/E99.D/1/E99.D_2015EDP7027/_article/-char/ja/

[21]

Ran Li, Hongbing Liu, Zhenghui Liu, Yanling Li, and Zhangjie Fu. 2017. Motion-compensated frame interpolation using patch-based sparseland model. Signal Processing: Image Communication 54 (2017), 36--48. https://sciencedirect.xilesou.top/science/article/pii/S0923596517300267

Digital Library

[22]

Ran Li, Yongfeng Lv, and Zhenghui Liu. 2018. Multi-scheme frame rate up-conversion using space-time saliency. IEEE Access 6, 99 (2018), 1905--1915. https://ieeexplore.ieee.xilesou.top/abstract/document/8169029/

[23]

Yanli Li, Wendan Ma, and Yue Han. 2019. A spatial prediction-based motion-compensated frame rate up-conversion. Future Internet 11, 2 (2019), 26--35. https://www.mdpi.xilesou.top/1999-5903/11/2/26

[24]

Yue Li, Gaobo Yang, Yapei Zhu, Xiangling Ding, and Xingming Sun. 2017. Adaptive inter CU depth decision for HEVC using optimal selection model and encoding parameters. IEEE Transactions on Broadcasting 63, 3 (2017), 535--546. https://ieeexplore.ieee.xilesou.top/abstract/document/7940104/

[25]

Yue Li, Gaobo Yang, Yapei Zhu, Xiangling Ding, and Xingming Sun. 2017. Unimodal stopping model-based early SKIP mode decision for high-efficiency video coding. IEEE Transactions on Multimedia 19, 7 (2017), 1431--1441. https://ieeexplore.ieee.xilesou.top/abstract/document/7857045/

Digital Library

[26]

Hongbin Liu, Ruiqin Xiong, Debin Zhao, Siwei Ma, and Wen Gao. 2012. Multiple hypotheses Bayesian frame rate up-conversion by adaptive fusion of motion-compensated interpolations. IEEE Transactions on Circuits and Systems for Video Technology 22, 8 (2012), 1188--1198. https://ieeexplore.ieee.xilesou.top/abstract/document/6193418/

Digital Library

[27]

Jacobson Natan, Yenlin Lee, and Mahadevan Vijay. 2010. Motion vector refinement for FRUC using saliency and segmentation. In Proceedings of the 11th IEEE International Conference on Multimedia and Expo. IEEE, 778--783. https://ieeexplore.ieee.xilesou.top/abstract/document/5582574/

[28]

Jacobson Natan and Nguyen Truong Q. 2012. Scale-aware saliency for application to frame rate upconversion. IEEE Transactions on Image Processing 21, 4 (2012), 2198--2206. https://ieeexplore.ieee.xilesou.top/abstract/document/6099617/

Digital Library

[29]

Michael T. Orchard and Gary J. Sullivan. 1994. Overlapped block motion compensation: An estimation-theoretic approach. IEEE Transactions on Image Processing 3, 5 (1994), 693--699. https://ieeexplore.ieee.xilesou.top/abstract/document/334974/

Digital Library

[30]

Aixi Qu, Ju Liu, Wenbo Wan, and Yifan Xiao. 2016. A frame rate up-conversion method with quadruple motion vector post-processing. In 2016 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 1686--1690. https://ieeexplore.ieee.xilesou.top/abstract/document/7471964/

Digital Library

[31]

Milan Sonka, Vaclav Hlavac, and Roger Boyle. 2014. Image Processing, Analysis, and Machine Vision. Cengage Learning.

[32]

Matthew C. Stamm, W. Sabrina Lin, and K. J. Ray Liu. 2012. Temporal forensics and anti-forensics for motion compensated video. IEEE Transactions on Information Forensics and Security 7, 4 (2012), 1315--1329. https://ieeexplore.ieee.xilesou.top/abstract/document/6222325/

Digital Library

[33]

Tsung-Han Tsai and Hsueh-Yi Lin. 2012. High visual quality particle based frame rate up conversion with acceleration assisted motion trajectory calibration. Journal of Display Technology 8, 6 (2012), 341--351. https://www.osapublishing.org/jdt/abstract.cfm?uri=jdt-8-6-341

[34]

Tsung-Han Tsai, An-Ting Shi, and Ko-Ting Huang. 2016. Accurate frame rate up-conversion for advanced visual quality. IEEE Transactions on Broadcasting 62, 2 (2016), 426--435. https://ieeexplore.ieee.xilesou.top/abstract/document/7464837/

[35]

Ci Wang, Lei Zhang, Yuwen He, and Yap-Peng Tan. 2010. Frame rate up-conversion using trilateral filtering. IEEE Transactions on Circuits and Systems for Video Technology 20, 6 (2010), 886--893. https://ieeexplore.ieee.xilesou.top/abstract/document/5433045/

Digital Library

[36]

Min Xia, Gaobo Yang, Leida Li, Ran Li, and Xingming Sun. 2017. Detecting video frame rate up-conversion based on frame-level analysis of average texture variation. Multimedia Tools and Applications 76, 6 (2017), 8399--8421. https://link.springer.xilesou.top/article/10.1007/s11042-016-3468-1

Digital Library

[37]

Yuxuan Yao, Gaobo Yang, Xingming Sun, and Leida Li. 2016. Detecting video frame-rate up-conversion based on periodic properties of edge-intensity. Journal of Information Security and Applications 26 (2016), 39--50. https://sciencedirect.xilesou.top/science/article/pii/S2214212615000691

Digital Library

[38]

Dong-Gon Yoo, Suk-Ju Kang, and Young Hwan Kim. 2013. Direction-select motion estimation for motion-compensated frame rate up-conversion. Journal of Display Technology 9, 10 (2013), 840--850. https://www.osapublishing.org/jdt/abstract.cfm?uri=jdt-9-10-840

[39]

Sung-Jun Yoon, Hyun-Ho Kim, and Munchurl Kim. 2018. Hierarchical extended bilateral motion estimation-based frame rate upconversion using learning-based linear mapping. IEEE Transactions on Image Processing 27, 12 (2018), 5918--5932. https://ieeexplore.ieee.xilesou.top/abstract/document/8423719/

Digital Library

[40]

Yun Zhai and Mubarak Shah. 2006. Visual attention detection in video sequences using spatiotemporal cues. In Proceedings of the 14th ACM International Conference on Multimedia. ACM, 815--824. https://dl.acm.org/doi/abs/10.1145/1180639.1180824

Digital Library

[41]

Hanling Zhang, Chenxing Xia, and Xiuju Gao. 2017. Robust saliency detection via corner information and an energy function. IET Computer Vision 11, 6 (2017), 379--388. https://digital-library.theiet.org/content/journals/10.1049/iet-cvi.2016.0492

[42]

Lebing Zhang, Fei Peng, Le Qin, and Min Long. 2018. Face spoofing detection based on color texture Markov feature and support vector machine recursive feature elimination. Journal of Visual Communication and Image Representation 51 (2018), 56--59. https://sciencedirect.xilesou.top/science/article/pii/S1047320318300014

[43]

Yongbing Zhang, Debin Zhao, Siwei Ma, Ronggang Wang, and Wen Gao. 2010. A motion-aligned auto-regressive model for frame rate up conversion. IEEE Transactions on Image Processing 19, 5 (2010), 1248--1258. https://ieeexplore.ieee.xilesou.top/abstract/document/5357389/

Digital Library

Cited By

Zhang DLou RChen JLiao XYang GDing X(2024)Dual Motion Attention and Enhanced Knowledge Distillation for Video Frame Interpolation2024 Asia Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA ASC)10.1109/APSIPAASC63619.2025.10848798(1-6)Online publication date: 3-Dec-2024
https://doi.org/10.1109/APSIPAASC63619.2025.10848798
Qu ZLiu WCui LYang X(2024)Video frame interpolation via spatial multi‐scale modellingIET Computer Vision10.1049/cvi2.12281Online publication date: 3-Apr-2024
https://doi.org/10.1049/cvi2.12281
Wang YWang YCui TFang Z(2024)Occupancy map-based low complexity motion prediction for video-based point cloud compressionJournal of Visual Communication and Image Representation10.1016/j.jvcir.2024.104110100:COnline publication date: 17-Jul-2024
https://dl.acm.org/doi/10.1016/j.jvcir.2024.104110
Show More Cited By

Index Terms

Spatio-temporal Saliency-based Motion Vector Refinement for Frame Rate Up-conversion
1. Computer systems organization
  1. Dependable and fault-tolerant systems and networks
    1. Redundancy
  2. Embedded and cyber-physical systems
    1. Embedded systems
    2. Robotics
2. Networks
  1. Network properties
    1. Network reliability

Recommendations

Adaptive frame rate up-conversion based on motion classification

In this paper, a new technique on video frame rate up-conversion (FRUC) is presented by combining the adaptive motion classification (AMC) for image sequences and the mixed motion estimation (ME). In the proposed FRUC scheme, the AMC classifies ...
Multiframe-based bilateral motion estimation with emphasis on stationary caption processing for frame rate up-conversion

In this paper, we present a new motion compensated frame rate up-conversion algorithm that uses multiframes to enhance the accuracy of motion estimation. We also develop adaptive motion vector smoothing to correct outliers in a motion vector field. In ...
Region-based motion-compensated frame rate up-conversion by homography parameter interpolation
ICIP'09: Proceedings of the 16th IEEE international conference on Image processing

A new region-based frame interpolation algorithm is proposed based on the segmented motion layers with planar perspective motion models. It is shown that performing the motion model interpolation in the homography parameter space is equivalent to ...

Comments

Information & Contributors

Information

Published In

cover image ACM Transactions on Multimedia Computing, Communications, and Applications

ACM Transactions on Multimedia Computing, Communications, and Applications Volume 16, Issue 2

May 2020

390 pages

ISSN:1551-6857

EISSN:1551-6865

DOI:10.1145/3401894

Editor:
Alberto Del Bimbo
University of Firenze, Italy

Issue’s Table of Contents

Copyright © 2020 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 22 May 2020

Online AM: 07 May 2020

Accepted: 01 February 2020

Revised: 01 September 2019

Received: 01 April 2019

Published in TOMM Volume 16, Issue 2

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed

Funding Sources

National Key R8D Program of China
National Natural Science Foundation of China

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

17
Total Citations
View Citations
231
Total Downloads

Downloads (Last 12 months)10
Downloads (Last 6 weeks)0

Reflects downloads up to 05 Mar 2025

Other Metrics

View Author Metrics

Citations

Cited By

Zhang DLou RChen JLiao XYang GDing X(2024)Dual Motion Attention and Enhanced Knowledge Distillation for Video Frame Interpolation2024 Asia Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA ASC)10.1109/APSIPAASC63619.2025.10848798(1-6)Online publication date: 3-Dec-2024
https://doi.org/10.1109/APSIPAASC63619.2025.10848798
Qu ZLiu WCui LYang X(2024)Video frame interpolation via spatial multi‐scale modellingIET Computer Vision10.1049/cvi2.12281Online publication date: 3-Apr-2024
https://doi.org/10.1049/cvi2.12281
Wang YWang YCui TFang Z(2024)Occupancy map-based low complexity motion prediction for video-based point cloud compressionJournal of Visual Communication and Image Representation10.1016/j.jvcir.2024.104110100:COnline publication date: 17-Jul-2024
https://dl.acm.org/doi/10.1016/j.jvcir.2024.104110
Tang ZXu CYan S(2024)An adaptive interpolation and 3D reconstruction algorithm for underwater imagesMachine Vision and Applications10.1007/s00138-024-01518-235:2Online publication date: 7-Mar-2024
https://dl.acm.org/doi/10.1007/s00138-024-01518-2
Liu WYang XFeng ZXu TGuo Q(2023)Video Frame Interpolation Based on Lightweight Convolutional Unit and Three-scale EncoderProceedings of the 15th International Conference on Digital Image Processing10.1145/3604078.3604169(1-7)Online publication date: 19-May-2023
https://dl.acm.org/doi/10.1145/3604078.3604169
Wang JWang HZhao YYuan RXu F(2023)Dense 3D Reconstruction of Non-cooperative Target Based on Pose MeasurementDigital Multimedia Communications10.1007/978-981-99-0856-1_3(30-43)Online publication date: 10-Mar-2023
https://doi.org/10.1007/978-981-99-0856-1_3
Nousias SArvanitis GLalos AMoustakas K(2022)Deep Saliency Mapping for 3D Meshes and ApplicationsACM Transactions on Multimedia Computing, Communications, and Applications10.1145/355007319:2(1-22)Online publication date: 27-Jul-2022
https://dl.acm.org/doi/10.1145/3550073
Zhang DHuang PDing XLi FZhu WSong YYang G(2022)LBEC: Local Lightweight Bidirectional Encoding and Channel Attention Cascade for Video Frame InterpolationACM Transactions on Multimedia Computing, Communications, and Applications10.1145/354766019:2(1-19)Online publication date: 15-Jul-2022
https://dl.acm.org/doi/10.1145/3547660
Gu QDing XZhang DYang C(2022)Forgery Detection Scheme of Deep Video Frame-rate Up-conversion Based on Dual-stream Multi-scale Spatial-temporal Representation2022 IEEE International Conference on Trust, Security and Privacy in Computing and Communications (TrustCom)10.1109/TrustCom56396.2022.00104(733-738)Online publication date: Dec-2022
https://doi.org/10.1109/TrustCom56396.2022.00104
Ding XHuang PZhang DZhao X(2022)Video Frame Interpolation via Local Lightweight Bidirectional Encoding with Channel Attention CascadeICASSP 2022 - 2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)10.1109/ICASSP43922.2022.9747182(1915-1919)Online publication date: 23-May-2022
https://doi.org/10.1109/ICASSP43922.2022.9747182
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

HTML Format

View this article in HTML Format.

Figures

Tables

Media

View Issue’s Table of Contents