Skip to main content
SpringerLink
Log in

Hierarchical prediction-based motion vector refinement for video frame-rate up-conversion

  • Original Research Paper
  • Published:
Journal of Real-Time Image Processing Aims and scope Submit manuscript

Abstract

Motion-compensated frame-rate up-conversion (MC-FRUC) often exploits either bilateral motion estimation (ME) or unidirectional ME with a fixed block size, which constrains the perceptual quality of up-converted video. In this paper, an advanced MC-FRUC approach is proposed by exploiting hierarchical prediction-based motion vector refinement. To reduce block mismatching in texture regions and color areas, an adaptive multi-layered block matching criterion is designed to extract color and edge information, which is integrated with motion information as constraint term. A hierarchical prediction-based motion vector refinement approach is proposed to obtain more accurate and dense motion vector fields (MVFs). To eliminate the outliers of MVFs, a robust dual-weighted motion vector smoothing scheme is adopted by using both spatial correlation and reliability of neighboring blocks. Experimental results show that the proposed approach has low computational complexity and outperforms state-of-the-art works in both objective and subjective qualities of interpolated frames.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8

Similar content being viewed by others

References

  1. Dai, W., Shen, Y., Tang, X., et al.: Sparse representation with spatio-temporal online dictionary learning for promising video coding. IEEE Trans. Image Process. 25(10), 4580–4595 (2016)

    Article  MathSciNet  Google Scholar 

  2. Kaviani, H.R., Shirani, S.: Frame rate upconversion using optical flow and patch-based reconstruction. IEEE Trans. Circuits Syst. Video Technol. 26(9), 1581–1594 (2016)

    Article  Google Scholar 

  3. Li, R., Liu, H., Liu, Z., et al.: Motion-compensated frame interpolation using patch-based sparseland model. Signal Process. Image Commun. 54, 36–48 (2017)

    Article  Google Scholar 

  4. Dikbas, S., Arici, T., Altunbasak, Y.: Fast motion estimation with interpolation-free sub-sample accuracy. IEEE Trans. Circuits Syst. Video Technol. 20(7), 1047–1051 (2010)

    Article  Google Scholar 

  5. Wang, D., Vincent, A., Blanchfield, P., et al.: Motion-compensated frame rate up-conversion—part II: new algorithms for frame interpolation. IEEE Trans. Broadcast. 56(2), 142–149 (2010)

    Article  Google Scholar 

  6. Wang, C., Zhang, L., He, Y., et al.: Frame rate up-conversion using trilateral filtering. IEEE Trans. Circuits Syst. Video Technol. 20(6), 886–893 (2010)

    Article  Google Scholar 

  7. Kang, S.J., Cho, K.R., Kim, Y.H.: Motion compensated frame rate up-conversion using extended bilateral motion estimation. IEEE Trans. Consum. Electron. 53(4), 1759–1767 (2007)

    Article  Google Scholar 

  8. Li, R., Liu, H., Chen, J., et al.: Wavelet pyramid based multi-resolution bilateral motion estimation for frame rate up-conversion. IEICE Trans. Inf. Syst. 99(1), 208–218 (2016)

    Article  Google Scholar 

  9. Choi, B.D., Han, J.W., Kim, C.S., et al.: Motion-compensated frame interpolation using bilateral motion estimation and adaptive overlapped block motion compensation. IEEE Trans. Circuits Syst. Video Technol. 17(4), 407–416 (2007)

    Article  Google Scholar 

  10. Kang, S.J., Yoo, S., Kim, Y.H.: Dual motion estimation for frame rate up-conversion. IEEE Trans. Circuits Syst. Video Technol. 20(12), 1909–1914 (2010)

    Article  Google Scholar 

  11. Tsai, T.H., Lin, H.Y.: High visual quality particle based frame rate up conversion with acceleration assisted motion trajectory calibration. J. Disp. Technol. 8(6), 341–351 (2012)

    Article  Google Scholar 

  12. Zhai, J., Yu, K., Li, J., et al.: A low complexity motion compensated frame interpolation method. In: IEEE International Symposium on Circuits and Systems (ISCAS), pp. 4927–4930. IEEE (2005)

  13. Yoo, D.G., Kang, S.J., Kim, Y.H.: Direction-select motion estimation for motion-compensated frame rate up-conversion. J. Disp. Technol. 9(10), 840–850 (2013)

    Article  Google Scholar 

  14. Kim, U.S., Sunwoo, M.H.: New frame rate up-conversion algorithms with low computational complexity. IEEE Trans. Circuits Syst. Video Technol. 24(3), 384–393 (2014)

    Article  Google Scholar 

  15. Jeong, S.G., Lee, C., Kim, C.S.: Exemplar-based frame rate up-conversion with congruent segmentation. In: IEEE International Conference on Image Processing (ICIP), pp. 845–848 (2012)

  16. Benois-Pineau, J., Nicolas, H.: A new method for region-based depth ordering in a video sequence: application to frame interpolation. J. Vis. Commun. Image Represent. 13(3), 363–385 (2002)

    Article  Google Scholar 

  17. Orchard, M.T., Sullivan, G.J.: Overlapped block motion compensation: an estimation-theoretic approach. IEEE Trans. Image Process. 3(5), 693–699 (1994)

    Article  Google Scholar 

  18. Zhang, Y., Zhao, D., Ma, S., et al.: A motion-aligned auto-regressive model for frame rate up conversion. IEEE Trans. Image Process. 19(5), 1248–1258 (2010)

    Article  MathSciNet  Google Scholar 

  19. Li, R., Gan, Z., Cui, Z., et al.: Multi-channel mixed-pattern based frame rate up-conversion using spatio-temporal motion vector refinement and dual-weighted overlapped block motion compensation. J. Disp. Technol. 10(12), 1010–1023 (2014)

    Article  Google Scholar 

  20. Bayer, B.E.: Color imaging array: U.S. Patent 3,971,065[P]. 1976-7-20

  21. Castagno, R., Haavisto, P., Ramponi, G.: A method for motion adaptive frame rate up-conversion. IEEE Trans. Circuits Syst. Video Technol. 6(5), 436–446 (1996)

    Article  Google Scholar 

  22. Liu, H., Xiong, R., Zhao, D., et al.: Multiple hypotheses Bayesian frame rate up-conversion by adaptive fusion of motion-compensated interpolations. IEEE Trans. Circuits Syst. Video Technol. 22(8), 1188–1198 (2012)

    Article  Google Scholar 

  23. Choi, B.T., Lee, S.H., Park, Y.J., et al.: Frame rate up-conversion using the wavelet transform. In: IEEE International Conference on Consumer Electronics (ICCE), pp. 172–173 (2000)

Download references

Acknowledgements

This work is supported in part by the National Natural Science Foundation of China (61572183, 61379143, 61232016, U1405254).

Author information

Authors and Affiliations

  1. School of Information Science and Engineering, Hunan University, Changsha, 410082, China

    Jiale He, Gaobo Yang & Xiangling Ding

  2. System Engineering Research Institute (SERI), Beijing, 100036, China

    Jingyu Song

  3. School of Computer and Information Technology, Xinyang Normal University, Xinyang, 464000, China

    Ran Li

Authors
  1. Jiale He
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Gaobo Yang
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Jingyu Song
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Xiangling Ding
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Ran Li
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Gaobo Yang.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

He, J., Yang, G., Song, J. et al. Hierarchical prediction-based motion vector refinement for video frame-rate up-conversion. J Real-Time Image Proc 17, 259–273 (2020). https://doi.org/10.1007/s11554-018-0767-y

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11554-018-0767-y

Keywords

Search

Navigation