Skip to main content
SpringerLink
Log in

Video frame rate up conversion under inconsistent camera motion

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

In this paper, we address the problem of video frame rate up-conversion (FRC) in the compressed domain. FRC is often recognized as video temporal interpolation. This problem is very challenging when targeted for video sequences with inconsistent camera and object motion, such as sports videos. A novel compressed domain motion compensation scheme is presented and applied in this paper, aiming at up-sampling frame rates in sports videos. MPEG-2 encoded motion vectors (MVs) are utilized as inputs in the proposed algorithm. The decoded MVs undergo a cumulative spatiotemporal interpolation. An iterative rejection scheme based on the dense motion vector field (MVF) and the generalized affine motion model is exploited to detect global camera motion. Subsequently, the foreground object separation is performed by additionally examining the temporal consistency of the output of iterative rejections. This consistency check process helps coalesce the resulting foreground blocks and weed out the unqualified blocks. Finally, different compensation strategies for the camera and object motions are applied to interpolate the new frames. Illustrative examples are provided to demonstrate the efficacy of the proposed approach. Experimental results are compared with the popular block and non-block based frame interpolation approaches.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. Ascenso J, Brites C, Pereira F (2005) Improving frame interpolation with spatial motion smoothing for pixel domain distributed video coding. In: Proceedings of 2005 European Association for Signal, Speech and Image Processing, Slovak Republic, July 2005

  2. Choi BT, Lee SH, Ko SJ (2000) New frame rate up-conversion using bi-directional motion estimation. IEEE Trans Consum Electron 46(3):603–609

    Article  Google Scholar 

  3. Favalli L, Mecocci A, Moschetti F (2000) Object tracking for retrieval applications in mpeg-2. IEEE Trans Circuits Syst Video Technol 10(3):427–432

    Article  Google Scholar 

  4. ISO/IEC (1996) Generic coding of moving pictures and associated audio information mpeg-2, iso/iec 13 818. Technical report. ISO/IEC, Geneva

  5. Ishwar P, Moulin P (2000) On spatial adaptation of motion-field smoothness in video coding. IEEE Trans Circuits Syst Video Technol 10(6):980–989

    Article  Google Scholar 

  6. Jeon B, Lee G, Lee S, Park R (2003) Coarse-to-fine frame interpolation for frame rate up-conversion using pyramid structure. IEEE Trans Consum Electron 49(3):499–508

    Article  MathSciNet  Google Scholar 

  7. Lee SH, Shin YC, Yang S, Moon HH, Park RH (2002) Adaptive motion-compensated interpolation for frame rate up-conversion. IEEE Trans Consum Electron 48(3):444–450

    Article  Google Scholar 

  8. Li D, Sethi I (1999) Mdc: A software tool for developing mpeg applications. In: Proceeding of 1999 IEEE International Conference on Multimedia Computing and Systems. IEEE, Piscataway, pp 445–450

  9. Liu S, Kim JW, Kuo CC (2001) Mci-embedded motion compensated prediction for quality enhancement of frame interpolation. In: Proceedings of SPIE, multimedia systems and applications III, vol 4209. SPIE, Bellingham, pp 251–261

    Google Scholar 

  10. ISO/IEC (2001) Mpeg-4 video verification model version 18.0, iso/iec jtc1/sc29/wg11. ISO/IEC, Geneva

  11. Musmann H, Pirsch P, Grallert HJ (1985) Advances in picture coding. In: Proceeding of 1985 IEEE, vol 73. IEEE, Piscataway, pp 523–548

    Google Scholar 

  12. Pelagotti A, Haan G (2001) High quality picture rate upconversion for video on tv and pc. In: Proceeding of 2001 IEEE international conference on image processing. IEEE, Piscataway, pp 375–378

    Google Scholar 

  13. Press WH, Flannery BP, Teukolsky SA, Vetterling WT (1993) Numerical recipes in C: The art of scientific computing. Cambridge University Press, Cambridge

    Google Scholar 

  14. Schutten RJ, Hann G (1994) Real-time 2-3 pull-down elimination applying motion estimation/compensation in a programmable device. IEEE Trans Consum Electron 4(3):930–938

    Google Scholar 

  15. Sekiguchi S, Idehara Y, Sugimoto K (2005) A low-cost bideo frame-rate up conversion using compressed-domain information. In: Proceeding of 2005 IEEE international conference on image processing. IEEE, Piscataway, pp 974–977

    Google Scholar 

  16. Shen B (2005) Submacroblock motion compensation for fast down-scale transcoding of compressed video. IEEE Trans Circuits Syst Video Technol 15(10):1291–1302

    Article  Google Scholar 

  17. Shen B, Sethi IK, Vasudev B (1999) Adaptive motion-vector resampling for compressed video downscaling. IEEE Trans Circuits Syst Video Technol 9(6):929–936

    Article  Google Scholar 

  18. Tan YP, Kulkarni SR, Ramadge PJ (1995) A new method for camera motion parameter estimation. In: Proceeding of 1995 IEEE international conference on image processing, vol 1. IEEE, Piscataway, pp 406–409

    Google Scholar 

  19. Tan Y, Saur DD, Kulkarni SR, Ramadge PJ (2000) Rapid estimation of camera motion from compressed video with application to video annotation. IEEE Trans Circuits Syst Video Technol 10(1):133–146

    Article  Google Scholar 

  20. Tekalp AM (1995) Digital video processing. Prentice Hall, Englewood Cliffs

    Google Scholar 

  21. Wang J, Patel N, Grosky W (2004) A fast block-based motion compensation video frame interpolation approach. In: The 38th Asilomar conference on signals, systems and computers, Pacific Grove, November 2004, pp 1740–1743

  22. Wang J, Patel N, Grosky W (2004) Video frame rate up conversion using region based motion compensation. In: Proceeding of 2004 IEEE Region 4 Electro/Infomation Technology Conference. IEEE, Piscataway

    Google Scholar 

  23. Wang Z, Bovik AC, Sheikh HR, Simoncelli EP (2004) Image quality assessment: from error visibility to structural similarity. IEEE Trans Image Process 13(4):600–612

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of Computer Science, Wayne State University, Detroit, MI, 48202, USA

    Jinsong Wang & Farshad Fotouhi

  2. Department of Computer and Information Systems, University of Michigan, Dearborn, Dearborn, MI, 48128, USA

    Nilesh Patel & William Grosky

Authors
  1. Jinsong Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Nilesh Patel
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. William Grosky
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Farshad Fotouhi
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Jinsong Wang.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Wang, J., Patel, N., Grosky, W. et al. Video frame rate up conversion under inconsistent camera motion. Multimed Tools Appl 39, 329–351 (2008). https://doi.org/10.1007/s11042-007-0162-3

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-007-0162-3

Keywords

Search

Navigation