skip to main content
research-article

Motion-aware temporal coherence for video resizing

Published: 01 December 2009 Publication History

Abstract

Temporal coherence is crucial in content-aware video retargeting. To date, this problem has been addressed by constraining temporally adjacent pixels to be transformed coherently. However, due to the motion-oblivious nature of this simple constraint, the retargeted videos often exhibit flickering or waving artifacts, especially when significant camera or object motions are involved. Since the feature correspondence across frames varies spatially with both camera and object motion, motion-aware treatment of features is required for video resizing. This motivated us to align consecutive frames by estimating interframe camera motion and to constrain relative positions in the aligned frames. To preserve object motion, we detect distinct moving areas of objects across multiple frames and constrain each of them to be resized consistently. We build a complete video resizing framework by incorporating our motion-aware constraints with an adaptation of the scale-and-stretch optimization recently proposed by Wang and colleagues. Our streaming implementation of the framework allows efficient resizing of long video sequences with low memory cost. Experiments demonstrate that our method produces spatiotemporally coherent retargeting results even for challenging examples with complex camera and object motion, which are difficult to handle with previous techniques.

Supplementary Material

Supplemental material. (127-wang.zip)

References

[1]
Avidan, S., and Shamir, A. 2007. Seam carving for content-aware image resizing. ACM Trans. Graph. 26, 3, 10.
[2]
Chen, L. Q., Xie, X., Fan, X., Ma, W. Y., Zhang, H. J., and Zhou, H. Q. 2003. A visual attention model for adapting images on small displays. ACM Multimedia Systems Journal 9, 4, 353--364.
[3]
Chen, B.-Y., Lee, K.-Y., Huang, W.-T., and Lin, J.-S. 2008. Capturing intention-based full-frame video stabilization. Computer Graphics Forum 27, 7, 1805--1814.
[4]
Cho, T. S., Butman, M., Avidan, S., and Freeman, W. T. 2008. The patch transform and its applications to image editing. In CVPR '08.
[5]
Deselaers, T., Dreuw, P., and Ney, H. 2008. Pan, zoom, scan - time-coherent, trained automatic video cropping. In CVPR '08.
[6]
Fischler, M. A., and Bolles, R. C. 1981. Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography. Commun. ACM 24, 6, 381--395.
[7]
Gal, R., Sorkine, O., and Cohen-Or, D. 2006. Feature-aware texturing. In EGSR '06, 297--303.
[8]
Gleicher, M. L., and Liu, F. 2008. Re-cinematography: Improving the camerawork of casual video. ACM Trans. Multimedia Comput. Commun. Appl. 5, 1, 1--28.
[9]
Itti, L., Koch, C., and Niebur, E. 1998. A model of saliency-based visual attention for rapid scene analysis. IEEE Trans. Pattern Anal. Mach. Intell. 20, 11, 1254--1259.
[10]
Kang, H.-W., Matsushita, Y., Tang, X., and Chen, X.-Q. 2006. Space-time video montage. In CVPR '06.
[11]
Kraevoy, V., Sheffer, A., Cohen-Or, D., and Shamir, A. 2008. Non-homogeneous resizing of complex models. ACM Trans. Graph. 27, 5, 111.
[12]
Krähenbühl, P., Lang, M., Hornung, A., and Gross, M. 2009. A system for retargeting of streaming video. ACM Trans. Graph. 28, 5.
[13]
Liu, F., and Gleicher, M. 2006. Video retargeting: automating pan and scan. In Multimedia '06, 241--250.
[14]
Liu, H., Xie, X., Ma, W.-Y., and Zhang, H.-J. 2003. Automatic browsing of large pictures on mobile devices. In Proceedings of ACM International Conference on Multimedia, 148--155.
[15]
Lowe, D. G. 2004. Distinctive image features from scale-invariant keypoints. Int. J. Comput. Vision 60, 2, 91--110.
[16]
Rasheed, Z., and Shah, M. 2003. Scene detection in hollywood movies and tv shows. In CVPR '03, vol. 2, II-343--8.
[17]
Rubinstein, M., Shamir, A., and Avidan, S. 2008. Improved seam carving for video retargeting. ACM Trans. Graph. 27, 3, 16.
[18]
Rubinstein, M., Shamir, A., and Avidan, S. 2009. Multioperator media retargeting. ACM Trans. Graph. 28, 3, 23.
[19]
Santella, A., Agrawala, M., DeCarlo, D., Salesin, D., and Cohen, M. 2006. Gaze-based interaction for semiautomatic photo cropping. In Proceedings of CHI, 771--780.
[20]
Setlur, V., Takagi, S., Raskar, R., Gleicher, M., and Gooch, B. 2005. Automatic image retargeting. In MUM '05, 59--68.
[21]
Simakov, D., Caspi, Y., Shechtman, E., and Irani, M. 2008. Summarizing visual data using bidirectional similarity. In CVPR '08.
[22]
Sorkine, O., Lipman, Y., Cohen-Or, D., Alexa, M., Rössl, C., and Seidel, H.-P. 2004. Laplacian surface editing. In SGP '04, 179--188.
[23]
Suh, B., Ling, H., Bederson, B. B., and Jacobs, D. W. 2003. Automatic thumbnail cropping and its effectiveness. In Proceedings of UIST, 95--104.
[24]
Szeliski, R. 2006. Image alignment and stitching: a tutorial. Foundations and Trends in Computer Graphics and Vision 2, 1, 1--104.
[25]
Tao, C., Jia, J., and Sun, H. 2007. Active window oriented dynamic video retargeting. In Workshop on Dynamical Vision, ICCV '07.
[26]
Wang, Y.-S., Lee, T.-Y., and Tai, C.-L. 2008. Focus+context visualization with distortion minimization. IEEE Trans. Visualization and Computer Graphics 14, 6.
[27]
Wang, Y.-S., Tai, C.-L., Sorkine, O., and Lee, T.-Y. 2008. Optimized scale-and-stretch for image resizing. ACM Trans. Graph. 27, 5, 118.
[28]
Wolf, L., Guttmann, M., and Cohen-Or, D. 2007. Non-homogeneous content-driven video-retargeting. In ICCV '07.
[29]
Zhang, Y.-F., Hu, S.-M., and Martin, R. R. 2008. Shrinkability maps for content-aware video resizing. In PG '08.

Cited By

View all

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Transactions on Graphics
ACM Transactions on Graphics  Volume 28, Issue 5
December 2009
646 pages
ISSN:0730-0301
EISSN:1557-7368
DOI:10.1145/1618452
Issue’s Table of Contents

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 01 December 2009
Published in TOG Volume 28, Issue 5

Permissions

Request permissions for this article.

Check for updates

Author Tags

  1. optimization
  2. spatial and temporal coherence
  3. video retargeting

Qualifiers

  • Research-article

Funding Sources

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)4
  • Downloads (Last 6 weeks)1
Reflects downloads up to 13 Feb 2025

Other Metrics

Citations

Cited By

View all
  • (2023)Synthesizing Physical Character-Scene InteractionsACM SIGGRAPH 2023 Conference Proceedings10.1145/3588432.3591525(1-9)Online publication date: 23-Jul-2023
  • (2022)Unbiased Caustics Rendering Guided by Representative Specular PathsSIGGRAPH Asia 2022 Conference Papers10.1145/3550469.3555381(1-8)Online publication date: 29-Nov-2022
  • (2022)Correcting Face Distortion in Wide-Angle VideosIEEE Transactions on Image Processing10.1109/TIP.2021.313104731(366-378)Online publication date: 2022
  • (2021)High Quality Disparity Remapping with Two-Stage Warping2021 IEEE/CVF International Conference on Computer Vision (ICCV)10.1109/ICCV48922.2021.00227(2249-2258)Online publication date: Oct-2021
  • (2020)Temporal Incoherence-Free Video Retargeting Using Foreground Aware ExtrapolationIEEE Transactions on Image Processing10.1109/TIP.2020.297717129(4848-4861)Online publication date: 2020
  • (2020)Extrapolation-Based Video Retargeting With Backward Warping Using an Image-to-Warping Vector Generation NetworkIEEE Signal Processing Letters10.1109/LSP.2020.297720627(446-450)Online publication date: 2020
  • (2019)Video RetargetingProceedings of the 27th ACM International Conference on Multimedia10.1145/3343031.3350895(882-889)Online publication date: 15-Oct-2019
  • (2019)Joint Stabilization and Direction of 360° VideosACM Transactions on Graphics10.1145/321188938:2(1-13)Online publication date: 18-Mar-2019
  • (2019)SmartGrid: Video Retargeting With Spatiotemporal Grid OptimizationIEEE Access10.1109/ACCESS.2019.29388657(127564-127579)Online publication date: 2019
  • (2019)Consistent video projection on curved displaysSignal Processing: Image Communication10.1016/j.image.2019.03.006Online publication date: Mar-2019
  • Show More Cited By

View Options

Login options

Full Access

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Figures

Tables

Media

Share

Share

Share this Publication link

Share on social media