Philipp Krähenbühl
Abstract
We present a novel, integrated system for content-aware video retargeting. A simple and interactive framework combines key frame based constraint editing with numerous automatic algorithms for video analysis. This combination gives content producers high level control of the retargeting process. The central component of our framework is a non-uniform, pixel-accurate warp to the target resolution which considers automatic as well as interactively defined features. Automatic features comprise video saliency, edge preservation at the pixel resolution, and scene cut detection to enforce bilateral temporal coherence. Additional high level constraints can be added by the producer to guarantee a consistent scene composition across arbitrary output formats. For high quality video display we adopted a 2D version of EWA splatting eliminating aliasing artifacts known from previous work. Our method seamlessly integrates into postproduction and computes the reformatting in real-time. This allows us to retarget annotated video streams at a high quality to arbitary aspect ratios while retaining the intended cinematographic scene composition. For evaluation we conducted a user study which revealed a strong viewer preference for our method.
Supplemental Material
Available for Download
- Avidan, S., and Shamir, A. 2007. Seam carving for content-aware image resizing. ACM Trans. Graph. 26, 3, 10. Google Scholar
Digital Library
- Botsch, M., Hornung, A., Zwicker, M., and Kobbelt, L. 2005. High-quality surface splatting on today's GPUs. In Symposium on Point-Based Graphics, 17--24. Google ScholarDigital Library
- Briggs, W. L., Henson, V. E., and McCormick, S. F. 2000. A multigrid tutorial: second edition. Society for Industrial and Applied Mathematics, Philadelphia, PA, USA. Google ScholarDigital Library
- Buck, I. 2007. GPU computing with NVIDIA CUDA. In SIGGRAPH '07 Course Notes. Google ScholarDigital Library
- Chen, L.-Q., Xie, X., Fan, X., Ma, W.-Y., Zhang, H., and Zhou, H.-Q. 2003. A visual attention model for adapting images on small displays. Multimedia Syst. 9, 4, 353--364.Google ScholarDigital Library
- David, H. A. 1963. The Method of Paired Comparisons. Charles Griffin&Company.Google Scholar
- Deselaers, T., Dreuw, P., and Ney, H. 2008. Pan, zoom, scan -- time-coherent, trained automatic video cropping. In CVPR.Google Scholar
- Ell, T. A., and Sangwine, S. J. 2007. Hypercomplex fourier transforms of color images. IEEE Transactions on Image Processing 16, 1, 22--35. Google ScholarDigital Library
- Gal, R., Sorkine, O., and Cohen-Or, D. 2006. Feature-aware texturing. In Proceedings of Eurographics Symposium on Rendering, 297--303. Google ScholarDigital Library
- Gonzalez, R. C., and Woods, R. E. 2002. Digital Image Processing. Prentice Hall. Google ScholarDigital Library
- Greene, N., and Heckbert, P. S. 1986. Creating raster omnimax images from multiple perspective views using the elliptical weighted average filter. IEEE Comput. Graph. Appl. 6, 6, 21--27. Google ScholarDigital Library
- Guo, C., Ma, Q., and Zhang, L. 2008. Spatio-temporal saliency detection using phase spectrum of quaternion fourier transform. CVPR.Google Scholar
- Horn, B. K. P., and Schunck, B. G. 1981. Determining optical flow. Artificial Intelligence 17, 1--3, 185--203.Google ScholarDigital Library
- Itti, L., Koch, C., and Niebur, E. 1998. A model of saliency-based visual attention for rapid scene analysis. IEEE PAMI 20, 11, 1254--1259. Google ScholarDigital Library
- Knoche, H., Papaleo, M., Sasse, M. A., and Vanelli-Coralli, A. 2007. The kindest cut: Enhancing the user experience of mobile tv through adequate zooming. In ACM Multimedia, 87--96. Google ScholarDigital Library
- Kraevoy, V., Sheffer, A., Shamir, A., and Cohen-Or, D. 2008. Non-homogeneous resizing of complex models. ACM Trans. Graph. 27, 5, 111. Google ScholarDigital Library
- Liu, F., and Gleicher, M. 2006. Video retargeting: automating pan and scan. In ACM Multimedia, 241--250. Google ScholarDigital Library
- Rubinstein, M., Shamir, A., and Avidan, S. 2008. Improved seam carving for video retargeting. ACM Trans. Graph. 27, 3, 16. Google ScholarDigital Library
- Rubinstein, M., Shamir, A., and Avidan, S. 2009. Multi-operator media retargeting. ACM Trans. Graph. 28, 3, 23. Google ScholarDigital Library
- Schaefer, S., McPhail, T., and Warren, J. D. 2006. Image deformation using moving least squares. ACM Trans. Graph. 25, 3, 533--540. Google ScholarDigital Library
- Segal, M., and Akeley, K., 2006. The OpenGL Graphics System: A Specification (Version 2.1). http://www.opengl.org.Google Scholar
- Setlur, V., Takagi, S., Raskar, R., Gleicher, M., and Gooch, B. 2005. Automatic image retargeting. In MUM, 59--68. Google ScholarDigital Library
- Viola, P. A., and Jones, M. J. 2004. Robust real-time face detection. IJCV 57, 2, 137--154. Google ScholarDigital Library
- Wang, Y.-S., Tai, C.-L., Sorkine, O., and Lee, T.-Y. 2008. Optimized scale-and-stretch for image resizing. ACM Trans. Graph. 27, 5, 118. Google ScholarDigital Library
- Wang, Y.-S., Fu, H., Sorkine, O., Lee, T.-Y., and Seidel, H.-P. 2009. Motion-aware temporal coherence for video resizing. ACM Trans. Graph. 28, 5. Google ScholarDigital Library
- Wolf, L., Guttmann, M., and Cohen-Or, D. 2007. Non-homogeneous content-driven video-retargeting. In ICCV, 1--6.Google Scholar
- Zabih, R., Miller, J., and Mai, K. 1995. A feature-based algorithm for detecting and classifying scene breaks. In ACM Multimedia, 189--200. Google ScholarDigital Library
- Zhang, Y.-F., Hu, S.-M., and Martin, R. R. 2008. Shrinkability maps for content-aware video resizing. In Pacific Graphics.Google Scholar
- Zwicker, M., Pfister, H., van Baar, J., and Gross, M. H. 2002. Ewa splatting. IEEE Trans. Vis. Comput. Graph. 8, 3, 223--238. Google ScholarDigital Library
- Zwicker, M., Räsänen, J., Botsch, M., Dachsbacher, C., and Pauly, M. 2004. Perspective accurate splatting. In Graphics Interface, 247--254. Google ScholarDigital Library
Index Terms
- A system for retargeting of streaming video
Recommendations
A system for retargeting of streaming video
SIGGRAPH Asia '09: ACM SIGGRAPH Asia 2009 papersWe present a novel, integrated system for content-aware video retargeting. A simple and interactive framework combines key frame based constraint editing with numerous automatic algorithms for video analysis. This combination gives content producers ...
Sea of Images: A Dense Sampling Approach for Rendering Large Indoor Environments
Visual simulation of large real-world environments is one of the grand challenges of computer graphics. Applications include remote education, virtual heritage, specialist training, electronic commerce, and entertainment. The sea of images image-based ...
3-D transformations of images in scanline order
SIGGRAPH '80: Proceedings of the 7th annual conference on Computer graphics and interactive techniquesCurrerntly texture mapping onto projections of 3-D surfaces is time consuming and subject to considerable aliasing errors. Usually the procedure is to perform some inverse mapping from the area of the pixel onto the surface texture. It is difficult to ...
Comments