Fan Zhong
Dani Lischinski
Abstract
We introduce a method for replacing the background in a video of a moving foreground subject, when both the source video capturing the subject, and the target video capturing the new background scene, are natural videos, casually captured using a freely moving hand-held camera. We assume that the foreground subject has already been extracted, and focus on the challenging task of generating a video with a new background, such that the new background motion appears compatible with the original one. Failure to match the motion results in disturbing slippage or moonwalk artifacts, where the subject's feet appear to slide or slip over the ground. While matching the motion across the entire frame is impossible for scenes with differing geometry, we aim to match the local motion of the ground in the vicinity of the subject. This is achieved by reordering and warping the available target background frames in a manner that optimizes a suitably designed objective function.
- Agarwala, A., Hertzmann, A., Salesin, D. H., and Seitz, S. M. 2004. Keyframe-based tracking for rotoscoping and animation. ACM Trans. Graph. 23, 3 (Aug.), 584--591. Google Scholar
Digital Library
- Agarwala, A., Zheng, K. C., Pal, C., Agrawala, M., Cohen, M., Curless, B., Salesin, D., and Szeliski, R. 2005. Panoramic video textures. ACM Trans. Graph. 24, 3 (July), 821--827. Google ScholarDigital Library
- Bai, X., Wang, J., Simons, D., and Sapiro, G. 2009. Video SnapCut: Robust video object cutout using localized classifiers. ACM Trans. Graph. 28, 3 (July), 70:1--70:11. Google ScholarDigital Library
- Chuang, Y.-Y., Agarwala, A., Curless, B., Salesin, D. H., and Szeliski, R. 2002. Video matting of complex scenes. ACM Trans. Graph. 21, 3 (July), 243--248. Google ScholarDigital Library
- Farbman, Z., and Lischinski, D. 2011. Tonal stabilization of video. ACM Transactions on Graphics (Proceedings of ACM SIGGRAPH 2011) 30, 4, 89:1--89:9. Google ScholarDigital Library
- Flagg, M., Nakazawa, A., Zhang, Q., Kang, S. B., Ryu, Y. K., Essa, I., and Rehg, J. M. 2009. Human video textures. In Proceedings of the 2009 symposium on Interactive 3D graphics and games, ACM, 199--206. Google ScholarDigital Library
- Germann, M., Popa, T., Keiser, R., Ziegler, R., and Gross, M. 2012. Novel-view synthesis of outdoor sport events using an adaptive view-dependent geometry. Comp. Graph. Forum 31, 2pt1 (May), 325--333. Google ScholarDigital Library
- Gleicher, M. L., and Liu, F. 2008. Re-cinematography: Improving the camerawork of casual video. ACM Transactions on Multimedia Computing, Communications, and Applications (TOMCCAP) 5, 1, 2. Google ScholarDigital Library
- Goldstein, A., and Fattal, R. 2012. Video stabilization using epipolar geometry. ACM Trans. Graph. 31, 5, 126. Google ScholarDigital Library
- Grundmann, M., Kwatra, V., and Essa, I. 2011. Auto-directed video stabilization with robust l1 optimal camera paths. In Computer Vision and Pattern Recognition (CVPR), 2011 IEEE Conference on, IEEE, 225--232. Google ScholarDigital Library
- Hartley, R., and Zisserman, A. 2004. Multiple view geometry in computer vision, 2nd ed. Cambridge Univ Press. Google ScholarDigital Library
- Liu, F., Gleicher, M., Jin, H., and Agarwala, A. 2009. Content-preserving warps for 3D video stabilization. ACM Trans. Graph. 28, 3 (July), 44:1--44:9. Google ScholarDigital Library
- Liu, F., Gleicher, M., Wang, J., Jin, H., and Agarwala, A. 2011. Subspace video stabilization. ACM Trans. Graph. 30, 1, 4. Google ScholarDigital Library
- Liu, S., Yuan, L., Tan, P., and Sun, J. 2013. Bundled camera paths for video stabilization. ACM Trans. Graph. 32, 4, 78. Google ScholarDigital Library
- Okabe, M., Anjyor, K., and Onai, R. 2011. Creating fluid animation from a single image using video database. Computer Graphics Forum 30, 7, 1973--1982.Google ScholarCross Ref
- Sand, P., and Teller, S. 2004. Video matching. ACM Trans. Graph. 23, 3, 592--599. Google ScholarDigital Library
- Schödl, A., Szeliski, R., Salesin, D. H., and Essa, I. 2000. Video textures. In Proc. 27th annual conference on Computer Graphics and interactive techniques, ACM Press/Addison-Wesley, 489--498. Google ScholarDigital Library
- Smith, A. R., and Blinn, J. F. 1996. Blue screen matting. In Proc. 23rd annual conference on Computer Graphics and interactive techniques, ACM, 259--268. Google ScholarDigital Library
- Steedly, D., Pal, C., and Szeliski, R. 2005. Efficiently registering video into panoramic mosaics. In Proc. ICCV '05, vol. 2. Google ScholarDigital Library
- Stich, T., Linz, C., Albuquerque, G., and Magnor, M. 2008. View and time interpolation in image space. Comp. Graph. Forum 27, 7, 1781--1787.Google ScholarCross Ref
- Sunkavalli, K., Johnson, M. K., Matusik, W., and Pfister, H. 2010. Multi-scale image harmonization. ACM Trans. Graphics 29, 4, 125:1--125:10. Google ScholarDigital Library
- Wang, J., Bhat, P., Colburn, R. A., Agrawala, M., and Cohen, M. F. 2005. Interactive video cutout. ACM Trans. Graph. 24, 3 (July), 585--594. Google ScholarDigital Library
- Xu, F., Liu, Y., Stoll, C., Tompkin, J., Bharaj, G., Dai, Q., Seidel, H.-P., Kautz, J., and Theobalt, C. 2011. Video-based characters: Creating new human performances from a multi-view video database. ACM Trans. Graph. 30, 4 (July), 32:1--32:10. Google ScholarDigital Library
- Zhang, Y., Correa, C. D., and Ma, K.-L. 2011. Graph-based fire synthesis. In Eurographics/ACM SIGGRAPH Symposium on Computer Animation, 187--194. Google ScholarDigital Library
Index Terms
- Slippage-free background replacement for hand-held video
Recommendations
Illumination-aware live videos background replacement using antialiasing optimization
We propose a real-time illumination-aware live videos background replacement approach with antialiasing optimization on GPU in this paper. The aim of background replacement for live videos is to substitute the current real-time backgrounds with ...
Homography-based block motion estimation for video coding of PTZ cameras
We propose a homography-based search (HBS) algorithm for block motion estimation.We use optical flow tracking algorithm to obtain homography between two frames.Adaptive thresholds are adopted in our method to classify different kinds of blocks. Due to ...
Visual Modeling with a Hand-Held Camera
In this paper a complete system to build visual models from camera images is presented. The system can deal with uncalibrated image sequences acquired with a hand-held camera. Based on tracked or matched features the relations between multiple views are ...
Comments