Abstract
With the increase in popularity of stereoscopic 3D imagery for film, TV, and interactive entertainment, an urgent need for editing tools to support stereo content creation has become apparent. In this paper we present an end-to-end system for object copy & paste in a stereoscopic setting to address this need. There is no straightforward extension of 2D copy & paste to support the addition of the third dimension as we show in this paper. For stereoscopic copy & paste we need to handle depth, and our core objective is to obtain a convincing 3D viewing experience. As one of the main contributions of our system, we introduce a stereo billboard method for stereoscopic rendering of the copied selection. Our approach preserves the stereo volume and is robust to the inevitable inaccuracies of the depth maps computed from a stereo pair of images. Our system also includes an interactive stereoscopic segmentation tool to achieve high quality object selection. Hence, we focus on intuitive and minimal user interaction, and our editing operations perform within interactive rates to provide immediate feedback.
Supplemental Material
Available for Download
- Agarwala, A., Hertzmann, A., Salesin, D. H., and Seitz, S. M. 2004. Keyframe-based tracking for rotoscoping and animation. ACM Trans. on Graph. 23, 3 (Aug.), 584--591. Google ScholarDigital Library
- Bai, X., Wang, J., Simons, D., and Sapiro, G. 2009. Video snapcut: robust video object cutout using localized classifiers. ACM Trans. on Graph. 28, 3 (Aug.). Google ScholarDigital Library
- Besl, P. J., and McKay, N. D. 1992. A method for registration of 3-d shapes. IEEE Trans. Pattern Anal. Mach. Intell. 14, 2, 239--256. Google ScholarDigital Library
- Boykov, Y., Veksler, O., and Zabih, R. 2001. Fast approximate energy minimization via graph cuts. IEEE Trans. on Pattern Anal. and Mach. Intell. 23, 11, 1222--1239. Google ScholarDigital Library
- Chuang, Y.-Y., Agarwala, A., Curless, B., Salesin, D. H., and Szeliski, R. 2002. Video matting of complex scenes. In Proc. of SIGGRAPH 2002, ACM Press / ACM SIGGRAPH, J. F. Hughes, Ed., ACM, 243--248. Google ScholarDigital Library
- Comaniciu, D., and Meer, P. 2002. Mean shift: A robust approach toward feature space analysis. IEEE Trans. on Pattern Anal. and Mach. Intell. 24, 603--619. Google ScholarDigital Library
- Dong Seon, C., and Figueiredo, M. A. T. 2007. Cosegmentation for image sequences. Int. Conf. on Image Anal. and Proc., 635--640. Google ScholarDigital Library
- Farbman, Z., Hoffer, G., Lipman, Y., Cohen-Or, D., and Lischinski, D. 2009. Coordinates for instant image cloning. ACM Trans. on Graph. 28, 3 (Aug.). Google ScholarDigital Library
- Fuji, 2009. Finepix REAL 3D W1. http://www.fujifilm.com/products/3d/camera/finepix_real3dw1/.Google Scholar
- Fukuda, K., Wilcox, L. M., Allison, R., and Howard, I. P. 2009. A reevaluation of the tolerance to vertical misalignment in stereopsis. Journal of Vision 9, 2 (February), 1--8.Google ScholarCross Ref
- Georgiev, T. 2006. Covariant derivatives and vision. Proc. of European Conf. on Comp. Vision 4, 56--69. Google ScholarDigital Library
- Hartley, R. I., and Zisserman, A. 2004. Multiple View Geometry in Computer Vision, second ed. Cambridge University Press, ISBN: 0521540518. Google ScholarDigital Library
- Howard, I. P., and Rogers, B. J. 2002. Seeing in Depth, Basic Mechanics & Depth Perception, vol. 1 & 2. I Porteous, Thornhill, Ontario.Google Scholar
- Jia, J., Sun, J., Tang, C.-K., and Shum, H.-Y. 2006. Drag-and-drop pasting. ACM Trans. on Graph. 25, 3 (July), 631--637. Google ScholarDigital Library
- Koppal, S., Zitnick, C., Cohen, M., Kang, S., Ressler, B., and Colburn, A. 2010. A viewer-centric editor for stereoscopic cinema. IEEE Comp. Graph. and Appl. Preprint.Google Scholar
- Lalonde, J.-F., Hoiem, D., Efros, A. A., Rother, C., Winn, J., and Criminisi, A. 2007. Photo clip art. ACM Trans. on Graph. 26, 3 (July). Google ScholarDigital Library
- Lambooij, M., IJsselsteijn, W., Fortuin, M., and Heynderickx, I. 2009. Visual discomfort and visual fatigue of stereoscopic displays: A review. Journal of Imaging Science and Tech. 53, 3, 030201.Google ScholarCross Ref
- Lang, M., Hornung, A., Wang, O., Poulakos, S., Smolic, A., and Gross, M. 2010. Nonlinear disparity mapping for stereoscopic 3d. ACM Trans. on Graph. 29, 4 (July). Google ScholarDigital Library
- Liu, F., Gleicher, M., Jin, H., and Agarwala, A. 2009. Content-preserving warps for 3d video stabilization. ACM Trans. on Graph. 28, 3. Google ScholarDigital Library
- Liu, J., Sun, J., and Shum, H.-Y. 2009. Paint selection. ACM Trans. on Graph. 28, 3 (Aug.). Google ScholarDigital Library
- Loos, B. J., and Sloan, P.-P. 2010. Volumetric obscurance. In ACM Symp. on Interactive 3D Graph. and Games, ACM, New York, NY, USA, 151--156. Google ScholarDigital Library
- Lu, F., Fu, Z., and Robles-Kelly, A. 2007. Efficient graph cuts for multiclass interactive image segmentation. Proc. of the Asian Conf. on Comp. vision, 134--144. Google ScholarDigital Library
- Mammen, A. 1989. Transparency and antialiasing algorithms implemented with the virtual pixel maps technique. IEEE Comp. Graph. and Appl. 9, 4, 43--55. Google ScholarDigital Library
- Ning, J., Zhang, L., Zhang, D., and Wu, C. 2010. Interactive image segmentation by maximal similarity based region merging. Pattern Recogn. 43, 2, 445--456. Google ScholarDigital Library
- Patterson, R. 2007. Human factors of 3d displays. Journal of the Soc. for Information Disp., 15, 861--871.Google ScholarCross Ref
- Pérez, P., Gangnet, M., and Blake, A. 2003. Poisson image editing. ACM Trans. on Graph. 22, 3 (July), 313--318. Google ScholarDigital Library
- Reinhard, E., Ashikhmin, M., Gooch, B., and Shirley, P. 2001. Color transfer between images. IEEE Comp. Graph. and Appl. 21, 5, 34--41. Google ScholarDigital Library
- Rhee, S.-M., Ziegler, R., Park, J., Naef, M., Gross, M., and Kim, M.-H. 2007. Low-cost telepresence for collaborative virtual environments. IEEE Trans on Vis. and Comp. Graph. 13, 1, 156--166. Google ScholarDigital Library
- Rother, C., Kolmogorov, V., and Blake, A. 2004. "grab-cut": interactive foreground extraction using iterated graph cuts. ACM Trans. on Graph. 23, 3 (Aug.), 309--314. Google ScholarDigital Library
- Rother, C., Minka, T., Blake, A., and Kolmogorov, V. 2006. Cosegmentation of image pairs by histogram matching. IEEE Conf. on Comp. Vision and Pattern Recog., 993--1000. Google ScholarDigital Library
- Scharstein, D., and Szeliski., R. 2002. A taxonomy and evaluation of dense two-frame stereo correspondence algorithms. Int. Journal of Comp. Vision 47, 1/2/3, 7--42. Google ScholarDigital Library
- Scharstein, D., and Szeliski, R., 2010. Middlebury Stereo Repository. http://vision.middlebury.edu/stereo/.Google Scholar
- Shum, H. Y., Sun, J., Yamazaki, S., Li, Y., and Tang, C. K. 2004. Pop-up light field: An interactive image-based modeling and rendering system. ACM Trans. on Graph. 23, 2 (Aug.), 143--162. Google ScholarDigital Library
- Smith, B., Zhang, L., and Jin, H. 2009. Stereo matching with nonparametric smoothness priors in feature space. IEEE Conf. on Comp. Vision and Pattern Recog., 485--492.Google Scholar
- Taguchi, Y., Wilburn, B., and Zitnick, C. L. 2008. Stereo reconstruction with mixed pixels using adaptive over-segmentation. IEEE Conf. on Comp. Vision and Pattern Recog., 1--8.Google Scholar
- The Foundry, 2010. Nuke - Ocula Plug-in. http://www.thefoundry.co.uk/.Google Scholar
- Wang, J., and Cohen, M. F. 2008. Image and video matting: A survey. Foundations and Trends in Comp. Graph. and Vision 3, 2, 97--175. Google ScholarDigital Library
- Wang, C., and Sawchuk, A. A. 2008. Disparity manipulation for stereo images and video. Stereoscopic Disp. and Appl. 6803, 1, 68031E.Google ScholarCross Ref
- Wang, L., Jin, H., Yang, R., and Gong, M. 2008. Stereoscopic inpainting: Joint color and depth completion from stereo images. In IEEE Conf. on Comp. Vision and Pattern Recog., 1--8.Google Scholar
- Zitnick, C. L., and Kang, S. B. 2007. Stereo for image-based rendering using image over-segmentation. Int. Journal of Comp. Vision 75, 1, 49--65. Google ScholarDigital Library
- Zitnick, C. L., Kang, S. B., Uyttendaele, M., Winder, S., and Szeliski, R. 2004. High-quality video view interpolation using a layered representation. ACM Trans. on Graph. 23, 3 (Aug.), 600--608. Google ScholarDigital Library
- Zitnick, C. L., Jojic, N., and Kang, S. B. 2005. Consistent segmentation for optical flow estimation. IEEE Int. Conf. on Comp. Vision, 1308--1315. Google ScholarDigital Library
- Zwicker, M., Pfister, H., van Baar, J., and Gross, M. 2002. Ewa splatting. IEEE Trans. on Vis. and Comp. Graph. 8, 3, 223--238. Google ScholarDigital Library
Index Terms
Stereoscopic 3D copy & paste
Recommendations
Nonlinear disparity mapping for stereoscopic 3D
This paper addresses the problem of remapping the disparity range of stereoscopic images and video. Such operations are highly important for a variety of issues arising from the production, live broadcast, and consumption of 3D content. Our work is ...
Stereoscopic 3D copy & paste
SIGGRAPH ASIA '10: ACM SIGGRAPH Asia 2010 papersWith the increase in popularity of stereoscopic 3D imagery for film, TV, and interactive entertainment, an urgent need for editing tools to support stereo content creation has become apparent. In this paper we present an end-to-end system for object ...
Copy and Paste: Temporally Consistent Stereoscopic Video Blending
We propose a novel method of stereoscopic video blending, targeted at achieving temporal disparity and color consistency. Video blending is one of the most frequent and important tasks in video editing, which is also true in stereoscopic video editing. ...
Comments