skip to main content
research-article

Stereoscopic 3D copy & paste

Published:15 December 2010Publication History
Skip Abstract Section

Abstract

With the increase in popularity of stereoscopic 3D imagery for film, TV, and interactive entertainment, an urgent need for editing tools to support stereo content creation has become apparent. In this paper we present an end-to-end system for object copy & paste in a stereoscopic setting to address this need. There is no straightforward extension of 2D copy & paste to support the addition of the third dimension as we show in this paper. For stereoscopic copy & paste we need to handle depth, and our core objective is to obtain a convincing 3D viewing experience. As one of the main contributions of our system, we introduce a stereo billboard method for stereoscopic rendering of the copied selection. Our approach preserves the stereo volume and is robust to the inevitable inaccuracies of the depth maps computed from a stereo pair of images. Our system also includes an interactive stereoscopic segmentation tool to achieve high quality object selection. Hence, we focus on intuitive and minimal user interaction, and our editing operations perform within interactive rates to provide immediate feedback.

Skip Supplemental Material Section

Supplemental Material

References

  1. Agarwala, A., Hertzmann, A., Salesin, D. H., and Seitz, S. M. 2004. Keyframe-based tracking for rotoscoping and animation. ACM Trans. on Graph. 23, 3 (Aug.), 584--591. Google ScholarGoogle ScholarDigital LibraryDigital Library
  2. Bai, X., Wang, J., Simons, D., and Sapiro, G. 2009. Video snapcut: robust video object cutout using localized classifiers. ACM Trans. on Graph. 28, 3 (Aug.). Google ScholarGoogle ScholarDigital LibraryDigital Library
  3. Besl, P. J., and McKay, N. D. 1992. A method for registration of 3-d shapes. IEEE Trans. Pattern Anal. Mach. Intell. 14, 2, 239--256. Google ScholarGoogle ScholarDigital LibraryDigital Library
  4. Boykov, Y., Veksler, O., and Zabih, R. 2001. Fast approximate energy minimization via graph cuts. IEEE Trans. on Pattern Anal. and Mach. Intell. 23, 11, 1222--1239. Google ScholarGoogle ScholarDigital LibraryDigital Library
  5. Chuang, Y.-Y., Agarwala, A., Curless, B., Salesin, D. H., and Szeliski, R. 2002. Video matting of complex scenes. In Proc. of SIGGRAPH 2002, ACM Press / ACM SIGGRAPH, J. F. Hughes, Ed., ACM, 243--248. Google ScholarGoogle ScholarDigital LibraryDigital Library
  6. Comaniciu, D., and Meer, P. 2002. Mean shift: A robust approach toward feature space analysis. IEEE Trans. on Pattern Anal. and Mach. Intell. 24, 603--619. Google ScholarGoogle ScholarDigital LibraryDigital Library
  7. Dong Seon, C., and Figueiredo, M. A. T. 2007. Cosegmentation for image sequences. Int. Conf. on Image Anal. and Proc., 635--640. Google ScholarGoogle ScholarDigital LibraryDigital Library
  8. Farbman, Z., Hoffer, G., Lipman, Y., Cohen-Or, D., and Lischinski, D. 2009. Coordinates for instant image cloning. ACM Trans. on Graph. 28, 3 (Aug.). Google ScholarGoogle ScholarDigital LibraryDigital Library
  9. Fuji, 2009. Finepix REAL 3D W1. http://www.fujifilm.com/products/3d/camera/finepix_real3dw1/.Google ScholarGoogle Scholar
  10. Fukuda, K., Wilcox, L. M., Allison, R., and Howard, I. P. 2009. A reevaluation of the tolerance to vertical misalignment in stereopsis. Journal of Vision 9, 2 (February), 1--8.Google ScholarGoogle ScholarCross RefCross Ref
  11. Georgiev, T. 2006. Covariant derivatives and vision. Proc. of European Conf. on Comp. Vision 4, 56--69. Google ScholarGoogle ScholarDigital LibraryDigital Library
  12. Hartley, R. I., and Zisserman, A. 2004. Multiple View Geometry in Computer Vision, second ed. Cambridge University Press, ISBN: 0521540518. Google ScholarGoogle ScholarDigital LibraryDigital Library
  13. Howard, I. P., and Rogers, B. J. 2002. Seeing in Depth, Basic Mechanics & Depth Perception, vol. 1 & 2. I Porteous, Thornhill, Ontario.Google ScholarGoogle Scholar
  14. Jia, J., Sun, J., Tang, C.-K., and Shum, H.-Y. 2006. Drag-and-drop pasting. ACM Trans. on Graph. 25, 3 (July), 631--637. Google ScholarGoogle ScholarDigital LibraryDigital Library
  15. Koppal, S., Zitnick, C., Cohen, M., Kang, S., Ressler, B., and Colburn, A. 2010. A viewer-centric editor for stereoscopic cinema. IEEE Comp. Graph. and Appl. Preprint.Google ScholarGoogle Scholar
  16. Lalonde, J.-F., Hoiem, D., Efros, A. A., Rother, C., Winn, J., and Criminisi, A. 2007. Photo clip art. ACM Trans. on Graph. 26, 3 (July). Google ScholarGoogle ScholarDigital LibraryDigital Library
  17. Lambooij, M., IJsselsteijn, W., Fortuin, M., and Heynderickx, I. 2009. Visual discomfort and visual fatigue of stereoscopic displays: A review. Journal of Imaging Science and Tech. 53, 3, 030201.Google ScholarGoogle ScholarCross RefCross Ref
  18. Lang, M., Hornung, A., Wang, O., Poulakos, S., Smolic, A., and Gross, M. 2010. Nonlinear disparity mapping for stereoscopic 3d. ACM Trans. on Graph. 29, 4 (July). Google ScholarGoogle ScholarDigital LibraryDigital Library
  19. Liu, F., Gleicher, M., Jin, H., and Agarwala, A. 2009. Content-preserving warps for 3d video stabilization. ACM Trans. on Graph. 28, 3. Google ScholarGoogle ScholarDigital LibraryDigital Library
  20. Liu, J., Sun, J., and Shum, H.-Y. 2009. Paint selection. ACM Trans. on Graph. 28, 3 (Aug.). Google ScholarGoogle ScholarDigital LibraryDigital Library
  21. Loos, B. J., and Sloan, P.-P. 2010. Volumetric obscurance. In ACM Symp. on Interactive 3D Graph. and Games, ACM, New York, NY, USA, 151--156. Google ScholarGoogle ScholarDigital LibraryDigital Library
  22. Lu, F., Fu, Z., and Robles-Kelly, A. 2007. Efficient graph cuts for multiclass interactive image segmentation. Proc. of the Asian Conf. on Comp. vision, 134--144. Google ScholarGoogle ScholarDigital LibraryDigital Library
  23. Mammen, A. 1989. Transparency and antialiasing algorithms implemented with the virtual pixel maps technique. IEEE Comp. Graph. and Appl. 9, 4, 43--55. Google ScholarGoogle ScholarDigital LibraryDigital Library
  24. Ning, J., Zhang, L., Zhang, D., and Wu, C. 2010. Interactive image segmentation by maximal similarity based region merging. Pattern Recogn. 43, 2, 445--456. Google ScholarGoogle ScholarDigital LibraryDigital Library
  25. Patterson, R. 2007. Human factors of 3d displays. Journal of the Soc. for Information Disp., 15, 861--871.Google ScholarGoogle ScholarCross RefCross Ref
  26. Pérez, P., Gangnet, M., and Blake, A. 2003. Poisson image editing. ACM Trans. on Graph. 22, 3 (July), 313--318. Google ScholarGoogle ScholarDigital LibraryDigital Library
  27. Reinhard, E., Ashikhmin, M., Gooch, B., and Shirley, P. 2001. Color transfer between images. IEEE Comp. Graph. and Appl. 21, 5, 34--41. Google ScholarGoogle ScholarDigital LibraryDigital Library
  28. Rhee, S.-M., Ziegler, R., Park, J., Naef, M., Gross, M., and Kim, M.-H. 2007. Low-cost telepresence for collaborative virtual environments. IEEE Trans on Vis. and Comp. Graph. 13, 1, 156--166. Google ScholarGoogle ScholarDigital LibraryDigital Library
  29. Rother, C., Kolmogorov, V., and Blake, A. 2004. "grab-cut": interactive foreground extraction using iterated graph cuts. ACM Trans. on Graph. 23, 3 (Aug.), 309--314. Google ScholarGoogle ScholarDigital LibraryDigital Library
  30. Rother, C., Minka, T., Blake, A., and Kolmogorov, V. 2006. Cosegmentation of image pairs by histogram matching. IEEE Conf. on Comp. Vision and Pattern Recog., 993--1000. Google ScholarGoogle ScholarDigital LibraryDigital Library
  31. Scharstein, D., and Szeliski., R. 2002. A taxonomy and evaluation of dense two-frame stereo correspondence algorithms. Int. Journal of Comp. Vision 47, 1/2/3, 7--42. Google ScholarGoogle ScholarDigital LibraryDigital Library
  32. Scharstein, D., and Szeliski, R., 2010. Middlebury Stereo Repository. http://vision.middlebury.edu/stereo/.Google ScholarGoogle Scholar
  33. Shum, H. Y., Sun, J., Yamazaki, S., Li, Y., and Tang, C. K. 2004. Pop-up light field: An interactive image-based modeling and rendering system. ACM Trans. on Graph. 23, 2 (Aug.), 143--162. Google ScholarGoogle ScholarDigital LibraryDigital Library
  34. Smith, B., Zhang, L., and Jin, H. 2009. Stereo matching with nonparametric smoothness priors in feature space. IEEE Conf. on Comp. Vision and Pattern Recog., 485--492.Google ScholarGoogle Scholar
  35. Taguchi, Y., Wilburn, B., and Zitnick, C. L. 2008. Stereo reconstruction with mixed pixels using adaptive over-segmentation. IEEE Conf. on Comp. Vision and Pattern Recog., 1--8.Google ScholarGoogle Scholar
  36. The Foundry, 2010. Nuke - Ocula Plug-in. http://www.thefoundry.co.uk/.Google ScholarGoogle Scholar
  37. Wang, J., and Cohen, M. F. 2008. Image and video matting: A survey. Foundations and Trends in Comp. Graph. and Vision 3, 2, 97--175. Google ScholarGoogle ScholarDigital LibraryDigital Library
  38. Wang, C., and Sawchuk, A. A. 2008. Disparity manipulation for stereo images and video. Stereoscopic Disp. and Appl. 6803, 1, 68031E.Google ScholarGoogle ScholarCross RefCross Ref
  39. Wang, L., Jin, H., Yang, R., and Gong, M. 2008. Stereoscopic inpainting: Joint color and depth completion from stereo images. In IEEE Conf. on Comp. Vision and Pattern Recog., 1--8.Google ScholarGoogle Scholar
  40. Zitnick, C. L., and Kang, S. B. 2007. Stereo for image-based rendering using image over-segmentation. Int. Journal of Comp. Vision 75, 1, 49--65. Google ScholarGoogle ScholarDigital LibraryDigital Library
  41. Zitnick, C. L., Kang, S. B., Uyttendaele, M., Winder, S., and Szeliski, R. 2004. High-quality video view interpolation using a layered representation. ACM Trans. on Graph. 23, 3 (Aug.), 600--608. Google ScholarGoogle ScholarDigital LibraryDigital Library
  42. Zitnick, C. L., Jojic, N., and Kang, S. B. 2005. Consistent segmentation for optical flow estimation. IEEE Int. Conf. on Comp. Vision, 1308--1315. Google ScholarGoogle ScholarDigital LibraryDigital Library
  43. Zwicker, M., Pfister, H., van Baar, J., and Gross, M. 2002. Ewa splatting. IEEE Trans. on Vis. and Comp. Graph. 8, 3, 223--238. Google ScholarGoogle ScholarDigital LibraryDigital Library

Index Terms

  1. Stereoscopic 3D copy & paste

      Recommendations

      Reviews

      Changyin Zhou

      Three-dimensional movies, television programs, and games are becoming increasingly popular. This trend fosters an urgent need for tools to edit stereoscopic 3D images. "Copy & paste" is one of the most-wanted editing functions. However, 2D copy & paste cannot be extended to 3D in a straightforward manner, since a 3D operation has to deal with depth. In this paper, the authors present a stereo billboard technique to perform 3D copy & paste. Three-dimensional copy & paste poses at least three major challenges. First, the disparity of the copied object has to be consistent with the target region (for example, the shape and the stereo volume should be preserved.) Second, occlusion should be correctly handled because it is an important depth cue for humans and humans are very sensitive to errors in it. Finally, the depth information that one can reconstruct from stereo images is often inaccurate and noisy due to the image noise and the lack of corresponding points in stereo images. A good 3D copy & paste technique must be insensitive to these imperfections. This paper proposes a technique that addresses all three challenges. The proposed technique first leverages user interaction and a previously proposed mean-shift segmentation method to select an object to copy. Users can then rotate the copied object under certain constraints and paste it on the target region. The proposed billboard technique ensures that the volume of the object won't change during this process. The objects are sorted according to depth so that occlusion can be rendered correctly. This technique is seemingly designed to be an essential tool for 3D image editing. It may not work when one wants to manipulate the surfaces of complicated shapes or rotate a selected object by a large angle. However, these limitations could be addressed in future work or other 3D editing tools. Online Computing Reviews Service

      Access critical reviews of Computing literature here

      Become a reviewer for Computing Reviews.

      Comments

      Login options

      Check if you have access through your login credentials or your institution to get full access on this article.

      Sign in

      Full Access

      • Published in

        cover image ACM Transactions on Graphics
        ACM Transactions on Graphics  Volume 29, Issue 6
        December 2010
        480 pages
        ISSN:0730-0301
        EISSN:1557-7368
        DOI:10.1145/1882261
        Issue’s Table of Contents

        Copyright © 2010 ACM

        Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

        Publisher

        Association for Computing Machinery

        New York, NY, United States

        Publication History

        • Published: 15 December 2010
        Published in tog Volume 29, Issue 6

        Permissions

        Request permissions about this article.

        Request Permissions

        Check for updates

        Qualifiers

        • research-article

      PDF Format

      View or Download as a PDF file.

      PDF

      eReader

      View online with eReader.

      eReader