Skip to main content
SpringerLink
Log in

View-Aware Image Object Compositing and Synthesis from Multiple Sources

  • Regular Paper
  • Published:
Journal of Computer Science and Technology Aims and scope Submit manuscript

Abstract

Image compositing is widely used to combine visual elements from separate source images into a single image. Although recent image compositing techniques are capable of achieving smooth blending of the visual elements from different sources, most of them implicitly assume the source images are taken in the same viewpoint. In this paper, we present an approach to compositing novel image objects from multiple source images which have different viewpoints. Our key idea is to construct 3D proxies for meaningful components of the source image objects, and use these 3D component proxies to warp and seamlessly merge components together in the same viewpoint. To realize this idea, we introduce a coordinate-frame based single-view camera calibration algorithm to handle general types of image objects, a structure-aware cuboid optimization algorithm to get the cuboid proxies for image object components with correct structure relationship, and finally a 3D-proxy transformation guided image warping algorithm to stitch object components. We further describe a novel application based on this compositing approach to automatically synthesize a large number of image objects from a set of exemplars. Experimental results show that our compositing approach can be applied to a variety of image objects, such as chairs, cups, lamps, and robots, and the synthesis application can create novel image objects with significant shape and style variations from a small set of exemplars.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. Perez P, Gangnet M, Blake A. Poisson image editing. ACM Transactions on Graphics, 2003, 22(3): 313–318.

    Article  Google Scholar 

  2. Jia J, Sun J, Tang C K, Shum H Y. Drag-and-drop pasting. ACM Transactions on Graphics, 2006, 25(3): 631–637.

    Article  Google Scholar 

  3. Farbman Z, Hoffer G, Lipman Y, Cohen-Or D, Lischinski D. Coordinates for instant image cloning. ACM Transactions on Graphics, 2009, 28(3): Article No. 67.

  4. Tao MW, Johnson M K, Paris S. Error-tolerant image compositing. In Proc. the 11th European Conference on Computer Vision, Sept. 2010, pp.31-44.

  5. Sunkavalli K, Johnson M K, Matusik W, Pfister H. Multiscale image harmonization. ACM Transactions on Graphics, 2010, 29(4): Article No. 125.

  6. Agarwala A, Dontcheva M, Agrawala M, Drucker S, Colburn A, Curless B, Salesin D, Cohen M. Interactive digital photomontage. ACM Transactions on Graphics, 2004, 23(3): 294–302.

    Article  Google Scholar 

  7. Rother C, Kumar S, Kolmogorov V, Blake A. Digital tapestry [automatic image synthesis]. In Proc. IEEE CVPR, June 2005, pp.589-596.

  8. Rother C, Bordeaux L, Hamadi Y, Blake A. AutoCollage. ACM Transactions on Graphics, 2006, 25(3): 847–852.

    Article  Google Scholar 

  9. Wang J, Quan L, Sun J, Tang X, Shum H Y. Picture collage. In Proc. IEEE CVPR, June 2006, pp.347-354.

  10. Chen T, Cheng M M, Tan P, Shamir A, Hu S M. Sketch2Photo: Internet image montage. ACM Transactions on Graphics, 2009, 28(5): 124:1–124:10.

    Google Scholar 

  11. Eitz M, Richter R, Hildebrand K, Boubekeur T, Alexa M. Photosketcher: Interactive sketch-based image synthesis. IEEE Computer Graphics and Applications, 2011, 31(6): 56–66.

    Article  Google Scholar 

  12. [12] Kalogerakis E, Chaudhuri S, Koller D, Koltun V. A probabilistic model for component-based shape synthesis. ACM Trans. Graph., 2012, 31(4): 55:1–55:11.

    Article  Google Scholar 

  13. Xu K, Zhang H, Cohen-Or D, Chen B. Fit and diverse: Set evolution for inspiring 3D shape galleries. ACM Trans. Graph., 2012, 31(4): 57:1–57:10.

    Article  Google Scholar 

  14. Burt P J, Adelson E H. A multiresolution spline with application to image mosaics. ACM Trans. Graph., 1983, 2(4): 217–236.

    Article  Google Scholar 

  15. Ogden J M, Adelson E H, Bergen J R, Burt P J. Pyramidbased computer graphics. RCA Engineer, 1985, 30(5): 4–15.

    Google Scholar 

  16. Porter T, Duff T. Compositing digital images. ACM SIGGRAPH Comput. Graph., 1984, 18(3): 253–259.

    Article  Google Scholar 

  17. Xue S, Agarwala A, Dorsey J, Rushmeier H. Understanding and improving the realism of image composites. ACM Transactions on Graphics, 2012, 31(4): Article No. 84.

  18. Diakopoulos N, Essa I, Jain R. Content based image synthesis. In Proc. the 3rd CIVR, July 2004, pp.299-307.

  19. Johnson M, Brostow G J, Shotton J et al. Semantic photo synthesis. Computer Graphics Forum, 2006, 25(3): 407–413.

    Article  Google Scholar 

  20. Lalonde J F, Hoiem D, Efros A A, Rother C, Winn J, Criminisi A. Photo clip art. ACM Transactions on Graphics, 2007, 26(3): Article No. 3.

  21. Hall P, Cai H, Wu Q, Corradi T. Cross-depiction problem: Recognition and synthesis of photographs and artwork. Computational Visual Media, 2015, 1(2): 91–103.

    Article  Google Scholar 

  22. Huang H, Zhang L, Zhang H C. Arcimboldo-like collage using internet images. ACM Transactions on Graphics, 2011, 30(6): Article No. 155.

  23. Yu Z, Lu L, Guo Y, Fan R, Liu M, Wang W. Content-aware photo collage using circle packing. IEEE Transactions on Visualization and Computer Graphics, 2014, 20(2): 182-195.

    Article  Google Scholar 

  24. Risser E, Han C, Dahyot R, Grinspun E. Synthesizing structured image hybrids. ACM Transactions on Graphics, 2010, 29(4): Article No. 85.

  25. Carroll R, Agarwala A, Agrawala M. Image warps for artistic perspective manipulation. ACM Transactions on Graphics, 2010, 29(4): Article No. 127.

  26. Zheng Y, Chen X, Cheng M M, Zhou K, Hu S M, Mitra N J. Interactive images: Cuboid proxies for smart image manipulation. ACM Trans. Graph., 2012, 31(4): 99:1–99:11.

    Google Scholar 

  27. Chen T, Zhu Z, Shamir A, Hu S M, Cohen-Or D. 3-sweep: Extracting editable objects from a single photo. ACM Transactions on Graphics, 2013, 32(6): Article No. 195.

  28. Miao Y, Hu F, Zhang X, Chen J, Pajarola R. SymmSketch: Creating symmetric 3D free-form shapes from 2D sketches. Computational Visual Media, 2015, 1(1): 3–16.

    Article  Google Scholar 

  29. Funkhouser T, Kazhdan M, Shilane P, Min P, KieferW, Tal A, Rusinkiewicz S, Dobkin D. Modeling by example. ACM Trans. Graph., 2004, 23(3): 652–663.

    Article  Google Scholar 

  30. Shin H, Igarashi T. Magic canvas: Interactive design of a 3-D scene prototype from freehand sketches. In Proc. Graphics Interface, May 2007, pp.63-70.

  31. Lee J, Funkhouser T. Sketch-based search and composition of 3D models. In Proc. the 5th SBM, June 2008, pp.97-104.

  32. Xu K, Chen K, Fu H, Sun W L, Hu S M. Sketch2Scene: Sketch-based co-retrieval and co-placement of 3D models. ACM Transactions on Graphics, 2013, 32(4): Article No. 123.

  33. Kreavoy V, Julius D, Sheffer A. Model composition from interchangeable components. In Proc. the 15th PG, Oct. 2007, pp.129-138.

  34. Chaudhuri S, Koltun V. Data-driven suggestions for creativity support in 3D modeling. ACM Trans. Graph., 2010, 29(6): 183:1–183:10.

    Article  Google Scholar 

  35. Chaudhuri S, Kalogerakis E, Guibas L, Koltun V. Probabilistic reasoning for assembly-based 3D modeling. ACM Trans. Graph., 2011, 30(4): 35:1–35:10.

    Article  Google Scholar 

  36. Li Y, Sun J, Tang C K, Shum H Y. Lazy snapping. ACM Transactions on Graphics, 2004, 23(3): 303–308.

    Article  Google Scholar 

  37. Russell B C, Torralba A, Murphy K P, Freeman W T. LabelMe: A database and web-based tool for image annotation. International Journal of Computer Vision, 2008, 77(1/2/3): 157–173.

  38. Barnes C, Shechtman E, Finkelstein A, Goldman D B. PatchMatch: A randomized correspondence algorithm for structural image editing. ACM Transactions on Graphics, 2009, 28(3): 24:1–24:11.

    Article  Google Scholar 

  39. Criminisi A, Reid I, Zisserman A. Single view metrology. International Journal of Computer Vision, 2000, 40(2): 123-148.

    Article  MATH  Google Scholar 

  40. Sinha S N, Steedly D, Szeliski R, Agrawala M, Pollefeys M. Interactive 3D architectural modeling from unordered photo collections. ACM Transactions on Graphics, 2008, 27(5): 159:1–159:10.

    Article  Google Scholar 

  41. Wilczkowiak M, Sturm P, Boyer E. Using geometric constraints through parallelepipeds for calibration and 3D modeling. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2005, 27(2): 194–207.

    Article  Google Scholar 

  42. Jiang N, Tan P, Cheong L F. Symmetric architecture modeling with a single image. ACM Transactions on Graphics, 2009, 28(5): 113:1–113:8.

    Article  Google Scholar 

  43. Shen C H, Fu H, Chen K, Hu S M. Structure recovery by part assembly. ACM Transactions on Graphics, 2012, 31(6): Article No. 180.

  44. O’Donovan P, Agarwala A, Hertzmann A. Color compatibility from large datasets. ACM Transactions on Graphics, 2011, 30(4): Article No. 63.

  45. Yu L F, Yeung S K, Terzopoulos D, Chan T F. DressUp!: Outfit synthesis through automatic optimization. ACM Transactions on Graphics, 2012, 31(6): 134:1–134:14.

    Article  Google Scholar 

  46. Xu K, Zheng H, Zhang H, Cohen-Or D, Liu L, Xiong Y. Photo-inspired model-driven 3D object modeling. ACM Trans. Graph., 2011, 30(4): 80:1–80:10.

    Google Scholar 

  47. Cootes T F, Taylor C J, Cooper D H, Graham J et al. Active shape models — Their training and application. Computer Vision and Image Understanding, 1995, 61(1): 38–59.

    Article  Google Scholar 

  48. Schwarz G. Estimating the dimension of a model. The Annals of Statistics, 1978, 6(2): 461–464.

    Article  MathSciNet  MATH  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. State Key Laboratory of Computer Aided Design and Computer Graphics, Zhejiang University, Hangzhou, 310058, China

    Xiang Chen, Wei-Wei Xu & Kun Zhou

  2. Vision, Graphics and Computational Design Group, Singapore University of Technology and Design, Singapore, 487372, Singapore

    Sai-Kit Yeung

Authors
  1. Xiang Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Wei-Wei Xu
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Sai-Kit Yeung
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Kun Zhou
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Kun Zhou.

Additional information

Special Section of CVM 2016

This work is partially supported by the National Natural Science Foundation of China under Grant Nos. 61272305, 61303136, 61272392, and 61322204, and the National Program for Special Support of Eminent Professionals of China.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Chen, X., Xu, WW., Yeung, SK. et al. View-Aware Image Object Compositing and Synthesis from Multiple Sources. J. Comput. Sci. Technol. 31, 463–478 (2016). https://doi.org/10.1007/s11390-016-1640-8

Download citation

  • Received:

  • Revised:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11390-016-1640-8

Keywords

Search

Navigation