Skip to main content
SpringerLink
Log in

Complex object relighting via split-then-composition by semantics and materials

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

Complex object illumination transfer is a special challenge in computer vision. In our paper, we put forward a method for complex object illumination transfer. Firstly, the input object image was divided into object components by semantic analysis, to find the reference object images consistent with the object component material in the physical world by material analysis. Material has a great influence on the illumination transfer of object image, so the use of material analysis can greatly reduce the influence of material on the illumination transfer in the later stage. Next, a block matching algorithm was used to deform each reference object image and made it match with each component shape of the input object image. Then, each component of the input object image and each warped reference object image were illuminated by local and global transfer model. Finally, semantic analysis was used to synthesize the re-illumination components of the input object image to obtain the re-illumination input object image. The experimental results prove that the method could make a good effect on the illumination transfer. Our main contribution is the use of semantic and material analysis to split complex objects into simple objects, and skillfully combine semantic and material parsing and composition, block matching algorithm, local and global light migration model to achieve the relighting of complex objects.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11

Similar content being viewed by others

References

  1. Barnes C, Shechtman E, Goldman DB, Finkelstein A (2010) The generalized patchmatch correspondence algorithm. In: Computer vision - ECCV 2010 11th European Conference on Computer Vision, Heraklion, Crete, Greece, September 5-11, 2010, Proceedings, Part III, pp 29–43

  2. Chabert C-F, Einarsson P, Jones A, Lamond B, Ma W-C, Sylwan S, Hawkins T, Debevec PE (2006) Relighting human locomotion with flowed reflectance fields. In: Finnegan JW, Pfister H (eds) International conference on computer graphics and interactive techniques, SIGGRAPH 2006, Boston, Massachusetts, USA, July 30 - August 3, 2006, Sketches, vol 33. ACM, p 76

  3. Chen X, Wang K, Jin X (2011) Single image based illumination estimation for lighting virtual object in real scene. In: 12th International conference on computer-aided design and computer graphics, CAD/Graphics 2011, Jinan, China, September 15-17, 2011, pp 450–455

  4. Chen X, Jin X, Zhao Q, Wu H (2012) Artistic illumination transfer for portraits. Comput Graph Forum 31(4):1425–1434

    Article  Google Scholar 

  5. Chen X, Jin X, Wang K (2014) Lighting virtual objects in a single image via coarse scene understanding. Sci Chin Inform Sci 57(9):1–14

    Article  Google Scholar 

  6. Chen X, X Jin H W u, Zhao Q (2015) Learning templates for artistic portrait lighting analysis. IEEE Trans Image Process 24(2):608–618

    Article  MathSciNet  MATH  Google Scholar 

  7. Geusebroek J -M, Burghouts GJ, Smeulders AWM (2005) The amsterdam library of object images. Int J Comput Vis 61(1):103–112

    Article  Google Scholar 

  8. Jin X, Zhao M, Chen X, Zhao Q, Zhu S -C (2010) Learning artistic lighting template from portrait photographs. In: Computer Vision - ECCV 2010, 11th European conference on computer vision, Heraklion, Crete, Greece, September 5-11, 2010, Proceedings, Part IV, pp 101–114

  9. Jin X, Tian Y, Liu N, Ye C, Chi J, Li X, Zhao G (2016) Object image relighting through patch match warping and color transfer, 235–241, 09

  10. Jin X, Li Y, Liu N, Li X, Jiang X, Tian Y, Ge S (2017) Scene relighting using a single reference image through material constrained layer decomposition. In: International symposium on artificial intelligence and robotics, Kitakyushu, Japan, 25-26 November, ISAIR 2017. Springer, pp 37–44

  11. Jin X, Li Y, Liu N, Li X, Jiang X, Xiao C, Ge S (2019) Single reference image based scene relighting via material guided filtering. Opt Laser Technol 110:7–12. Special Issue: Optical Imaging for Extreme Environment

    Article  Google Scholar 

  12. Li X, Han R, Ning N, Zhang X, Jin X (2019) Complex objectillumination transfer through semantic and material parsing and composition. In: The 3rd EAI international conference on robotic sensor networks (ROSENET), Kitakyushu, Japan, 17 August

  13. Lu H, Li Y, Min C, Kim H, Serikawa S (2017) Brain intelligence: go beyond artificial intelligence. Mob Netw Appl 23(7553):368–375

    Google Scholar 

  14. Lu H, Li Y, Mu S, Wang D, Kim H, Serikawa S (2018) Motor anomaly detection for unmanned aerial vehicles using reinforcement learning. IEEE Internet Things J 5(4):2315–2322

    Article  Google Scholar 

  15. Lu H, Li Y, Uemura T, Kim H, Serikawa S (2018) Low illumination underwater light field images reconstruction using deep convolutional neural networks. Futur Gener Comput Syst, 82

  16. Lu H, Wang D, Li Y, Li J, Humar I (2019) Conet: a cognitive ocean network

  17. Quan Z, Zheng B, Zhu W, Latecki LJ (2016) Multi-scale context for scene labeling via flexible segmentation graph. Pattern Recogn 59(C):S0031320316300085

    Google Scholar 

  18. Quan Z, Cheng Z, Yu W, Fan Y, Hu Z, Wu X, Ou W, Zhu W, Latecki LJ (2017) Face recognition via fast dense correspondence. Multimed Tools Appl 12:1–19

    Google Scholar 

  19. Quan Z, Yang W, Gao G, Ou W, Lu H, Jie hen, Latecki LJ (2018) Multi-scale deep context convolutional neural networks for semantic segmentation. World Wide Web-internet and Web Information Systems 22(7):1–16

    Google Scholar 

  20. Ren P, Dong Y, Lin S, Tong X, Guo B (2015) Image based relighting using neural networks. ACM Trans Graph 34(4):111,1–111,12

    Article  Google Scholar 

  21. Serikawa S, Lu H (2014) Underwater image dehazing using joint trilateral filter. Comput Electr Eng 40(1):41–50

    Article  Google Scholar 

  22. Shih Y -C, Paris S, Durand F, Freeman WT (2013) Data-driven hallucination of different times of day from a single outdoor photo. ACM Trans Graph 32(6):200:1–200:11

    Article  Google Scholar 

  23. Xu Z, Sunkavalli K, Hadap S, Ramamoorthi R (2018) Deep image-based relighting from optimal sparse samples. ACM Trans Graph 37(4):126:1–126:13

    Article  Google Scholar 

  24. Yu H, He F, Pan Y (2019) A novel segmentation model for medical images with intensity inhomogeneity based on adaptive perturbation. Multimedia Tools Appl 78(9):11779–11798

    Article  Google Scholar 

  25. Yu H, He F, Pan Y (2020) A scalable region-based level set method using adaptive bilateral filter for noisy image segmentation. Multimed Tools Appl 79 (9–10):5743–5765

    Article  Google Scholar 

  26. Zhang J, He F, Chen Y (2020) A new haze removal approach for sky/river alike scenes based on external and internal clues. Multimed Tools Appl 79(3–4):2085–2107

    Article  Google Scholar 

  27. Zhang S, He F, Ren W, Yao J (2020) Joint learning of image detail and transmission map for single image dehazing. Vis Comput 36(2):305–316

    Article  Google Scholar 

  28. Zhou Q, Cheng J, Lu H, Fan Y, Zhang S, Wu X, Zheng B, Ou W, Latecki LJ (2018) Learning adaptive contrast combinations for visual saliency detection. Multimed Tools Appl 1–29:11

    Google Scholar 

Download references

Acknowledgements

Parts of the results and figures presented in this paper have previously appeared in our previous work [12]. We add more technical details and experimental results in this version. This work is partially supported by the National Natural Science Foundation of China (grant numbers 61701008, 61772047), the Open Project Program of State Key Laboratory of Cryptology (grant number MMKFKT201804), the Beijing Natural Science Foundation (grant number 19L2040), the Open Project Program of State Key Laboratory of Virtual Reality Technology and Systems, Beihang University (grant number VRLAB2019C03) and the Fundamental Research Funds for the Central Universities (grant number 328201907).

Author information

Authors and Affiliations

  1. Beijing Electronic Science and Technology Institute, Beijing, 100070, China

    Xin Jin, Ning Ning, Rui Han, Xiaodong Li & Xiaokun Zhang

  2. State Key Laboratory of Cryptology, P.O. Box 5159, Beijing, 100878, China

    Xin Jin

Authors
  1. Xin Jin
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Ning Ning
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Rui Han
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Xiaodong Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Xiaokun Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Xiaodong Li.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Jin, X., Ning, N., Han, R. et al. Complex object relighting via split-then-composition by semantics and materials. Multimed Tools Appl 79, 24185–24197 (2020). https://doi.org/10.1007/s11042-020-09071-6

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-020-09071-6

Keywords

Search

Navigation