Skip to main content
SpringerLink
Log in

Texture Transfer Attention for Realistic Image Completion

  • Conference paper
  • First Online:
Book cover Intelligent Computing (SAI 2022)

Part of the book series: Lecture Notes in Networks and Systems ((LNNS,volume 507))

Included in the following conference series:

  • 951 Accesses

Abstract

Over the last few years, the performance of inpainting to fill missing regions has shown significant improvements by using deep neural networks. Although the recent inpainting works create visually plausible structure, but insufficient expression of the texture of objects or color distortion make feel heterogeneity. Motivated by these observations, we propose a method for transferring texture patches using skip-connection that Texture Transfer Attention network that better produces the missing region inpainting with fine details. The network is a single refinement network and takes the form of U-Net architecture that transfers fine texture features of encoder to coarse semantic features of decoder through skip-connection. Texture transfer attention is used to create a new reassembled texture map using fine textures and coarse semantics that can efficiently transfer texture information as a result.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 189.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 249.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Barnes, C., Shechtman, E., Finkelstein, A., Goldman, D,B.: Patchmatch: a randomized correspondence algorithm for structural image editing. ACM Trans. Graphics (TOG) 28(3), 24 (2009), Proceedings of SIGGRAPH 2009

    Google Scholar 

  2. Bertalmio, M., Sapiro, G., Caselles, V., Ballester, C.: Image inpainting. In: Proceedings of the 27th Annual Conference On Computer Graphics And Interactive Techniques, pp. 417-424 (2000)

    Google Scholar 

  3. Cimpoi, M., Maji, S., Kokkinos, I., Mohamed, S., Vedaldi, A.: Describing textures in the wild. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3606-3614 (2014)

    Google Scholar 

  4. Criminisi, A., Perez, P., Toyama, K.: Region filling and object removal by exemplar-based image inpainting. IEEE Trans. Image Process. 13(9), 1200–1212 (2004)

    Article  Google Scholar 

  5. Efros, A., Freeman, W.E.: Image quilting for texture synthesis and transfer. In: Proceedings of the 28th Annual Conference on Computer Graphics and Interactive Techniques, SIGGRAPH 2001, pp. 341-346. Association for Computing Machinery (2001)

    Google Scholar 

  6. Efros, A.A., Leung, T.K.: Texture synthesis by non-parametric sampling. In: Proceedings of the International Conference on Computer Vision, ICCV 1999, vol. 2, p. 1033. IEEE Computer Society (1999)

    Google Scholar 

  7. Gatys, L.A., Ecker, A.S., Bethge, M.: Image style transfer using convolutional neural networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, CVPR, pp. 2414-2423 (2016)

    Google Scholar 

  8. Goodfellow, I.J., et al.: Generative adversarial nets. In: Advances in neural information processing systems, pp. 2672-2680 (2014)

    Google Scholar 

  9. Hays, J., Efros, A.A.: Scene completion using millions of photographs. ACM Trans. Graphics (TOG) 26(3) (2007)

    Google Scholar 

  10. Heusel, M., Ramsauer, H., Unterthiner, T., Nessler, B., Hochreiter, S.: Gans trained by a two time-scale update rule converge to a local nash equilibrium. In: NeurIPS, pp. 6626-6637 (2017)

    Google Scholar 

  11. Huang, J.-B., Kang, S.B., Ahuja, N., Kopf, J.: Image completion using planar structure guidance. ACM Trans. Graphics (TOG) 33(4), 129 (2014)

    Google Scholar 

  12. Iizuka, S., Simo-Serra, E., Ishikawa, H.: Globally and locally consistent image completion. ACM Trans. Graphics (TOG) 36(4), 1-14 (2017)

    Google Scholar 

  13. Irani, M., Shechtman, E., Simakov, D., Caspi, Y.: Summarizing visual data using bidirectional similarity. In: 2008 IEEE Conference on Computer Vision and Pattern Recognition, CVPR. pp. 1–8. IEEE Computer Society (2008)

    Google Scholar 

  14. Isola, P., Zhu, J., Zhou, T., Efros, A.A.: Image-to-image translation with conditional adversarial networks. In: Proceedings of the IEEE Conference On Computer Vision and Pattern Recognition, pp. 1125-1134 (2018)

    Google Scholar 

  15. Jaderberg, M., Simonyan, K., Zisserman, A., Kavukcuoglu, K.: Spatial transformer networks. In: Advances in Neural Information Processing Systems, vol. 28. Curran Associates, Inc. (2015)

    Google Scholar 

  16. Johnson, J., Alahi, A., Fei-Fei, L.: Perceptual losses for real-time style transfer and super-resolution. In: European Conference on Computer Vision (2016)

    Google Scholar 

  17. Karras, T., Aila, I., Laine, S., Lehtinen, J.: Progressive growing of GANs for improved quality, stability, and variation. In: International Conference on Learning Representations (2018)

    Google Scholar 

  18. Köhler, R., Schuler, C., Schölkopf, B., Harmeling, S.: Mask-specific inpainting with deep neural networks. In: Jiang, X., Hornegger, J., Koch, R. (eds.) GCPR 2014. LNCS, vol. 8753, pp. 523–534. Springer, Cham (2014). https://doi.org/10.1007/978-3-319-11752-2_43

    Chapter  Google Scholar 

  19. Levin, A., Zomet, A., Peleg, S., Weiss, Y.: Seamless image stitching in the gradient domain. In: Pajdla, T., Matas, J. (eds.) ECCV 2004. LNCS, vol. 3024, pp. 377–389. Springer, Heidelberg (2004). https://doi.org/10.1007/978-3-540-24673-2_31

    Chapter  Google Scholar 

  20. Liu, G., Reda, F.A., Shih, K.J., Wang, T.-C., Tao, A., Catanzaro, B.: Image inpainting for irregular holes using partial convolutions. In: Ferrari, V., Hebert, M., Sminchisescu, C., Weiss, Y. (eds.) ECCV 2018. LNCS, vol. 11215, pp. 89–105. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-01252-6_6

    Chapter  Google Scholar 

  21. Liu, Z., Luo, P., Wang, X., Tang, X.: Deep learning face attributes in the wild. In: Proceedings of International Conference on Computer Vision, ICCV (2015)

    Google Scholar 

  22. Nazeri, K., Ng, E., Joseph, T., Qureshi, F., Ebrahimi, M.: Edgeconnect: structure guided image inpainting using edge prediction. In: The IEEE International Conference on Computer Vision, ICCV Workshops, October 2019

    Google Scholar 

  23. Park, E., Yang, J., Yumer, E., Ceylan, D., Berg, A.C.: Transformation-grounded image generation network for novel 3d view synthesis. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, CVPR, July 2017

    Google Scholar 

  24. Pathak, D., Krahenbuhl, P., Donahue, J., Darrell, T., Efros, A.: Context encoders: feature learning by inpainting. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR, pp. 2536–2544, June 2016

    Google Scholar 

  25. Ronneberger, O., Fischer, P., Brox, T.: U-Net: convolutional networks for biomedical image segmentation. In: Navab, N., Hornegger, J., Wells, W.M., Frangi, A.F. (eds.) MICCAI 2015. LNCS, vol. 9351, pp. 234–241. Springer, Cham (2015). https://doi.org/10.1007/978-3-319-24574-4_28

    Chapter  Google Scholar 

  26. Wang, Z., Simoncelli, E.P., Bovik., A.C.: Multiscale structural similarity for image quality assessment. In: ACSSC, vol. 2, pp. 1398-1402 (2003)

    Google Scholar 

  27. Yang, C., Lu, X., Lin, Z., Shechtman, E., Wang, O., Li, H.: High-resolution image inpainting using multi-scale neural patch synthesis. In: CVPR, pp. 4076–4084 (2017)

    Google Scholar 

  28. Yang, F., Yang, H., Fu, J., Lu, H., Guo, B.: Learning texture transformer network for image super-resolution. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, CVPR, pp. 5791–5798 (2019)

    Google Scholar 

  29. Yu, F., Koltun, V.: Multi-scale context aggregation by dilated convolutions. In: ICLR (2016)

    Google Scholar 

  30. Yu, J., Lin, Z., Yang, J., Shen, X., Lu, X., Huang, T.S.: Generative image inpainting with contextual attention. In: CVPR, pp. 5505-5514 (2018)

    Google Scholar 

  31. Yu, J., Lin, Z., Yang, J., Shen, X., Lu, X., Huang, T.S.: Image style transfer using convolutional neural networks. In: Free-Form Image Inpainting With Gated Convolution (2018)

    Google Scholar 

  32. Zeng, Y., Fu, J., Chao, H., Guo, B.: Learning pyramid-context encoder network for high-quality image inpainting. In: The IEEE Conference on Computer Vision and Pattern Recognition, CVPR, pp. 1486–1494 (2019)

    Google Scholar 

  33. Zhang, R., Isola, P., Efros, A.A., Shecht-man, E., Wang, O.: The unreasonable effectiveness of deep features as a perceptual metric. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (2018)

    Google Scholar 

  34. Zhang, Z., Wang, Z., Lin, Z., Qi, H.: Image super-resolution by neural texture transfer. In: CVPR, pp. 7982-7991 (2019)

    Google Scholar 

  35. Zheng, H., Ji, M., Wang, H., Liu, Y., Fang, L.: CrossNet: an end-to-end reference-based super resolution network using cross-scale warping. In: Ferrari, V., Hebert, M., Sminchisescu, C., Weiss, Y. (eds.) ECCV 2018. LNCS, vol. 11210, pp. 87–104. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-01231-1_6

Download references

Author information

Authors and Affiliations

  1. LG Electronics, Seoul, Korea

    Yejin Kim, Manri Cheon & Junwoo Lee

Authors
  1. Yejin Kim
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Manri Cheon
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Junwoo Lee
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Junwoo Lee .

Editor information

Editors and Affiliations

  1. Saga University, Saga, Japan

    Kohei Arai

Rights and permissions

Reprints and permissions

Copyright information

© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Kim, Y., Cheon, M., Lee, J. (2022). Texture Transfer Attention for Realistic Image Completion. In: Arai, K. (eds) Intelligent Computing. SAI 2022. Lecture Notes in Networks and Systems, vol 507. Springer, Cham. https://doi.org/10.1007/978-3-031-10464-0_22

Download citation

Publish with us

Policies and ethics

Search

Navigation