Skip to main content
SpringerLink
Log in

Subject-aware image outpainting

  • Original Paper
  • Published:
Signal, Image and Video Processing Aims and scope Submit manuscript

Abstract

Image outpainting aims at extending the field of an existing image. A current challenge with image outpainting is that background noise may interfere with the expansion of the subject, resulting in visual distortions and artifacts. In this study, a subject-aware image outpainting (SAIO) method, which reduces background pixel interference and emphasizes the subject, is proposed to solve this issue. After training a Matting model as a pre-extractor for the subject, two networks are trained in series: the subject outpainting network (SO-Net) for subject extension and background completion network (BC-Net) for background extension. First, the Matting model is used to simultaneously extract the input image subject and separate the background, and the subject is transferred to SO-Net to generate the predicted subject. Second, the predicted subject is fused with the background separated in the previous step as the input of the second network. Finally, BC-Net outputs the complete image. To improve the training ability of the network and quality of the output image, both networks adopt the conditional training strategy. The qualitative and quantitative results show that our method achieves good performance; a performance of 28.99 in peak signal-to-noise ratio (PSNR) on the test dataset was achieved. The proposed method can be widely applied to intelligent image processing systems.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6

Similar content being viewed by others

Availability of data and materials

All accompanying data are provided in the manuscript.

References

  1. Kopf, J., Kienzle, W., Drucker, S., Kang, S.B.: Quality prediction for image completion. ACM Trans. Graph. 31, 131:1-131:8 (2012). https://doi.org/10.1145/2366145.2366150

    Article  Google Scholar 

  2. Zhang, Y., Xiao, J., Hays, J., Tan, P.: FrameBreak: dramatic image extrapolation by guided shift-maps. In: 2013 IEEE Conference on Computer Vision and Pattern Recognition, pp. 1171–1178 (2013)

  3. Wang, M., Lai, Y.-K., Liang, Y., Martin, R.R., Hu, S.-M.: BiggerPicture: data-driven image extrapolation using graph matching. ACM Trans. Graph. 33, 173:1-173:13 (2014). https://doi.org/10.1145/2661229.2661278

    Article  Google Scholar 

  4. Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., Bengio, Y.: Generative adversarial nets. In: Advances in Neural Information Processing Systems. Curran Associates, Inc. (2014)

  5. Sabini, M., Rusak, G.: Painting outside the box: image outpainting with GANs. arXiv:1808.08483 [cs] (2018)

  6. Yang, Z., Dong, J., Liu, P., Yang, Y., Yan, S.: Very long natural scenery image prediction by outpainting. In: 2019 IEEE/CVF International Conference on Computer Vision (ICCV), pp. 10560–10569 (2019)

  7. Krishnan, D., Teterwak, P., Sarna, A., Maschinot, A., Liu, C., Belanger, D., Freeman, W.: Boundless: generative adversarial networks for image extension. In: 2019 IEEE/CVF International Conference on Computer Vision (ICCV), pp. 10520–10529 (2019)

  8. Sengupta, S., Jayaram, V., Curless, B., Seitz, S.M., Kemelmacher-Shlizerman, I.: Background matting: the world is your green screen. In: 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2288–2297 (2020)

  9. Zhou, Q., Wang, S., Wang, Y., Huang, Z., Wang, X.: Human de-occlusion: invisible perception and recovery for humans. In: 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3690–3700 (2021)

  10. Bowen, R.S., Chang, H., Herrmann, C., Teterwak, P., Liu, C., Zabih, R.: OCONet: image extrapolation by object completion. In: 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2307–2317. IEEE, Nashville (2021)

  11. Miyato, T., Koyama, M.: cGANs with Projection Discriminator. arXiv:1802.05637 [cs, stat] (2018)

  12. Arjovsky, M., Chintala, S., Bottou, L.: Wasserstein generative adversarial networks. In: Proceedings of the 34th International Conference on Machine Learning, pp. 214–223. PMLR (2017)

  13. Sivic, J., Kaneva, B., Torralba, A., Avidan, S., Freeman, W.T.: Creating and exploring a large photorealistic virtual space. In: 2008 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops, pp. 1–8 (2008)

  14. Wang, Y., Tao, X., Shen, X., Jia, J.: Wide-context semantic image extrapolation. In: 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1399–1408 (2019)

  15. Zhang, L., Wang, J., Shi, J.: Multimodal image outpainting with regularized normalized diversification. In: 2020 IEEE Winter Conference on Applications of Computer Vision (WACV), pp. 3422–3431 (2020)

  16. Kim, K., Yun, Y., Kang, K.W., Kong, K., Lee, S., Kang, S.J.: Painting outside as inside: edge guided image outpainting via bidirectional rearrangement with progressive step learning. In: 2021 IEEE Winter Conference on Applications of Computer Vision (WACV) (2021). https://doi.org/10.1109/WACV48630.2021.00217

  17. Wang, Y., Wei, Y., Qian, X., Zhu, Li., Yang, Yi.: Sketch-guided scenery image outpainting. IEEE Trans. Image Process. 30, 2643–2655 (2021). https://doi.org/10.1109/TIP.2021.3054477

    Article  Google Scholar 

  18. Lin, H., Pagnucco, M., Song, Y.: Edge guided progressively generative image outpainting. In: 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pp. 806–815. IEEE, Nashville (2021)

  19. Lu, C.-N., Chang, Y.-C., Chiu, W.-C.: Bridging the visual gap: wide-range image blending. arXiv:2103.15149 [cs] (2021)

  20. Cheng, Y.C., Lin, C.H., Lee, H.Y., Ren, J., Tulyakov, S., Yang, M.H.: In&Out: diverse image outpainting via GAN inversion. arXiv:2104.00675 [cs] (2021)

  21. Li, X., Zhang, H., Feng, L., Hu, J., Zhang, R., Qiao, Q.: Edge‐aware image outpainting with attentional generative adversarial networks. In: IET Image Processing. ipr2.12447 (2022). https://doi.org/10.1049/ipr2.12447

  22. Kong, D., Kong, K., Kim, K., Min, S.J., Kang, S.J.: Image-adaptive hint generation via vision transformer for outpainting. In: 2022 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), pp. 4029–4038. IEEE, Waikoloa (2022)

  23. Yao, K., Gao, P., Yang, X., Huang, K., Sun, J., Zhang, R.: Outpainting by queries. (2022). https://doi.org/10.48550/arXiv.2207.05312

  24. Gao, P., Yang, X., Zhang, R., Huang, K., Geng, Y.: Generalised image outpainting with U-transformer. arXiv:2201.11403 [cs] (2022)

  25. Chang, H., Zhang, H., Jiang, L., Liu, C., Freeman, W.T.: MaskGIT: masked generative image transformer, vol. 11 (2022)

  26. Li, X., Ren, Y., Ren, H., Shi, C., Zhang, X., Wang, L., Mumtaz, I., Wu, X.: Perceptual image outpainting assisted by low-level feature fusion and multi-patch discriminator. Comput, Mater Contin 71, 5021–5037 (2022). https://doi.org/10.32604/cmc.2022.023071

    Article  Google Scholar 

  27. Yang, C.-A., Tan, C.-Y., Fan, W.-C., Yang, C.-F., Wu, M.-L., Wang, Y.-C.F.: Scene graph expansion for semantics-guided image outpainting. arXiv:2205.02958 (2022)

  28. Wei, G., Guo, J., Ke, Y., Wang, K., Yang, S., Sheng, N.: A Three-stage GAN model based on edge and color prediction for image outpainting. Expert Syst. Appl. (2022). https://doi.org/10.1016/j.eswa.2022.119136

  29. Zhang, X., Chen, F., Wang, C., Tao, M., Jiang, G.-P.: SiENet: Siamese expansion network for image extrapolation. IEEE Signal Process. Lett. 27, 1590–1594 (2020). https://doi.org/10.1109/LSP.2020.3019705

    Article  Google Scholar 

  30. Yu, J., Lin, Z., Yang, J., Shen, X., Lu, X., Huang, T.: Free-Form Image Inpainting With Gated Convolution. In: 2019 IEEE/CVF International Conference on Computer Vision (ICCV). pp. 4470–4479 (2019)

  31. Heusel, M., Ramsauer, H., Unterthiner, T., Nessler, B., Hochreiter, S.: GANs trained by a two time-scale update rule converge to a local nash equilibrium. arXiv:1706.08500 [cs, stat] (2018)

Download references

Funding

Received no external funding.

Author information

Authors and Affiliations

  1. Tiangong University, No. 399 Binshui Road, Tianjin, 300387, China

    Yongzhen Ke, Nan Sheng, Gang Wei, Kai Wang & Jing Guo

  2. Tianjin Key Laboratory of Autonomous Intelligence Technology and Systems, Tianjin, 300387, China

    Yongzhen Ke, Nan Sheng, Gang Wei, Kai Wang & Jing Guo

  3. Business School, Nankai University, Tianjin, 300387, China

    Fan Qin

Authors
  1. Yongzhen Ke
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Nan Sheng
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Gang Wei
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Kai Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Fan Qin
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Jing Guo
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

Yongzhen Ke contributed to conceptualization, methodology, supervision, and project administration. Nan Sheng contributed to methodology, software, writing—original draft, and writing—review and editing. Gang Wei contributed to methodology, software, and writing—original draft. Kai Wang contributed to resources, validation, and data curation. Fan Qin contributed to validation and writing—review and editing. Jing Guo contributed to writing—review and editing, formal analysis, and visualization.

Corresponding author

Correspondence to Yongzhen Ke.

Ethics declarations

Conflict of interest

The authors declare no competing interests.

Ethical approval

Not applicable.

Consent for publication

The work described has not been published before. It is not under consideration for publication elsewhere. Its publication has been approved by all co-authors.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Ke, Y., Sheng, N., Wei, G. et al. Subject-aware image outpainting. SIViP 17, 2661–2669 (2023). https://doi.org/10.1007/s11760-022-02444-4

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11760-022-02444-4

Keywords

Search

Navigation