Skip to main content
SpringerLink
Log in

The improved image inpainting algorithm via encoder and similarity constraint

  • Original article
  • Published:
The Visual Computer Aims and scope Submit manuscript

Abstract

Existing image inpainting algorithms based on neural network models are affected by structural distortions and blurred textures on visible connectivity. As a result, overfitting and overlearning phenomena can easily emerge during the image inpainting procedure. Image inpainting refers to the repairing of missing parts of an image, given an image that is broken or incomplete. After the repairing operation is complete, there are obvious signs of repair in damaged areas, semantic discontinuities, and unclearness. This paper proposes an improved image inpainting method based on a new encoder combined with a context loss function. In order to obtain clear repaired images and ensure that the semantic features of images are fully learned, a generative network based on the fusion model of squeeze-and-excitation networks deep residual learning has been proposed to improve the application of network features in order to obtain clear images and reduce network parameters. At the same time, a discriminative network based on the squeeze-and-excitation residual Network has been proposed to strengthen the capability of the discriminative network. In order to make the generated image more realistic, so that the restored image will be more similar to the original image, a joint context-awareness loss training method (contextual perception loss network) has also been proposed to generate the similarity of the local features of the network constraint, with the result that the repaired image is closer to the original picture and more realistic. The experimental results can demonstrate that the proposed algorithm demonstrates better adaptive capability than the comparison algorithms on a number of image categories. In addition, the processing results of the image inpainting procedure were also superior to those of five state-of-the-art algorithms.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12

Similar content being viewed by others

References

  1. Barnes, C., Shechtman, E., Finkelstein, A., Goldman, D.: Patchmatch: a randomized correspondence algorithm for structural image editing. ACM Trans. Graph. 28(3), 24–34 (2009)

    Article  Google Scholar 

  2. Chen, Y., Xiong, J., Xu, W., Zuo, J.: A novel online incremental and decremental learning algorithm based on variable support vector machine. Cluster Comput. 22, 7435–7445 (2019)

    Article  Google Scholar 

  3. Wang, C., Chan, S., Zhu, Z., Zhang, L., Shum, H.: Superpixel-based color-depth restoration and dynamic environment modeling for Kinect-assisted image-based rendering systems. Vis. Comput. 34, 67–81 (2018)

    Article  Google Scholar 

  4. Haouchine, N., Roy, F., Courtecuisse, H., Niebner, M., Cotidn, S.: Calipso: physics-based image and video editing through CAD model proxies. Vis. Comput. 36, 211–226 (2020)

    Article  Google Scholar 

  5. Liu, B., Li, P., Sheng, B., Nie, Y., Wu, E.: Structure-preserving image completion with multi-level dynamic patches. Vis. Comput. 35, 85–98 (2019)

    Article  Google Scholar 

  6. Zhao, H., Rosin, P., Lai, Y., Wang, Y.: Automatic semantic style transfer using deep convolutional neural networks and soft masks. Vis. Comput. 36, 1307–1324 (2020)

    Article  Google Scholar 

  7. Aharon, M., Elad, M., Bruckstein, A.: K-SVD: an algorithm for designing overcomplete dictionaries for sparse representation. IEEE Trans. Signal Process. 54(11), 4311–4322 (2006)

    Article  Google Scholar 

  8. Zhang, J., Wu, Y., Feng, W., Wang, J.: Spatially attentive visual tracking using multi-model adaptive response fusion. IEEE Access 7, 83873–83887 (2019)

    Article  Google Scholar 

  9. Chen, Y., Xu, W., Zuo, J., Yang, K.: The fire recognition algorithm using dynamic feature fusion and IV-SVM classifier. Cluster Comput. 22, 7665–7675 (2019)

    Article  Google Scholar 

  10. Wang, J., Qin, J., Xiang, X., Tan, Y., Pan, N.: CAPTCHA recognition based on deep convolutional neural network. Math. Biosci. Eng. 16(5), 5851–5861 (2019)

    Article  MathSciNet  Google Scholar 

  11. Chen, Y., Tao, J., Liu, L., Xiong, J., Xia, R., Xie, J., Zhang, Q., Yang, K.: Research of improving semantic image segmentation based on a feature fusion model. J. Ambient Intell. Humaniz. Comput. (2020). https://doi.org/10.1007/s12652-020-02066-z

    Article  Google Scholar 

  12. Zhang, J., Zhong, S., Wang, T., Chao, H., Wang, J.: Blockchain-based systems and applications: a survey. J. Internet Technol. 21(1), 1–14 (2020)

    Google Scholar 

  13. Altantawy, D., Saleh, A., Kishk, S.: Texture-guided depth upsampling using Bregman split: a clustering graph-based approach. Vis. Comput. 36, 333–359 (2020)

    Article  Google Scholar 

  14. Yang, C., Feng, H., Xu, Z., Li, Q., Chen, Y.: Correction of overexposure utilizing haze removal model and image fusion technique. Vis. Comput. 35, 695–705 (2019)

    Article  Google Scholar 

  15. Liu, Y., Pan, J., Su, Z., Tang, K.: Robust dense correspondence using deep convolutional features. Vis. Comput. 36, 827–841 (2020)

    Article  Google Scholar 

  16. Yin, B., We, X., Wang, J., Xiong, N., Gu, K.: An industrial dynamic skyline based similarity joins for multi-dimensional big data applications. IEEE Trans. Ind. Inform. 16(4), 2520–2532 (2020)

    Article  Google Scholar 

  17. Beckouche, S., Starck, J., Fadili, J.: Astronomical Image Denoising Using Dictionary Learning. (2013). arXiv arXiv:1304.3573

  18. Hu, G., Ling, X.: Criminisi-based sparse representation for image inpainting. In: Proceedings of IEEE International Conference on Multimedia Big Data, Laguna Hills, CA, USA, 19–21 April 2017, pp. 389–393 (2017)

  19. Darabi, S., Shechtman, E., Barnes, C., Goldman, D., Sen, P.: Image melding: combining inconsistent images using patch-based synthesis. ACM Trans. Graph. 31(4), 1–10 (2012)

    Article  Google Scholar 

  20. Pathak, D., Krahenbuhl, P., Donahue, J., Darrell, T., Efros, A.: Context encoders: feature learning by inpainting. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA, 27–30 June 2016, pp. 2536–2544 (2016)

  21. Yu, J., Zhe, L., Yang, J., Shen, X., Xin, L., Huang, T.: Generative image inpainting with contextual attention. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA, 18–22 June 2018, pp. 5505–5014 (2018)

  22. Luo, Y., Qin, J., Xiang, X., Tan, Y., Liu, Q., Xiang, L.: Coverless real-time image information hiding based on image block matching and Dense convolutional network. J. Real-Time Image Process. 17(1), 125–135 (2020)

    Article  Google Scholar 

  23. Naderahmadian, T., Beheshti, S., Ali, M.: Correlation based online dictionary learning algorithm. IEEE Trans. Signal Process. 64(3), 592–602 (2015)

    Article  MathSciNet  Google Scholar 

  24. Yang, H., Zhang, Z.: Depth image upsampling based on guided filter with low gradient minimization. Vis. Comput. 36, 1411–1422 (2020)

    Article  Google Scholar 

  25. Li, W., Xu, H., Li, H., Yang, Y., Sharma, P., Wang, J., Singh, S.: Complexity and algorithms for superposed data uploading problem in networks with smart devices. IEEE Internet Things J. (2019). https://doi.org/10.1109/JIOT.2019.2949352

    Article  Google Scholar 

  26. HaCohen, Y., Fattal, R., Lischinski, D.: Image upsampling via texture hallucination. In: Proceedings of International Conference on Computational Photography, Cambridge, MA, USA, 23 September 2010, pp. 1–8 (2010)

  27. Liao, N., Song, Y., Huang, X., Wang, J.: Detection of probe flow anomalies using information entropy. J. Intell. Fuzzy. Syst. (2020). https://doi.org/10.3233/IFS-191448

    Article  Google Scholar 

  28. Yu, F., Liu, L., Qian, S., Li, L., Huang, Y., Shi, C., Cai, S., Wu, X., Du, S., Wan, Q.: Chaos-based application of a novel multistable 5D memristive hyperchaotic system with coexisting multiple attractors. Complexity 2020, Article ID 8034196 (2020)

  29. Sheng, G., Tang, X., Xie, K., Xiong, J.: Hydraulic fracturing microseismic first arrival picking method based on non-subsampled shearlet transform and higher-order-statistics. J. Seism. Explor. 28(6), 593–618 (2019)

    Google Scholar 

  30. Gu, K., Zhang, W., Lim, S., Sharma, P., Al-Makhadmeh, Z., Tolba, A.: Reusable mesh signature scheme for protecting identity privacy of IoT devices. Sensors 20, 758 (2020)

    Article  Google Scholar 

  31. Liu, Y., Cheng, M., Fan, D., Zhang, L., Bian, J., Tao, D.: Semantic edge detection with diverse deep supervision (2018). arXiv arXiv:1804.02864v3

  32. Liu, Y., Li, S., Cheng, M.: Refinedbox: refining for fewer and high-quality object proposals. Neurocomputing (2020). https://doi.org/10.1016/j.neucom.2020.04.017

    Article  Google Scholar 

  33. Iizuka, S., Simo-Serra, E., Ishikawa, H.: Globally and locally consistent image completion. ACM Trans. Graph. 36(4), 107:1–107:14 (2017)

    Article  Google Scholar 

  34. Yu, F., Liu, L., Shen, H., Zhang, Z., Huang, Y., Shi, C., Cai, S., Wu, X., Du, S., Wan, Q.: Dynamic analysis, circuit design and synchronization of a novel 6D memristive four-wing hyperchaotic system with multiple coexisting attractors. Complexity 2020, Article ID 5904607 (2020)

  35. Chen, Y., Wang, J., Liu, S., Chen, X., Xiong, J., Xie, J., Yang, K.: Multiscale fast correlation filtering tracking algorithm based on a feature fusion model. Concurr. Comput. Pract. Exp. (2019). https://doi.org/10.1002/cpe.5533

    Article  Google Scholar 

  36. Liao, Z., Peng, J., Chen, Y., Zhang, J., Wang, J.: A fast Q-learning based data storage optimization for low latency in data center networks. IEEE Access 8, 90630–90639 (2020)

    Article  Google Scholar 

  37. Mikaeli, E., Aghagolzadeh, A., Azghani, M.: Single-image super-resolution via patch-based and group-based local smoothness modeling. Vis. Comput. (2019). https://doi.org/10.1007/s00371-019-01756-w

    Article  Google Scholar 

  38. Nie, G., Cheng, M., Liu, Y., Liang, Z., Fan, D., Liu, Y., Wang, Y.: Multi-level context ultra-aggregation for stereo matching. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, Long Beach, CA, USA, 16–20 June 2019, pp. 3283–3291 (2019)

  39. Pan, N., Qin, J., Tan, Y., Xiang, X., Hou, G.: A video coverless information hiding algorithm based on semantic segmentation. EURASIP J. Image Video Process. (2020). https://doi.org/10.1186/s13640-020-00512-8

    Article  Google Scholar 

  40. Hu, J., Shen, L., Sun, G.: Squeeze-and-excitation networks. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA, 18–22 June 2018, pp. 7132–7141 (2018)

  41. He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA, 27–30 June 2016, pp. 770–778 (2016)

  42. Yang, C., Yang, M.: Fast direct super-resolution by simple functions. In: Proceedings of IEEE International Conference on Computer Vision, Sydney, Australia, 1–8 December 2013, pp. 561–568 (2013)

  43. Hore, A., Ziou, D.: Image quality metrics: PSNR vs. SSIM. In: Proceedings of International Conference on Pattern Recognition, Istanbul, Turkey, 23–26 August 2010, pp. 2366–2369 (2010)

  44. Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., Bengio, Y.: Generative adversarial nets. In: Proceedings of Annual Conference on Neural Information Processing System, Montreal, Quebec, Canada, 7–12 December 2015, pp. 5672–2680 (2015)

  45. Chen, Y., Wang, J., Xia, R., Zhang, Q., Cao, Z., Yang, K.: The visual object tracking algorithm research based on adaptive combination kernel. J. Ambient Intell. Humaniz. Comput. 10(12), 4855–4867 (2019)

    Article  Google Scholar 

  46. Yu, F., Shen, H., Liu, L., Zhang, Z., Huang, Y., He, B., Cai, S., Song, Y., Yin, B., Du, S., Xu, Q.: CCII and FPGA realization: a multistable modified four-order autonomous Chua’s chaotic system with coexisting multiple attractors. Complexity 2020, Article ID 5212601 (2020)

  47. Zhou, L., Zhang, T., Tian, Y., Huang, H.: Fraction-order total variation image blind restoration based on self-similarity features. IEEE Access 8, 30346–30444 (2020)

    Google Scholar 

  48. Zhang, J., Xie, Z., Sun, J., Zou, X., Wang, J.: A cascaded R-CNN with multiscale attention and imbalanced samples for traffic sign detection. IEEE Access 8, 29742–29754 (2020)

    Article  Google Scholar 

  49. Li, Y., Liu, S., Yang, J., Yang, M.: Generative face completion. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA, 21–26 July 2017, pp. 5892–5900 (2017)

  50. Yeh, A., Chen, C., Lim, T., Schwing, A., Hasegawa-Johnson, M.: Do M: Semantic image inpainting with deep generative models. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA, 21–26 July 2017, pp. 6882–6890 (2017)

  51. Sheikh, H., Bovik, A., Veciana, G.: An information fidelity criterion for image quality assessment using natural scene statistics. IEEE Trans. Image Process. 14(12), 2117–2128 (2005)

    Article  Google Scholar 

  52. Zhang, J., Wang, W., Lu, C., Wang, J., Sangaiah, A.: Lightweight deep network for traffic sign classification. Ann. Telecommun. (2019). https://doi.org/10.1007/s12243-019-00731-9

    Article  Google Scholar 

  53. Chen, Y., Wang, J., Chen, X., Zhu, M., Yang, K., Wang, Z., Xia, R.: Single-image super-resolution algorithm based on structural self-similarity and deformation block features. IEEE Access 7, 58791–58801 (2019)

    Article  Google Scholar 

  54. Chen, Y., Tao, J., Zhang, Q., Yang, K., Chen, X., Xiong, J., Xia, R., Xie, J.: Saliency detection via improved hierarchical principle component analysis method. Wirel. Commun. Mob. Comput. 2020, Article ID 8822777 (2020)

  55. Cheng, M., Liu, X., Wang, J., Lu, S.: Structure-preserving neural style transfer. IEEE Trans. Image Process. 29, 909–920 (2019)

    Article  MathSciNet  Google Scholar 

Download references

Acknowledgements

We would like to thank Jin Wang, Jianming Zhang, Ke Gu, Fei Yu, and Shuo Cai for their help in this research. We are grateful to anonymous referees for useful comments and suggestions.

Funding

This study was funded by the National Natural Science Foundation of China [61972056, 61772454, 61402053, 61981340416], the Natural Science Foundation of Hunan Province of China [2020JJ4623], the Scientific Research Fund of Hunan Provincial Education Department [17A007, 19C0028, 19B005], the Changsha Science and Technology Planning [KQ1703018, KQ1706064, KQ1703018-01, KQ1703018-04], the Junior Faculty Development Program Project of Changsha University of Science and Technology [2019QJCZ011], the “Double First-class” International Cooperation and Development Scientific Research Project of Changsha University of Science and Technology [2019IC34], the Practical Innovation and Entrepreneurship Ability Improvement Plan for Professional Degree Postgraduate of Changsha University of Science and Technology [SJCX202072], and the Postgraduate Training Innovation Base Construction Project of Hunan Province [2019-248-51].

Author information

Authors and Affiliations

  1. School of Computer and Communication Engineering and Hunan Provincial Key Laboratory of Intelligent Processing of Big Data on Transportation, Changsha University of Science and Technology, Changsha, 410114, Hunan, China

    Yuantao Chen, Linwu Liu, Jiajun Tao & Xi Chen

  2. Hunan Institute of Scientific and Technical Information, Changsha, 411105, Hunan, China

    Runlong Xia

  3. Department of Electronic Products, Hunan ZOOMLION Intelligent Technology Corporation Limited, Changsha, 410005, Hunan, China

    Qian Zhang & Kai Yang

  4. Electronics and Information School, Yangtze University, Jingzhou, 434023, China

    Jie Xiong

Authors
  1. Yuantao Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Linwu Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Jiajun Tao
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Runlong Xia
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Qian Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Kai Yang
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Jie Xiong
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. Xi Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Yuantao Chen.

Ethics declarations

Conflict of interest

The authors declare that they have no conflict of interest.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Chen, Y., Liu, L., Tao, J. et al. The improved image inpainting algorithm via encoder and similarity constraint. Vis Comput 37, 1691–1705 (2021). https://doi.org/10.1007/s00371-020-01932-3

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00371-020-01932-3

Keywords

Search

Navigation