Skip to main content
Springer Nature Link
Log in

An objective method to evaluate exemplar‐based inpainted images quality using Jaccard index

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

Objective evaluation of images is one of the most essential and practical aspects of image inpainting. The existing objective evaluation methods of image inpainting are functional only on an individual basis and do not provide an accurate and useful objective evaluation of inpainted images. Currently, there is no objective measure for evaluating inpainted images. In this study, an objective evaluation method was developed for image inpainting. In the proposed method, first, 100 images were inpainted using an exemplar-based algorithm. Then, the saliency map and its complementary region in the original image were obtained and a new objective measure was proposed for the evaluation of inpainted images based on the saliency map features. To make the assessment more realistic and comparable to human judgments, two terms, namely penalty and compensation, were taken into account. To assess the performance of our proposed objective measure, the inpainted images were also evaluated using a subjective test. The experiments demonstrates that the proposed objective measure correlated with the qualitative opinion in a human observer study. Finally, the objective measure was compared against three other measures, and the results showed that our proposed objective measure performed better than the other evaluation measures.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6

Similar content being viewed by others

References

  1. Alpert S, Galun M, Brandt A, Basri R (2011) Image segmentation by probabilistic bottom-up aggregation and cue integration. IEEE Trans Pattern Anal Mach Intell 34(2):315–327

    Article  Google Scholar 

  2. Ardis P, Singhal A (2009) Visual salience metrics for image inpainting visual salience metrics for image inpainting. Proc. SPIE/IS&T Electron. Imaging, San Jose, CA, USA, vol 7257

  3. Arias P, Caselles V, Facciolo G (2012) Analysis of a variational framework for exemplar-based image inpainting. Multiscale Model Simul 10(2):473–514

    Article  MathSciNet  Google Scholar 

  4. Atapour-Abarghouei A, Breckon TP (2018) Extended patch prioritization for depth filling within constrained exemplar-based RGB-D image completion,. In: International Conference Image Analysis and Recognition, pp 306–314

  5. Barbu T, Marinoschi G, Moroșanu C, Munteanu I (2018) Advances in variational and partial differential equation-based models for image processing and computer vision. Math Probl Eng 2018

  6. Bertalmio M, Sapiro G, Caselles, Balleste C (2000) Image inpainting. New Orleans Proc. SIGGRAPH, pp 102–133

  7. Bertalmio M, Vese L, Sapiro G, Osher S (2003) Simultaneous structure and texture image inpainting. IEEE Trans Image Process 12(8):882–889

    Article  Google Scholar 

  8. Bredies K, Lorenz D (2018) Partial differential equations in image processing. In: Mathematical Image Processing. Springer, Berlin, pp 171–250

  9. Brkić AL, Novak A (2018) A nonlocal image inpainting problem using the linear allen–cahn equation. In: Conference on Non-integer Order Calculus and Its Applications, pp 229–239

  10. Bruce N, Tsotsos J (2006) Saliency based on information maximization. Adv Neural Inf Process Syst 18:155

    Google Scholar 

  11. Cao F, Gousseau Y, Masnou S, Pérez P (2011) Geometrically guided exemplar-based inpainting. SIAM J Imaging Sci 4(4):1143–1179

    Article  MathSciNet  Google Scholar 

  12. Criminisi A, Pérez P, Toyama K (2004) Region filling and object removal by exemplar-based image inpainting. IEEE Trans image Process 13(9):1200–1212

    Article  Google Scholar 

  13. Demanet L, Song B, Chan T (2003) Image inpainting by correspondence maps: a deterministic approach. Appl Comput Math 1100:217–250

    Google Scholar 

  14. Everingham M, Van Gool L, Williams CKI, Winn J, Zisserman A. The PASCAL visual object classes challenge, 2012 (VOC2012) results. [Online]. Available http://www.pascalnetwork.org/challenges/VOC/voc2012/workshop/index.html2012/workshop/index.html. Accessed May 2013

  15. Fan Q, Zhang L (2018) A novel patch matching algorithm for exemplar-based image inpainting. Multimed Tools Appl 77(9):10807–10821

    Article  Google Scholar 

  16. Harel J, Koch C, Perona P (2006) Graph-based visual saliency. In: NIPS, vol 1, no 2, p 5

  17. Hou X, Zhang L (2007) Saliency detection: A spectral residual approach. In: Computer Vision and Pattern Recognition, 2007. CVPR’07. IEEE Conference on, pp 1–8

  18. ITU (2002) Methodology for the subjective assessment of the quality of television pictures. Itu-R Bt.500 – 11, vol 211, no BT.500 – 11, pp 1–48

  19. Itti L, Koch C, Niebur E (1998) A model of saliency-based visual attention for rapid scene analysis. IEEE Trans Pattern Anal Mach Intell 20(11):1254–1259

    Article  Google Scholar 

  20. Jia J, Tang C-K (2004) Inference of segmented color and texture description by tensor voting. IEEE Trans Pattern Anal Mach Intell 26(6):771–786

    Article  Google Scholar 

  21. Koch C, Ullman S (1987) Shifts in selective visual attention: towards the underlying neural circuitry. In: Matters of intelligence. Springer, Berlin, pp 115–141

  22. Komodakis N, Tziritas G (2007) Image completion using efficient belief propagation via priority scheduling and dynamic pruning. IEEE Trans Image Process 16(11):2649–2661

    Article  MathSciNet  Google Scholar 

  23. Kumar BVR, Halim A (2019) A linear fourth-order PDE-based gray-scale image inpainting model. Comput Appl Math 38(1):6

    Article  MathSciNet  Google Scholar 

  24. Kumar H, Sharma S, Choudhury T (2018) Improved exemplar-based image inpainting approach. In: Intelligent Computing and Information and Communication. Springer, Berlin, pp 705–717

  25. Levin A, Zomet A, Weiss Y (2003) Learning how to inpaint from global image statistics. In: ICCV, vol 1, pp 305–312

  26. Li S, Mak LC-M, Ngan KN (2011) Visual quality evaluation for images and videos. In: Multimedia Analysis, Processing and Communications, Springer, Berlin, pp 497–544

  27. Li S, Zhang F, Ma L, Ngan KN (2011) Image quality assessment by separately evaluating detail losses and additive impairments. IEEE Trans Multimed 13(5):935–949

    Article  Google Scholar 

  28. Liu T et al (2011) Learning to detect a salient object. IEEE Trans Pattern Anal Mach Intell 33(2):353–367

    Article  Google Scholar 

  29. Ma L, Lin W, Deng C, Ngan KN (2012) Image retargeting quality assessment: A study of subjective scores and objective metrics. IEEE J Sel Top Signal Process 6(6):626–639

    Article  Google Scholar 

  30. Mahalingam VV (2010) Digital inpainting algorithms and evaluation,. PhD thesis, Univ. Kentucky

  31. Martin D, Fowlkes C, Tal D, Malik J (2001) A database of humansegmented naturalimages and its application to evaluating segmentationalgorithms and measuringecological statistics. In: Proc. 8th IEEE Int. Conf. Comput. Vis.(ICCV), vol 2, pp 416–423

  32. Masnou S (2002) Disocclusion: a variational approach using level lines. IEEE Trans Image Process 11(2):68–76

    Article  MathSciNet  Google Scholar 

  33. Newson A, Almansa A, Fradet M, Gousseau Y, Pérez P (2013) Towards fast, generic video inpainting. In: Proceedings of the 10th European Conference on Visual Media Production, p 7

  34. Qureshi MA, Deriche M, Beghdadi A, Amin A (2017) A critical survey of state-of-the-art image inpainting quality assessment metrics. J Vis Commun Image Represent 49:177–191

    Article  Google Scholar 

  35. Ribeiro F, Florêncio D, Zhang C, Seltzer M (2011) Crowdmos: An approach for crowdsourcing mean opinion score studies. In: Acoustics, Speech and Signal Processing (ICASSP), (2011) IEEE International Conference on, pp 2416–2419

  36. Roth S, Black MJ (2005) Fields of experts: A framework for learning image priors. In: Computer Vision and Pattern Recognition, 2005. CVPR 2005. IEEE Computer Society Conference on, vol 2, pp 860–867

  37. Rother C, Kolmogorov V, Blake A (2004) GrabCut’: Interactive foreground extraction using iterated graph cuts. ACM Trans. Graph, vol 23, no 3, pp 309–314

  38. Seshadrinathan K, Soundararajan R, Bovik AC, Cormack LK (2010) Study of subjective and objective quality assessment of video. IEEE Trans Image Process 19(6):1427–1441

    Article  MathSciNet  Google Scholar 

  39. Shen J, Chan TF (2002) Mathematical models for local nontexture inpaintings. SIAM J Appl Math 62(3):1019–1043

    Article  MathSciNet  Google Scholar 

  40. Shi R, Ngan KN, Li S, Paramesran R, Li H (2015) Visual quality evaluation of image object segmentation: subjective assessment and objective measure. IEEE Trans Image Process 24(12):5033–5045

    Article  MathSciNet  Google Scholar 

  41. Shi R, Ngan KN, Li S. Image object segmentation visual quality evaluation database. [Online]. Available: http://www.ee.cuhk.edu.hk/~rshi/. Accessed Jan 2015

  42. Trampert P et al (2018) Exemplar-based inpainting as a solution to the missing wedge problem in electron tomography. Ultramicroscopy 191:1–10

    Article  Google Scholar 

  43. Tschumperle D, Deriche R (2005) Vector-valued image regularization with PDEs: A common framework for different applications. IEEE Trans Pattern Anal Mach Intell 27(4):506–517

    Article  Google Scholar 

  44. Underwood G, Foulsham T, van Loon E, Humphreys L, Bloyce J (2006) Eye movements during scene inspection: A test of the saliency map hypothesis. Eur J Cogn Psychol 18(3):321–342

    Article  Google Scholar 

  45. Van Dijk AM, Martens J-B, Watson AB (1995) Quality asessment of coded images using numerical category scaling. In: Advanced Networks and Services, pp 90–101

  46. Viacheslav V, Vladimir F, Vladimir M, Nikolay G, Roman S, Valentin F (2014) Low-level features for inpainting quality assessment. Int. Conf. Signal Process. Proceedings, ICSP, vol 2015–Janua, no. October, pp 643–647

  47. Voronin V, Marchuk V, Semenishchev E, Maslennikov S, Svirin I (2015) Inpainted image quality assessment based on machine learning

  48. Walther D (2006) Interactions of visual attention and object recognition: computational modeling, algorithms, and psychophysics. Thesis by, PhD thesis, Calif. Inst. Technol. Pasadena, CA

  49. Wei W, Zhou B, Połap D, Woźniak M (2019) A regional adaptive variational PDE model for computed tomography image reconstruction. Pattern Recognit 92:64–81

    Article  Google Scholar 

  50. Wexler Y, Shechtman E, Irani M (2007) Space-time completion of video. IEEE Trans Pattern Anal Mach Intell 29(3):463–476

    Article  Google Scholar 

  51. Xiang S, Deng H, Zhu L, Wu J, Yu L (2019) Exemplar-based depth inpainting with arbitrary-shape patches and cross-modal matching. Signal Process Image Commun 71:56–65

    Article  Google Scholar 

  52. Xiao M, Li G, Xie L, Peng L, Chen Q (2018) Exemplar-based image completion using image depth information. PLoS One 13(9):e0200404

    Article  Google Scholar 

  53. Xu Z, Sun J (2010) Image inpainting by patch propagation using patch sparsity. IEEE Trans Image Process 19(5):1153–1165

    Article  MathSciNet  Google Scholar 

  54. Yang X, Guo B, Xiao Z, Liang W (2019) Improved structure tensor for fine-grained texture inpainting. Signal Process Image Commun 73:84–95

    Article  Google Scholar 

  55. Yao F (2018) Damaged region filling by improved criminisi image inpainting algorithm for thangka. Cluster Comput :1–9

  56. Zhang J, Sclaroff S (2013) Saliency detection: A boolean map approach. In: Proceedings of the IEEE International Conference on Computer Vision, pp 153–160

  57. Zhang D, Liang Z, Yang G, Li Q, Li L, Sun X (2018) A robust forgery detection algorithm for object removal by exemplar-based image inpainting. Multimed Tools Appl 77(10):11823–11842

    Article  Google Scholar 

  58. Zhang N, Ji H, Liu L, Wang G (2019) Exemplar-based image inpainting using angle-aware patch matching. EURASIP J Image Video Process, vol 2019, no 1, p 70

Download references

Author information

Authors and Affiliations

  1. Department of Engineering and Media, Islamic Republic of Iran Broadcast University, Tehran, Iran

    Dariush Amirkhani

  2. Department of MechatronicEngineering, Islamic Azad University of Karaj, Karaj, Iran

    Azam Bastanfard

Authors
  1. Dariush Amirkhani
    View author publications

    You can also search for this author inPubMed Google Scholar

  2. Azam Bastanfard
    View author publications

    You can also search for this author inPubMed Google Scholar

Corresponding author

Correspondence to Dariush Amirkhani.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Amirkhani, D., Bastanfard, A. An objective method to evaluate exemplar‐based inpainted images quality using Jaccard index. Multimed Tools Appl 80, 26199–26212 (2021). https://doi.org/10.1007/s11042-021-10883-3

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-021-10883-3

Keywords