Skip to main content
SpringerLink
Log in

Object removal by depth-wise image inpainting

  • Original Paper
  • Published:
Signal, Image and Video Processing Aims and scope Submit manuscript

Abstract

The problem posed in this paper was to fill-up the hole in the digital image left behind after the removal of an object from it. The real challenge is to deal with images of cluttered scenes of objects with complex and infrequent textures. In the past, this problem has been addressed by two classes of algorithms: (i) exemplar-based algorithms to fill the hole using the patches available in a given example and (ii) depth-based algorithms to differentiate the foreground and background before inpainting. This paper presents a novel method that combines advantages of these two approaches. We use depth map of an image to find the order of all the objects in the target image and a database of multi-views of objects to fill the holes. To reduce the size of the database, we propose a method called Keyview Extraction. An object retrieval followed by geometric and photometric registration algorithms is employed to make every object an exact match for inpainting. A number of experiments on both real and synthetic images demonstrate the advantage of depth-wise image inpainting compared with other inpainting methods. In these experiments, we compare three different types of image inpainting methods with our method quantitatively by computing their SSIM values using a ground-truth.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9

Similar content being viewed by others

References

  1. Mirkamali, S.S., Nagabhusha, P.: Depth-wise image inpainting. In: 21th International Conference on Pattern Recognition (ICPR 2012), pp. 141–144. Tsukuba (2012)

  2. Fischler, M.A., Bolles, R.C.: Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography. Commun. ACM 24, 381–395 (1981)

    Article  MathSciNet  Google Scholar 

  3. Lowe, D.: Distinctive image features from scale-invariant keypoints. Int. J. Comput. Vis. 60, 91–110 (2004)

    Article  Google Scholar 

  4. Bertalmio, M., et al.: Image inpainting. In: SIGGRAPH, pp. 417–424 (2000)

  5. Efros, A.A., Leung, T.K.: Texture synthesis by non-parametric sampling. In: The Proceedings of the Seventh IEEE International Conference on Computer Vision (ICCV), pp. 1033–1038 (1999)

  6. Bertalmio, M., et al.: Simultaneous structure and texture image inpainting. IEEE Trans. Image Process. 12, 882–889 (2003)

    Article  MathSciNet  Google Scholar 

  7. Calvetti, D., et al.: Image inpainting with structural bootstrap priors. Image Vis. Comput. 24, 782–793 (2006)

    Article  Google Scholar 

  8. Criminisi, A., et al.: Object removal by exemplar-based inpainting. In: IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR 2003), vol. 2, pp. II-721–II-728 (2003)

  9. Barnes, C., et al.: PatchMatch: a randomized correspondence algorithm for structural image editing. ACM Trans. Graph. TOG 28, 24 (2009)

    Google Scholar 

  10. Guillemot, C., et al.: Image inpainting using LLE-LDNR and linear subspace mappings. In: IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 1558–1562 (2013)

  11. Sun, J., et al.: Image completion with structure propagation. ACM Trans. Graph. (ToG) 24, 861–868 (2005)

    Article  Google Scholar 

  12. Whyte, O., et al.: Get out of my picture! Internet-based inpainting. In: BMVC, pp. 1–11 (2009)

  13. Hays, J., Efros, A.A.: Scene completion using millions of photographs. In: SIGGRAPH, p. 4 (2007)

  14. Pauly, M.: et al.: Example-based 3D scan completion. In: SGP’05, p. 23 (2005)

  15. Ahn, I., Kim, C.: Depth-based disocclusion filling for virtual view synthesis. In: IEEE International Conference on Multimedia and Expo (ICME 2012), pp. 109–114 (2012)

  16. Oh, K.J., et al.: Hole filling method using depth based in-painting for view synthesis in free viewpoint television and 3-d video. In: Picture Coding Symposium, pp. 1–4 (2009)

  17. Comport, A.I., et al.: Real-time markerless tracking for augmented reality: the virtual visual servoing framework. IEEE Trans. Vis. Comput. Graph. 12, 615–628 (2006)

    Article  Google Scholar 

  18. Drummond, T., Cipolla, R.: Real-time visual tracking of complex structures. IEEE Trans. Pattern Anal. Mach. Intell. 24, 932–946 (2002)

    Article  Google Scholar 

  19. Park, Y., et al.: Extended keyframe detection with stable tracking for multiple 3D object tracking. IEEE Trans. Vis. Comput. Graph. 17, 1728–1735 (2011)

    Article  Google Scholar 

  20. Mikolajczyk, K., Schmid, C.: Scale and affine invariant interest point detectors. Int. J. Comput. Vis. 60, 63–86 (2004)

    Article  Google Scholar 

  21. Hartley, R., et al.: Multiple View Geometry in Computer Vision, vol. 2. Cambridge Univ Press, Cambridge (2003)

    Google Scholar 

  22. Wang, Z., et al.: Image quality assessment: from error visibility to structural similarity. IEEE Trans. Image Process. 13, 600–612 (2004)

  23. Zhang, G., et al.: Consistent depth maps recovery from a video sequence. IEEE Trans. Pattern Anal. Mach. Intell. 31, 974–988 (2009)

    Article  Google Scholar 

  24. Sutherland, E.E., et al.: A characterization of ten hidden-surface algorithms. ACM Comput. Surv. 6, 1–55 (1974)

    Article  MATH  Google Scholar 

  25. Sheikh, H.R., Bovik, A.C.: Image information and visual quality. IEEE Trans. Image Process. 15, 430–444 (2006)

    Article  Google Scholar 

Download references

Author information

Author notes

  1. Seyed Saeid Mirkamali

    Present address: Department of Computer, Fakhrodin Asad Institute of Higher Education, Gorgan, Iran

Authors and Affiliations

  1. Department of Studies in Computer Science, University of Mysore, Mysore, India

    Seyed Saeid Mirkamali & P. Nagabhushan

Authors
  1. Seyed Saeid Mirkamali
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. P. Nagabhushan
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Seyed Saeid Mirkamali.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Mirkamali, S.S., Nagabhushan, P. Object removal by depth-wise image inpainting. SIViP 9, 1785–1794 (2015). https://doi.org/10.1007/s11760-014-0673-0

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11760-014-0673-0

Keywords

Search

Navigation