research-article

RGBD temporal resampling for real-time occlusion removal

Authors:

Voicu PopescuAuthors Info & Claims

I3D '19: Proceedings of the ACM SIGGRAPH Symposium on Interactive 3D Graphics and Games

Article No.: 7, Pages 1 - 9

https://doi.org/10.1145/3306131.3317025

Published: 21 May 2019 Publication History

Abstract

Occlusions disrupt the visualization of an object of interest, or target, in a real world scene. Video inpainting removes occlusions from a video stream by cutting out occluders and filling in with a plausible visualization of the object, but the approach is too slow for real-time performance. In this paper, we present a method for realtime occlusion removal in the visualization of a real world scene that is captured with an RGBD stream. Our pipeline segments the current RGBD frame to find the target and the occluders, searches for the best matching disoccluded view of the target in an earlier frame, computes a mapping between the target in the current frame and the target in the best matching frame, inpaints the missing pixels of the target in the current frame by resampling from the earlier frame, and visualizes the disoccluded target in the current frame. We demonstrate our method in the case of a walking human occluded by stationary or walking humans. Our method does not rely on a known 2D or 3D model of the target or of the occluders, and therefore it generalizes to other shapes. Our method runs at an interactive frame rate of 30fps.

Supplementary Material

ZIP File (a7-wu.zip)

Supplemental material.

Download
55.12 MB

References

[1]

Connelly Barnes, Eli Shechtman, Adam Finkelstein, and Dan B Goldman. 2009. Patch-Match: A Randomized Correspondence Algorithm for Structural Image Editing. In ACM SIGGRAPH 2009 Papers (SIGGRAPH '09). ACM, New York, NY, USA, Article 24, 11 pages.

Digital Library

[2]

Connelly Barnes and Fang-Lue Zhang. 2017. A survey of the state-of-the-art in patch-based synthesis. Computational Visual Media 3, 1 (01 Mar 2017), 3--20.

[3]

Marcelo Bertalmio, Guillermo Sapiro, Vincent Caselles, and Coloma Ballester. 2000. Image Inpainting. In Proceedings of the 27th Annual Conference on Computer Graphics and Interactive Techniques (SIGGRAPH '00). ACM Press/Addison-Wesley Publishing Co., New York, NY, USA, 417--424.

Digital Library

[4]

M. Bertalmio, L. Vese, G. Sapiro, and S. Osher. 2003. Simultaneous structure and texture image inpainting. IEEE Transactions on Image Processing 12, 8 (Aug 2003), 882--889.

Digital Library

[5]

G. Bradski. 2000. The OpenCV Library. Dr. Dobb's Journal of Software Tools (2000).

[6]

A. Bugeau, M. Bertalmio, V. Caselles, and G. Sapiro. 2010. A Comprehensive Framework for Image Inpainting. IEEE Transactions on Image Processing 19, 10 (Oct 2010), 2634--2645.

Digital Library

[7]

S. C. S. Cheung, J. Zhao, and M. V. Venkatesh. 2006. Efficient Object-Based Video Inpainting. In 2006 International Conference on Image Processing. 705--708.

[8]

A. Criminisi, P. Perez, and K. Toyama. 2004. Region filling and object removal by exemplar-based image inpainting. IEEE Transactions on Image Processing 13, 9 (Sept 2004), 1200--1212.

Digital Library

[9]

Alexei Efros and Thomas Leung. 1999. Texture Synthesis by Non-parametric Sampling. In In International Conference on Computer Vision. 1033--1038.

Digital Library

[10]

M. Elad, J.-L. Starck, P. Querre, and D.L. Donoho. 2005. Simultaneous cartoon and texture image inpainting using morphological component analysis (MCA). Applied and Computational Harmonic Analysis 19, 3 (2005), 340 -- 358.

[11]

Miguel Granados, James Tompkin, K Kim, Oliver Grau, Jan Kautz, and Christian Theobalt. 2012. How not to be seen-object removal from videos of crowded scenes. In Computer Graphics Forum, Vol. 31. Wiley Online Library, 219--228.

Digital Library

[12]

J. Herling and W. Broll. 2014. High-Quality Real-Time Video Inpainting with PixMix. IEEE Transactions on Visualization and Computer Graphics 20, 6 (June 2014), 866--879.

Digital Library

[13]

J. Huang and X. Tang. 2016. A fast video inpainting algorithm based on state matching. In 2016 9th International Congress on Image and Signal Processing, BioMedical Engineering and Informatics (CISP-BMEI). 114--118.

[14]

Yun-Tao Jia, Shi-Min Hu, and R. Ralph Martin. 2005. Video completion using tracking and fragment merging. The Visual Computer 21, 8 (2005), 601--610.

[15]

L. Joyeux, O. Buisson, B. Besserer, and S. Boukir. 1999. Detection and removal of line scratches in motion picture films. In Computer Vision and Pattern Recognition, 1999. IEEE Computer Society Conference on., Vol. 1. 553 Vol. 1.

[16]

N. Kawai, T. Sato, and N. Yokoya. 2016. Diminished Reality Based on Image Inpainting Considering Background Geometry. IEEE Transactions on Visualization and Computer Graphics 22, 3 (March 2016), 1236--1247.

Digital Library

[17]

Felix Klose, Oliver Wang, Jean-Charles Bazin, Marcus Magnor, and Alexander Sorkine-Hornung. 2015. Sampling Based Scene-space Video Processing. ACM Trans. Graph. 34, 4, Article 67 (July 2015), 11 pages.

Digital Library

[18]

A. C. Kokaram. 2004. On missing data treatment for degraded video and film archives: a survey and a new Bayesian approach. IEEE Transactions on Image Processing 13, 3 (March 2004), 397--415.

Digital Library

[19]

Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. 2012. ImageNet Classification with Deep Convolutional Neural Networks. In Advances in Neural Information Processing Systems 25, F. Pereira, C. J. C. Burges, L. Bottou, and K. Q. Weinberger (Eds.). Curran Associates, Inc., 1097--1105. http://papers.nips.cc/paper/4824-imagenet-classification-with-deep-convolutional-neural-networks.pdf

Digital Library

[20]

C. H. Ling, C. W. Lin, C. W. Su, Y. S. Chen, and H. Y. M. Liao. 2011. Virtual Contour Guided Video Object Inpainting Using Posture Mapping and Retrieval. IEEE Transactions on Multimedia 13, 2 (April 2011), 292--302.

Digital Library

[21]

Guilin Liu, Fitsum A. Reda, Kevin J. Shih, Ting-Chun Wang, Andrew Tao, and Bryan Catanzaro. 2018. Image Inpainting for Irregular Holes Using Partial Convolutions. In The European Conference on Computer Vision (ECCV).

[22]

J. Liu, N. Akhtar, and A. Mian. 2017. Viewpoint Invariant RGB-D Human Action Recognition. In 2017 International Conference on Digital Image Computing: Techniques and Applications (DICTA). 1--8.

[23]

Ming Liu, Shifeng Chen, Jianzhuang Liu, and Xiaoou Tang. 2009. Video Completion via Motion Guided Spatial-temporal Global Optimization. In Proceedings of the 17th ACM International Conference on Multimedia (MM '09). ACM, New York, NY, USA, 537--540.

Digital Library

[24]

Alasdair Newson, Andrés Almansa, Matthieu Fradet, Yann Gousseau, and Patrick Pérez. 2014. Video Inpainting of Complex Scenes. SIAM Journal on Imaging Sciences 7, 4 (2014), 1993--2019.

Digital Library

[25]

Occipital. 2013. Structure Sensor. (2013). http://structure.io/

[26]

Manuel M Oliveira, Brian Bowen, Richard McKenna, and Yu-Sung Chang. 2001. Fast digital image inpainting. In Proceedings of the International Conference on Visualization, Imaging and Image Processing (VIIP 2001), Marbella, Spain. 106--107.

[27]

E. Rublee, V. Rabaud, K. Konolige, and G. Bradski. 2011. ORB: An efficient alternative to SIFT or SURF. In 2011 International Conference on Computer Vision. 2564--2571.

Digital Library

[28]

T. Shiratori, Y. Matsushita, Xiaoou Tang, and Sing Bing Kang. 2006. Video Completion by Motion Field Transfer. In 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'06), Vol. 1. 411--418.

Digital Library

[29]

Jian Sun, Lu Yuan, Jiaya Jia, and Heung-Yeung Shum. 2005. Image Completion with Structure Propagation. ACM Trans. Graph. 24, 3 (July 2005), 861--868.

Digital Library

[30]

Chuan Wang, Haibin Huang, Xiaoguang Han, and Jue Wang. 2019. Video Inpainting by Jointly Learning Temporal Structure and Spatial Details. In Proceedings of the Thirty-Third AAAI Conference on Artificial Intelligence.

[31]

Y. Wexler, E. Shechtman, and M. Irani. 2007. Space-Time Completion of Video. IEEE Transactions on Pattern Analysis and Machine Intelligence 29, 3 (March 2007), 463--476.

Digital Library

[32]

Tianfan Xue, Michael Rubinstein, Ce Liu, and William T. Freeman. 2015. A Computational Approach for Obstruction-Free Photography. ACM Transactions on Graphics (Proc. SIGGRAPH) 34, 4 (2015).

Digital Library

[33]

Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. 2018. The Unreasonable Effectiveness of Deep Features as a Perceptual Metric. In CVPR.

Cited By

Liao SZhou YPopescu V(2023)AR Interfaces for Disocclusion—A Comparative Study2023 IEEE Conference Virtual Reality and 3D User Interfaces (VR)10.1109/VR55154.2023.00068(530-540)Online publication date: Mar-2023
https://doi.org/10.1109/VR55154.2023.00068
Lin CPopescu V(2022)Fast Intra-Frame Video Splicing for Occlusion Removal in Diminished RealityVirtual Reality and Mixed Reality10.1007/978-3-031-16234-3_7(111-134)Online publication date: 3-Sep-2022
https://doi.org/10.1007/978-3-031-16234-3_7
Dima ESjostrom M(2021)Camera and Lidar-Based View Generation for Augmented Remote Operation in Mining ApplicationsIEEE Access10.1109/ACCESS.2021.30868949(82199-82212)Online publication date: 2021
https://doi.org/10.1109/ACCESS.2021.3086894

Index Terms

RGBD temporal resampling for real-time occlusion removal
1. Computing methodologies
  1. Artificial intelligence
    1. Computer vision
      1. Computer vision problems
        Reconstruction
  2. Computer graphics
    1. Image manipulation
      1. Image-based rendering
2. Human-centered computing
  1. Visualization

Recommendations

An RGBD Tracker Based on KCF Adaptively Handling Long-Term Occlusion
Intelligent Computing Theories and Application
Abstract
Since occlusion still be a challenge for object tracking in RGB data. In this paper, we propose an RGBD single-object tracker that built upon the well-known base KCF tracker and exploit how the depth information fusing to handle partial and long-...
Nonpinhole Approximations for Interactive Rendering

Researchers have used depth images to approximate scene geometry in a variety of interactive 3D graphics applications. Previous researchers constructed images using orthographic or perspective projection, which limits the approximation quality to what's ...
GPU-based real-time RGBD data filtering

Commodity RGBD cameras such as Kinect sensor have recently proved a large success in many indoor robotics and computer vision applications. Nevertheless, professional applications cannot rely on their raw outputs because of the low accuracy. These ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

I3D '19: Proceedings of the ACM SIGGRAPH Symposium on Interactive 3D Graphics and Games

May 2019

152 pages

ISBN:9781450363105

DOI:10.1145/3306131

Editor:
Stephen N. Spencer
University of Washington
,
General Chairs:
Sheldon Andrews
École de Technologie Supérieure
,
Natalya Tatarchuk
Unity Technologies

Copyright © 2019 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGGRAPH: ACM Special Interest Group on Computer Graphics and Interactive Techniques

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 21 May 2019

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

I3D '19

Sponsor:

SIGGRAPH

I3D '19: Symposium on Interactive 3D Graphics and Games

May 21 - 23, 2019

Quebec, Montreal, Canada

Acceptance Rates

Overall Acceptance Rate 148 of 485 submissions, 31%

Upcoming Conference

I3D '25

Sponsor:
siggraph

Symposium on Interactive 3D Graphics and Games

May 7 - 9, 2025

Jersey City , NJ , USA

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

3
Total Citations
View Citations
191
Total Downloads

Downloads (Last 12 months)13
Downloads (Last 6 weeks)0

Reflects downloads up to 27 Feb 2025

Other Metrics

View Author Metrics

Citations

Cited By

Liao SZhou YPopescu V(2023)AR Interfaces for Disocclusion—A Comparative Study2023 IEEE Conference Virtual Reality and 3D User Interfaces (VR)10.1109/VR55154.2023.00068(530-540)Online publication date: Mar-2023
https://doi.org/10.1109/VR55154.2023.00068
Lin CPopescu V(2022)Fast Intra-Frame Video Splicing for Occlusion Removal in Diminished RealityVirtual Reality and Mixed Reality10.1007/978-3-031-16234-3_7(111-134)Online publication date: 3-Sep-2022
https://doi.org/10.1007/978-3-031-16234-3_7
Dima ESjostrom M(2021)Camera and Lidar-Based View Generation for Augmented Remote Operation in Mining ApplicationsIEEE Access10.1109/ACCESS.2021.30868949(82199-82212)Online publication date: 2021
https://doi.org/10.1109/ACCESS.2021.3086894

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten