Meng-Lin Wu
Voicu Popescu
ABSTRACT
Occlusions disrupt the visualization of an object of interest, or target, in a real world scene. Video inpainting removes occlusions from a video stream by cutting out occluders and filling in with a plausible visualization of the object, but the approach is too slow for real-time performance. In this paper, we present a method for realtime occlusion removal in the visualization of a real world scene that is captured with an RGBD stream. Our pipeline segments the current RGBD frame to find the target and the occluders, searches for the best matching disoccluded view of the target in an earlier frame, computes a mapping between the target in the current frame and the target in the best matching frame, inpaints the missing pixels of the target in the current frame by resampling from the earlier frame, and visualizes the disoccluded target in the current frame. We demonstrate our method in the case of a walking human occluded by stationary or walking humans. Our method does not rely on a known 2D or 3D model of the target or of the occluders, and therefore it generalizes to other shapes. Our method runs at an interactive frame rate of 30fps.
Supplemental Material
Available for Download
Supplemental material.
- Connelly Barnes, Eli Shechtman, Adam Finkelstein, and Dan B Goldman. 2009. Patch-Match: A Randomized Correspondence Algorithm for Structural Image Editing. In ACM SIGGRAPH 2009 Papers (SIGGRAPH '09). ACM, New York, NY, USA, Article 24, 11 pages. Google Scholar
Digital Library
- Connelly Barnes and Fang-Lue Zhang. 2017. A survey of the state-of-the-art in patch-based synthesis. Computational Visual Media 3, 1 (01 Mar 2017), 3--20.Google Scholar
- Marcelo Bertalmio, Guillermo Sapiro, Vincent Caselles, and Coloma Ballester. 2000. Image Inpainting. In Proceedings of the 27th Annual Conference on Computer Graphics and Interactive Techniques (SIGGRAPH '00). ACM Press/Addison-Wesley Publishing Co., New York, NY, USA, 417--424. Google ScholarDigital Library
- M. Bertalmio, L. Vese, G. Sapiro, and S. Osher. 2003. Simultaneous structure and texture image inpainting. IEEE Transactions on Image Processing 12, 8 (Aug 2003), 882--889. Google ScholarDigital Library
- G. Bradski. 2000. The OpenCV Library. Dr. Dobb's Journal of Software Tools (2000).Google Scholar
- A. Bugeau, M. Bertalmio, V. Caselles, and G. Sapiro. 2010. A Comprehensive Framework for Image Inpainting. IEEE Transactions on Image Processing 19, 10 (Oct 2010), 2634--2645. Google ScholarDigital Library
- S. C. S. Cheung, J. Zhao, and M. V. Venkatesh. 2006. Efficient Object-Based Video Inpainting. In 2006 International Conference on Image Processing. 705--708.Google Scholar
- A. Criminisi, P. Perez, and K. Toyama. 2004. Region filling and object removal by exemplar-based image inpainting. IEEE Transactions on Image Processing 13, 9 (Sept 2004), 1200--1212. Google ScholarDigital Library
- Alexei Efros and Thomas Leung. 1999. Texture Synthesis by Non-parametric Sampling. In In International Conference on Computer Vision. 1033--1038. Google ScholarDigital Library
- M. Elad, J.-L. Starck, P. Querre, and D.L. Donoho. 2005. Simultaneous cartoon and texture image inpainting using morphological component analysis (MCA). Applied and Computational Harmonic Analysis 19, 3 (2005), 340 -- 358.Google ScholarCross Ref
- Miguel Granados, James Tompkin, K Kim, Oliver Grau, Jan Kautz, and Christian Theobalt. 2012. How not to be seen-object removal from videos of crowded scenes. In Computer Graphics Forum, Vol. 31. Wiley Online Library, 219--228. Google ScholarDigital Library
- J. Herling and W. Broll. 2014. High-Quality Real-Time Video Inpainting with PixMix. IEEE Transactions on Visualization and Computer Graphics 20, 6 (June 2014), 866--879. Google ScholarDigital Library
- J. Huang and X. Tang. 2016. A fast video inpainting algorithm based on state matching. In 2016 9th International Congress on Image and Signal Processing, BioMedical Engineering and Informatics (CISP-BMEI). 114--118.Google Scholar
- Yun-Tao Jia, Shi-Min Hu, and R. Ralph Martin. 2005. Video completion using tracking and fragment merging. The Visual Computer 21, 8 (2005), 601--610.Google ScholarCross Ref
- L. Joyeux, O. Buisson, B. Besserer, and S. Boukir. 1999. Detection and removal of line scratches in motion picture films. In Computer Vision and Pattern Recognition, 1999. IEEE Computer Society Conference on., Vol. 1. 553 Vol. 1.Google Scholar
- N. Kawai, T. Sato, and N. Yokoya. 2016. Diminished Reality Based on Image Inpainting Considering Background Geometry. IEEE Transactions on Visualization and Computer Graphics 22, 3 (March 2016), 1236--1247. Google ScholarDigital Library
- Felix Klose, Oliver Wang, Jean-Charles Bazin, Marcus Magnor, and Alexander Sorkine-Hornung. 2015. Sampling Based Scene-space Video Processing. ACM Trans. Graph. 34, 4, Article 67 (July 2015), 11 pages. Google ScholarDigital Library
- A. C. Kokaram. 2004. On missing data treatment for degraded video and film archives: a survey and a new Bayesian approach. IEEE Transactions on Image Processing 13, 3 (March 2004), 397--415. Google ScholarDigital Library
- Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. 2012. ImageNet Classification with Deep Convolutional Neural Networks. In Advances in Neural Information Processing Systems 25, F. Pereira, C. J. C. Burges, L. Bottou, and K. Q. Weinberger (Eds.). Curran Associates, Inc., 1097--1105. http://papers.nips.cc/paper/4824-imagenet-classification-with-deep-convolutional-neural-networks.pdf Google ScholarDigital Library
- C. H. Ling, C. W. Lin, C. W. Su, Y. S. Chen, and H. Y. M. Liao. 2011. Virtual Contour Guided Video Object Inpainting Using Posture Mapping and Retrieval. IEEE Transactions on Multimedia 13, 2 (April 2011), 292--302. Google ScholarDigital Library
- Guilin Liu, Fitsum A. Reda, Kevin J. Shih, Ting-Chun Wang, Andrew Tao, and Bryan Catanzaro. 2018. Image Inpainting for Irregular Holes Using Partial Convolutions. In The European Conference on Computer Vision (ECCV).Google Scholar
- J. Liu, N. Akhtar, and A. Mian. 2017. Viewpoint Invariant RGB-D Human Action Recognition. In 2017 International Conference on Digital Image Computing: Techniques and Applications (DICTA). 1--8.Google Scholar
- Ming Liu, Shifeng Chen, Jianzhuang Liu, and Xiaoou Tang. 2009. Video Completion via Motion Guided Spatial-temporal Global Optimization. In Proceedings of the 17th ACM International Conference on Multimedia (MM '09). ACM, New York, NY, USA, 537--540. Google ScholarDigital Library
- Alasdair Newson, Andrés Almansa, Matthieu Fradet, Yann Gousseau, and Patrick Pérez. 2014. Video Inpainting of Complex Scenes. SIAM Journal on Imaging Sciences 7, 4 (2014), 1993--2019.Google ScholarDigital Library
- Occipital. 2013. Structure Sensor. (2013). http://structure.io/Google Scholar
- Manuel M Oliveira, Brian Bowen, Richard McKenna, and Yu-Sung Chang. 2001. Fast digital image inpainting. In Proceedings of the International Conference on Visualization, Imaging and Image Processing (VIIP 2001), Marbella, Spain. 106--107.Google Scholar
- E. Rublee, V. Rabaud, K. Konolige, and G. Bradski. 2011. ORB: An efficient alternative to SIFT or SURF. In 2011 International Conference on Computer Vision. 2564--2571. Google ScholarDigital Library
- T. Shiratori, Y. Matsushita, Xiaoou Tang, and Sing Bing Kang. 2006. Video Completion by Motion Field Transfer. In 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'06), Vol. 1. 411--418. Google ScholarDigital Library
- Jian Sun, Lu Yuan, Jiaya Jia, and Heung-Yeung Shum. 2005. Image Completion with Structure Propagation. ACM Trans. Graph. 24, 3 (July 2005), 861--868. Google ScholarDigital Library
- Chuan Wang, Haibin Huang, Xiaoguang Han, and Jue Wang. 2019. Video Inpainting by Jointly Learning Temporal Structure and Spatial Details.. In Proceedings of the Thirty-Third AAAI Conference on Artificial Intelligence.Google ScholarCross Ref
- Y. Wexler, E. Shechtman, and M. Irani. 2007. Space-Time Completion of Video. IEEE Transactions on Pattern Analysis and Machine Intelligence 29, 3 (March 2007), 463--476. Google ScholarDigital Library
- Tianfan Xue, Michael Rubinstein, Ce Liu, and William T. Freeman. 2015. A Computational Approach for Obstruction-Free Photography. ACM Transactions on Graphics (Proc. SIGGRAPH) 34, 4 (2015). Google ScholarDigital Library
- Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. 2018. The Unreasonable Effectiveness of Deep Features as a Perceptual Metric. In CVPR.Google Scholar
Index Terms
- RGBD temporal resampling for real-time occlusion removal
Recommendations
An RGBD Tracker Based on KCF Adaptively Handling Long-Term Occlusion
Intelligent Computing Theories and ApplicationAbstractSince occlusion still be a challenge for object tracking in RGB data. In this paper, we propose an RGBD single-object tracker that built upon the well-known base KCF tracker and exploit how the depth information fusing to handle partial and long-...
Nonpinhole Approximations for Interactive Rendering
Researchers have used depth images to approximate scene geometry in a variety of interactive 3D graphics applications. Previous researchers constructed images using orthographic or perspective projection, which limits the approximation quality to what's ...
GPU-based real-time RGBD data filtering
Commodity RGBD cameras such as Kinect sensor have recently proved a large success in many indoor robotics and computer vision applications. Nevertheless, professional applications cannot rely on their raw outputs because of the low accuracy. These ...
Comments