Abstract
Generating novel views from a single input is a challenging task that requires the prediction of occluded and non-visible content. Nevertheless, it is an interesting and active area of research due to its several applications such as entertainment. In this work, we propose an end-to-end architecture for monocular view synthesis based on the layered scene inference (LSI) method. The LSI uses layered depth images that can represent complex scenes with a reduced number of layers. To improve the LSI predictions, we develop two new strategies: (i) a pyramidal architecture that learns LDI predictions for different resolutions of the input and (ii) an image outpainting for filling the missing information at the LDI borders. We evaluate our method on the KITTI dataset, and show that the proposed versions outperform the baseline.
This work was funded by Samsung Eletrônica da Amazônia Ltda., through the project “Parallax Effect”, within the scope of the Informatics Law No. 8248/91.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Chen, X., Chen, X., Zha, Z.J.: Structure-aware residual pyramid network for monocular depth estimation. In: International Joint Conference on Artificial Intelligence (2019)
Dhamo, H., Navab, N., Tombari, F.: Object-driven multi-layer scene decomposition from a single image. In: IEEE/CVF International Conference on Computer Vision (2019)
Dhamo, H., Tateno, K., Laina, I., Navab, N., Tombari, F.: Peeking behind objects: layered depth prediction from a single image. Pattern Recognit. Lett. (2019)
Flynn, J., et al.: DeepView: view synthesis with learned gradient descent. In: IEEE Conference on Computer Vision and Pattern Recognition (2019)
Geiger, A., Lenz, P., Stiller, C., Urtasun, R.: Vision meets robotics: the KITTI dataset. Int. J. Robot. Res. (2013)
Han, D., Kim, J., Kim, J.: Deep pyramidal residual networks. In: IEEE Conference on Computer Vision and Pattern Recognition (2017)
Hoorick, B.V.: Image Outpainting and Harmonization using GANs - PyTorch Implementation (2020). https://github.com/basilevh/image-outpainting
Layton, O.W., Fajen, B.R.: Computational mechanisms for perceptual stability using disparity and motion parallax. J. Neurosci. (2020)
Liu, S., Li, T., Chen, W., Li, H.: Soft rasterizer: a differentiable renderer for image-based 3D reasoning. In: IEEE International Conference on Computer Vision (2019)
Luvizon, D.C., et al.: Adaptive multiplane image generation from a single internet picture. In: Winter Conference on Applications of Computer Vision (2021)
Mayer, N., et al.: A large dataset to train convolutional networks for disparity, optical flow, and scene flow estimation. In: IEEE Conference on Computer Vision and Pattern Recognition (2016)
Pinto, A., et al.: Parallax motion effect generation through instance segmentation and depth estimation. In: International Conference on Image Processing. IEEE (2020)
Shade, J., Gortler, S., He, L.w., Szeliski, R.: Layered depth images. In: 25th Annual Conference on Computer Graphics and Interactive Techniques (1998)
Shih, M.L., Su, S.Y., Kopf, J., Huang, J.B.: 3D photography using context-aware layered depth inpainting. In: IEEE Conference on Computer Vision and Pattern Recognition (2020)
Srinivasan, P.P., Tucker, R., Barron, J.T., Ramamoorthi, R., Ng, R., Snavely, N.: Pushing the boundaries of view extrapolation with multiplane images. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition (2019)
Sun, D., Yang, X., Liu, M.Y., Kautz, J.: PWC-Net: CNNs for optical flow using pyramid, warping, and cost volume. In: IEEE Conference on Computer Vision and Pattern Recognition (2018)
Tucker, R., Snavely, N.: Single-view view synthesis with multiplane images. In: IEEE Conference on Computer Vision and Pattern Recognition (2020)
Tulsiani, S., Tucker, R., Snavely, N.: Layer-structured 3D scene inference via view synthesis. In: Ferrari, V., Hebert, M., Sminchisescu, C., Weiss, Y. (eds.) ECCV 2018. LNCS, vol. 11211, pp. 311–327. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-01234-2_19
Van Hoorick, B.: Image outpainting and harmonization using generative adversarial networks. arXiv preprint arXiv:1912.10960 (2019)
Wiles, O., Gkioxari, G., Szeliski, R., Johnson, J.: SynSin: end-to-end view synthesis from a single image. In: Conference on Computer Vision and Pattern Recognition (2020)
Zhang, M., Zhang, Y., Piao, Y., Liu, J., Ji, X., Zhang, Y.: Parallax based motion estimation in integral imaging. In: Digital Holography and Three-Dimensional Imaging. Optical Society of America (2019)
Zhou, T., Tucker, R., Flynn, J., Fyffe, G., Snavely, N.: Stereo magnification: learning view synthesis using multiplane images. ACM Trans. Graph. (2018)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2021 Springer Nature Switzerland AG
About this paper
Cite this paper
Souza, M.R. et al. (2021). Pyramidal Layered Scene Inference with Image Outpainting for Monocular View Synthesis. In: Tsapatsoulis, N., Panayides, A., Theocharides, T., Lanitis, A., Pattichis, C., Vento, M. (eds) Computer Analysis of Images and Patterns. CAIP 2021. Lecture Notes in Computer Science(), vol 13052. Springer, Cham. https://doi.org/10.1007/978-3-030-89128-2_4
Download citation
DOI: https://doi.org/10.1007/978-3-030-89128-2_4
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-89127-5
Online ISBN: 978-3-030-89128-2
eBook Packages: Computer ScienceComputer Science (R0)