Skip to main content
SpringerLink
Log in

Point-based rendering enhancement via deep learning

  • Original Article
  • Published:
The Visual Computer Aims and scope Submit manuscript

Abstract

Current state-of-the-art point rendering techniques such as splat rendering generally require very high-resolution point clouds in order to create high-quality photo realistic renderings. These can be very time consuming to acquire and oftentimes also require high-end expensive scanners. This paper proposes a novel deep learning-based approach that can generate high-resolution photo realistic point renderings from low-resolution point clouds. More specifically, we propose to use co-registered high-quality photographs as the ground truth data to train the deep neural network for point-based rendering. The proposed method can generate high-quality point rendering images very efficiently and can be used for interactive navigation of large-scale 3D scenes as well as image-based localization. Extensive quantitative evaluations on both synthetic and real datasets show that the proposed method outperforms state-of-the-art methods.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6

Similar content being viewed by others

Notes

  1. Note the sub-dividing in the spherical coordinates (which is used in real LIDAR) is the primary reason for the produced scattered point clouds because there is a higher density in the center region than in the outer region.

References

  1. Abadi, M., Agarwal, A., Barham, P., Brevdo, E., Chen, Z., Citro, C., Corrado, G.S., Davis, A., Dean, J., Devin, M., et al.: Tensorflow: large-scale machine learning on heterogeneous systems. http://www.tensorflow.org/. Software available from www.tensorflow.org (2015)

  2. Botsch, M., Hornung, A., Zwicker, M., Kobbelt, L.: High-quality surface splatting on today’s GPUS. In: Proceedings Eurographics/IEEE VGTC Symposium Point-Based Graphics, IEEE, 2005, pp. 17–141 (2005)

  3. Brown, M., Lowe, D.G.: Unsupervised 3D object recognition and reconstruction in unordered datasets. In: Fifth International Conference on 3-D Digital Imaging and Modeling, IEEE, 2005. 3DIM 2005, pp. 56–63 (2005)

  4. Chang, H., Yeung, D.Y., Xiong, Y.: Super-resolution through neighbor embedding. In: Proceedings of the 2004 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, IEEE, 2004. CVPR 2004, vol. 1, pp. I–I (2004)

  5. Cui, Z., Chang, H., Shan, S., Zhong, B., Chen, X.: Deep network cascade for image super-resolution. In: Fleet, D., Pajdla, T., Schiele, B., Tuytelaars, T. (eds.) European Conference on Computer Vision, pp. 49–64. Springer, Berlin (2014)

    Google Scholar 

  6. Dai, D., Timofte, R., Van Gool, L.: Jointly optimized regressors for image super-resolution. In: Computer Graphics Forum, vol. 34, pp. 95–104. Wiley, New York (2015)

  7. Denton, E.L., Chintala, S., Fergus, R., et al.: Deep generative image models using a Laplacian pyramid of adversarial networks. In: Advances in Neural Information Processing Systems, pp. 1486–1494 (2015)

  8. Dong, C., Loy, C.C., He, K., Tang, X.: Image super-resolution using deep convolutional networks. IEEE Trans. Pattern Anal. Mach. Intell. 38, 295–307 (2015)

    Article  Google Scholar 

  9. Dong, C., Loy, C.C., Tang, X.: Accelerating the super-resolution convolutional neural network. In: European Conference on Computer Vision, pp. 391–407. Springer (2016)

  10. Freedman, G., Fattal, R.: Image and video upscaling from local self-examples. ACM Trans. Graph. (TOG) 30(2), 12 (2011)

    Article  Google Scholar 

  11. Glasner, D., Bagon, S., Irani, M.: Super-resolution from a single image. In: 2009 IEEE 12th International Conference on Computer Vision, IEEE, pp. 349–356 (2009)

  12. Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., Bengio, Y.: Generative adversarial nets. In: Advances in Neural Information Processing Systems, pp. 2672–2680 (2014)

  13. Hartley, R., Zisserman, A.: Multiple View Geometry in Computer Vision. Cambridge University Press, Cambridge (2003)

    MATH  Google Scholar 

  14. He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 770–778 (2016)

  15. Huang, J.B., Singh, A., Ahuja, N.: Single image super-resolution from transformed self-exemplars. In: 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), IEEE, pp. 5197–5206 (2015)

  16. Irschara, A., Zach, C., Frahm, J.M., Bischof, H.: From structure-from-motion point clouds to fast location recognition. In: IEEE Conference on Computer Vision and Pattern Recognition, IEEE. CVPR 2009, pp. 2599–2606 (2009)

  17. Jia, K., Wang, X., Tang, X.: Image transformation based on learning dictionaries across image spaces. IEEE Trans. Pattern Anal. Mach. Intell. 35(2), 367–380 (2013)

    Article  Google Scholar 

  18. Johnson, J., Alahi, A., Fei-Fei, L.: Perceptual losses for real-time style transfer and super-resolution. In: European Conference on Computer Vision, pp. 694–711. Springer (2016)

  19. Kim, J., Lee, J.K., Lee, K.M.: Accurate image super-resolution using very deep convolutional networks. arXiv preprint arXiv:1511.04587 (2015)

  20. Kim, J., Lee, J.K., Lee, K.M.: Deeply-recursive convolutional network for image super-resolution. arXiv preprint arXiv:1511.04491 (2015)

  21. Kim, K.I., Kwon, Y.: Single-image super-resolution using sparse regression and natural image prior. IEEE Trans. Pattern Anal. Mach. Intell. 32(6), 1127–1133 (2010)

    Article  Google Scholar 

  22. Kingma, D., Ba, J.: Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014)

  23. Kobbelt, L., Botsch, M.: A survey of point-based techniques in computer graphics. Comput. Graph. 28(6), 801–814 (2004)

    Article  Google Scholar 

  24. Ledig, C., Theis, L., Huszár, F., Caballero, J., Cunningham, A., Acosta, A., Aitken, A., Tejani, A., Totz, J., Wang, Z., et al.: Photo-realistic single image super-resolution using a generative adversarial network. arXiv preprint arXiv:1609.04802 (2016)

  25. Lee, C.Y., Xie, S., Gallagher, P., Zhang, Z., Tu, Z.: Deeply-supervised nets. In: Artificial Intelligence and Statistics, pp. 562–570 (2015)

  26. Li, Y., Snavely, N., Huttenlocher, D.P.: Location recognition using prioritized feature matching. In: European Conference on Computer Vision, pp. 791–804. Springer (2010)

  27. Lim, B., Son, S., Kim, H., Nah, S., Lee, K.M.: Enhanced deep residual networks for single image super-resolution. In: 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pp. 1132–1140. IEEE (2017)

  28. Lipponer, S.: Surface splatting. https://github.com/sebastianlipponer/surface_splatting (2015)

  29. Liu, Y., Xiong, Y.: Automatic segmentation of unorganized noisy point clouds based on the gaussian map. Comput. Aided Des. 40(5), 576–594 (2008)

    Article  MathSciNet  Google Scholar 

  30. Lowe, D.G.: Distinctive image features from scale-invariant keypoints. Int. J. Comput. Vis. 60(2), 91–110 (2004)

    Article  Google Scholar 

  31. Mathieu, M., Couprie, C., LeCun, Y.: Deep multi-scale video prediction beyond mean square error. arXiv preprint arXiv:1511.05440 (2015)

  32. Nistér, D.: An efficient solution to the five-point relative pose problem. IEEE Trans. Pattern Anal. Mach. Intell. 26(6), 756–770 (2004)

    Article  Google Scholar 

  33. Qi, C.R., Su, H., Mo, K., Guibas, L.J.: Deep learning on point sets for 3D classification and segmentation. IEEE Proc. Comput. Vis. Pattern Recogn. (CVPR) 1(2), 4 (2017)

    Google Scholar 

  34. Radford, A., Metz, L., Chintala, S.: Unsupervised representation learning with deep convolutional generative adversarial networks. arXiv preprint arXiv:1511.06434 (2015)

  35. Savva, M., Yu, F., Su, H., Aono, M., Chen, B., Cohen-Or, D., Deng, W., Su, H., Bai, S., Bai, X., et al.: Shrec16 track large-scale 3D shape retrieval from shapenet core55. In: Proceedings of the Eurographics Workshop on 3D Object Retrieval (2016)

  36. Schulter, S., Leistner, C., Bischof, H.: Fast and accurate image upscaling with super-resolution forests. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3791–3799 (2015)

  37. Shi, W., Caballero, J., Huszár, F., Totz, J., Aitken, A.P., Bishop, R., Rueckert, D., Wang, Z.: Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1874–1883 (2016)

  38. Sibbing, D., Sattler, T., Leibe, B., Kobbelt, L.: Sift-realistic rendering. In: International Conference on 3D Vision, pp. 56–63 (2013)

  39. Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. CoRR abs/1409.1556 (2014)

  40. Snavely, N., Seitz, S.M., Szeliski, R.: Photo tourism: exploring photo collections in 3D. ACM Trans. Graph. (TOG) 25, 835–846 (2006)

    Article  Google Scholar 

  41. Timofte, R., Agustsson, E., Van Gool, L., Yang, M.H., Zhang, L., Lim, B., Son, S., Kim, H., Nah, S., Lee, K.M., et al.: Ntire 2017 challenge on single image super-resolution: methods and results. In: 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), IEEE, pp. 1110–1121 (2017)

  42. Timofte, R., De Smet, V., Van Gool, L.: Anchored neighborhood regression for fast example-based super-resolution. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 1920–1927 (2013)

  43. Timofte, R., De Smet, V., Van Gool, L.: A+: Adjusted anchored neighborhood regression for fast super-resolution. In: Asian Conference on Computer Vision, pp. 111–126. Springer (2014)

  44. Timofte, R., Rothe, R., Van Gool, L.: Seven ways to improve example-based single image super resolution. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1865–1873 (2016)

  45. Vinyals, O., Bengio, S., Kudlur, M.: Order matters: sequence to sequence for sets. arXiv preprint arXiv:1511.06391 (2015)

  46. Xie, S., Tu, Z.: Holistically-nested edge detection. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 1395–1403 (2015)

  47. Yang, C.Y., Huang, J.B., Yang, M.H.: Exploiting self-similarities for single frame super-resolution. In: Proceedings of the Asian Conference on Computer Vision, pp. 497–510 (2011)

  48. Yang, C.Y., Ma, C., Yang, M.H.: Single-image super-resolution: a benchmark. In: European Conference on Computer Vision, pp. 372–386. Springer (2014)

  49. Yang, J., Wang, Z., Lin, Z., Cohen, S., Huang, T.: Coupled dictionary training for image super-resolution. IEEE Trans. Image Process. 21(8), 3467–3478 (2012)

    Article  MathSciNet  MATH  Google Scholar 

  50. Yang, J., Wright, J., Huang, T., Ma, Y.: Image super-resolution as sparse representation of raw image patches. In: IEEE Conference on Computer Vision and Pattern Recognition, IEEE. CVPR 2008, pp. 1–8 (2008)

  51. Yang, J., Wright, J., Huang, T.S., Ma, Y.: Image super-resolution via sparse representation. IEEE Trans. Image Process. 19(11), 2861–2873 (2010)

    Article  MathSciNet  MATH  Google Scholar 

  52. Zwicker, M., Pfister, H., Van Baar, J., Gross, M.: Surface splatting. In: Proceedings of the 28th Annual Conference on Computer Graphics and Interactive Techniques, pp. 371–378. ACM (2001)

Download references

Acknowledgements

We would like to thank Sebastian Lipponer for providing open source code [28] of which our splat rendering implementation is mainly based on. We also thank him for all the suggestions during the implementation. We would like to thank Qing Lei and Xu Wang for helping us to generate the video. We also like to thank Roger Kiew, Fan Gao and Chuhang Wang for helping us to generate the training data.

Author information

Authors and Affiliations

  1. University of Missouri, Columbia, USA

    Giang Bui, Truc Le & Ye Duan

  2. University of North Carolina Wilmington, Wilmington, USA

    Brittany Morago

Authors
  1. Giang Bui
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Truc Le
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Brittany Morago
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Ye Duan
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Ye Duan.

Ethics declarations

Conflict of interest

Giang Bui declares that he has no conflict of interest. Truc Le declares that he has no conflict of interest. Brittany Morago declares that she has no conflict of interest. Ye Duan declares that he has no conflict of interest.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Bui, G., Le, T., Morago, B. et al. Point-based rendering enhancement via deep learning. Vis Comput 34, 829–841 (2018). https://doi.org/10.1007/s00371-018-1550-6

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00371-018-1550-6

Keywords

Search

Navigation