Abstract
NeRF (Neural Radiance Fields), as an implicit 3D representation, has demonstrated the capability to generate highly realistic and dynamically consistent images. However, its hierarchical sampling approach introduces a significant amount of redundant computation, leading to erroneous geometric information, particularly in high-frequency facial details. In this paper, we propose SGFNeRF, a novel 3D face generation model by integrating a 2D CNN-based generator and face depth priors optimization method in the same framework. We employ a Gaussian distribution for sampling to extract facial surface information. Additionally, we design a feature decoder to incorporates depth uncertainty into our method, enabling the method to explore regions further away from face surfaces while preserving its ability to capture fine-grained details. We conduct experiments on the FFHQ dataset to evaluate the performance of our proposed method. The results demonstrate a significant improvement compared to previous approaches in terms of various evaluation metrics.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Mildenhall, B., Srinivasan, P.P., Tancik, M., Barron, J.T., Ramamoorthi, R., Ng, R.: NeRF: representing scenes as neural radiance fields for view synthesis. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, J.-M. (eds.) ECCV 2020. LNCS, vol. 12346, pp. 405–421. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-58452-8_24
Shahbazi M, Ntavelis E, Tonioni A, et al.: NeRF-GAN distillation for efficient 3D-aware generation with convolutions. arXiv preprint arXiv:2303.12865 (2023)
Sun J, Wang X, Zhang Y, et al.: FENeRF: face editing in neural radiance fields. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7672–7682 (2022)
Gu J, Liu L, Wang P, et al.: StyleNeRF: a style-based 3d-aware generator for high-resolution image synthesis. arXiv preprint arXiv:2110.08985 (2021)
Hong Y, Peng B, Xiao H, et al.: HeadNeRF: a real-time nerf-based parametric head model. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 20374–20384(2022)
Mihajlovic, M., Bansal, A., Zollhoefer, M., et al.: KeypointNeRF: generalizing image-based volumetric avatars using relative spatial encoding of keypoints. In: Vedaldi, A., Gupta, A., Lempitsky, V., Schiele, B., Fitzgibbon, A. (eds.) ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part XV, pp. 179–197. Springer Nature Switzerland, Cham (2022). https://doi.org/10.1007/978-3-031-19784-0_11
Karras, T., Laine, S., Aila, T.: A style-based generator architecture for generative adversarial networks. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4401–4410 (2019)
Guo, J., Zhu, X., Yang, Y., Yang, F., Lei, Z., Li, S.Z.: Towards fast, accurate and stable 3D dense face alignment. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, J.-M. (eds.) ECCV 2020. LNCS, vol. 12364, pp. 152–168. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-58529-7_10
Luo, Y., Lü, J., Jiang, X., Zhang, B.: Learning from architectural redundancy: enhanced deep supervision in deep multipath encoder-decoder networks. In: IEEE Transactions on Neural Networks and Learning Systems (2021). https://doi.org/10.1109/TNNLS.2021.3056384
Zhang, K., Riegler, G., Snavely, N., et al.: NeRF++: Analyzing and improving neural radiance fields. arXiv preprint arXiv:2010.07492 (2020)
Qu, Q., Liu, K., Wang, W., et al.: Spacecraft proximity maneuvering and rendezvous with collision avoidance based on reinforcement learning. IEEE Trans. Aerosp. Electron. Syst. 58(6), 5823–5834 (2022)
Barron, J.T., Mildenhall, B., Tancik, M., et al.: Mip-NeRF: a multiscale representation for anti-aliasing neural radiance fields. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 5855–5864 (2021)
Sun, J., Wang, X., Shi, Y., et al.: IDE-3D: interactive disentangled editing for high-resolution 3D-aware portrait synthesis. ACM Trans. Graph. (TOG) 41(6), 1–10 (2022)
Sauer, A., Schwarz, K., Geiger, A.: StyleGAN-XL: scaling StyleGAN to large diverse datasets. In: ACM SIGGRAPH 2022 Conference Proceedings, pp. 1–10 (2022)
Liang, D., Wang, R., Tian, X., et al.: PCGAN: partition-controlled human image generation. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 33, no. 01, pp. 8698–8705 (2019)
Deng, Y., Yang, J., Chen, D., et al.: Disentangled and controllable face image generation via 3D imitative-contrastive learning. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5154–5163 (2020)
Chan, E.R., Lin, C.Z., Chan, M.A., et al.: Efficient geometry-aware 3D generative adversarial networks. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 16123–16133 (2022)
Kajiya, J.T., Herzen, B.P.V.: Ray tracing volume densities. In: Computer Graphics (SIGGRAPH) (1984)
Dai, T., Cai, J., Zhang, Y., et al.: Second-order attention network for single image super-resolution. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11065–11074 (2019)
Karras, T., Aittala, M., Hellsten, J., et al.: Training generative adversarial networks with limited data. In: Advances in Neural Information Processing Systems, vol. 33, pp. 12104–12114 (2020)
Niemeyer, M., Geiger, A.: Giraffe: Representing scenes as compositional generative neural feature fields. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11453–11464 (2021)
Or-El, R., Luo, X., Shan, M., et al.: StylesDF: high-resolution 3D-consistent image and geometry generation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 13503–13513 (2022)
Heusel, M., Ramsauer, H., Unterthiner, T., et al.: GANs trained by a two time-scale update rule converge to a local Nash equilibrium. In: Advances in Neural Information Processing Systems, vol. 30 (2017)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Zhou, P., Liu, X., Zhang, B. (2023). SGFNeRF: Shape Guided 3D Face Generation in Neural Radiance Fields. In: Lu, H., Blumenstein, M., Cho, SB., Liu, CL., Yagi, Y., Kamiya, T. (eds) Pattern Recognition. ACPR 2023. Lecture Notes in Computer Science, vol 14408. Springer, Cham. https://doi.org/10.1007/978-3-031-47665-5_20
Download citation
DOI: https://doi.org/10.1007/978-3-031-47665-5_20
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-47664-8
Online ISBN: 978-3-031-47665-5
eBook Packages: Computer ScienceComputer Science (R0)