Abstract
Accurate modeling of facial hair is crucial not only for reconstructing a clean-shaven face but also for enhancing realism when creating digital human avatars. Previous methods have primarily focused on reconstructing sparse facial hairs from static scans, and more recently, efforts have been made to track facial performances. However, there is still room for improvement in reconstructing thick hairs. In this paper, we address the challenges of facial hair reconstruction and tracking, enhancing the realism and detail of facial hair in digital human avatars. For facial hair reconstruction, we propose a method that combines line-based multi-view stereo with line segment matching to recover a dense hair point cloud. From this point cloud, hair strands are extracted using the forward Euler method. For tracking, we introduce a space-time optimization method that registers the reference facial hair strands to subsequent frames, taking into account both the global shape and the motion between frames. After capturing the facial hair structures, we refine the underlying skin meshes by replacing the noisy hair region points with strand roots. We conducted experiments on various examples captured under different systems, demonstrating that our facial hair capture method outperforms previous methods in terms of accuracy and completeness. Our method provides a comprehensive and accurate solution for capturing and modeling facial hair in various facial performance scenarios.
Similar content being viewed by others
Explore related subjects
Discover the latest articles, news and stories from top researchers in related subjects.Data availability
No datasets were generated or analyzed during the current study.
Notes
To protect the privacy of testers, we applied a black mosaic over the eye area when displaying the reference images in figures. The same approach was utilized in subsequent figures.
References
Beeler, T., Bickel, B., Beardsley, P., Sumner, B., Gross, M.: High-quality single-shot capture of facial geometry. In: ACM SIGGRAPH 2010 Papers, pp. 1–9 (2010)
Bradley, D., Heidrich, W., Popa, T., Sheffer, A.: High resolution passive facial performance capture. In: ACM SIGGRAPH 2010 Papers, pp. 1–10 (2010)
Debevec, P., Hawkins, T., Tchou, C., Duiker, H.P., Sarokin, W., Sagar, M.: Acquiring the reflectance field of a human face. In: Proceedings of the 27th Annual Conference on Computer Graphics and Interactive Techniques, pp. 145–156 (2000)
Riviere, J., Gotardo, P., Bradley, D., Ghosh, A., Beeler, T.: Single-shot high-quality facial geometry and skin appearance capture (2020)
Beeler, T., Hahn, F., Bradley, D., Bickel, B., Beardsley, P., Gotsman, C., Sumner, R.W., Gross, M.: High-quality passive facial performance capture using anchor frames. In: ACM SIGGRAPH 2011 Papers, pp. 1–10 (2011)
Fyffe, G., Nagano, K., Huynh, L., Saito, S., Busch, J., Jones, A., Li, H., Debevec, P.: Multi-view stereo on consistent face topology. In: Computer Graphics Forum, vol. 36, pp. 295–309. Wiley Online Library (2017)
Li, H., Lin, Y., Liu, X.: Full head performance capture using multi-scale mesh propagation. In: Pattern Recognition and Computer Vision: 5th Chinese Conference, PRCV 2022, Shenzhen, China, November 4–7, 2022, Proceedings, Part III. Springer, pp. 56–67 (2022)
Bleyer, M., Rhemann, C., Rother, C.: Patchmatch stereo-stereo matching with slanted support windows. In: Bmvc, vol. 11, pp. 1–11 (2011)
Galliani, S., Lasinger, K., Schindler, K.: Massively parallel multiview stereopsis by surface normal diffusion. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 873–881 (2015)
Yao, Y., Luo, Z., Li, S., Fang, T., Quan, L.: Mvsnet: depth inference for unstructured multi-view stereo. In: Proceedings of the European Conference on Computer Vision (ECCV), pp. 767–783 (2018)
Wu, F., Bao, L., Chen, Y., Ling, Y., Song, Y., Li, S., Ngan, K.N., Liu, W.: Mvf-net: multi-view 3D face morphable model regression. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 959–968 (2019)
Lattas, A., Moschoglou, S., Gecer, B., Ploumpis, S., Triantafyllou, V., Ghosh, A., Zafeiriou, S.: AvatarMe: realistically renderable 3D facial reconstruction “in-the-wild”. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 760–769 (2020)
Gafni, G., Thies, J., Zollhofer, M., Nießner, M.: Dynamic neural radiance fields for monocular 4D facial avatar reconstruction. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8649–8658 (2021)
Paris, S., Briceno, H.M., Sillion, F.X.: Capture of hair geometry from multiple images. ACM Trans. Graph. (TOG) 23(3), 712–719 (2004)
Paris, S., Chang, W., Kozhushnyan, O.I., Jarosz, W., Matusik, W., Zwicker, M., Durand, F.: Hair photobooth: geometric and photometric acquisition of real hairstyles. ACM Trans. Graph. 27(3), 30 (2008)
Luo, L., Li, H., Rusinkiewicz, S.: Structure-aware hair capture. ACM Trans. Graph. (TOG) 32(4), 1–12 (2013)
Xu, Z., Wu, H.T., Wang, L., Zheng, C., Tong, X., Qi, Y.: Dynamic hair capture using spacetime optimization. To appear in ACM TOG, vol. 33, p. 6 (2014)
Zhang, M., Chai, M., Wu, H., Yang, H., Zhou, K.: A data-driven approach to four-view image-based hair modeling. ACM Trans. Graph. 36(4), 156–1 (2017)
Hu, L., Ma, C., Luo, L., Li, H.: Single-view hair modeling using a hairstyle database. ACM Trans. Graph. (ToG) 34(4), 1–9 (2015)
Nam, G., Wu, C., Kim, M.H., Sheikh, Y.: Strand-accurate multi-view hair capture. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 155–164 (2019)
Rosu, R.A., Saito, S., Wang, Z., Wu, C., Behnke, S., Nam, G.: Neural strands: Learning hair geometry and appearance from multi-view images. In: European Conference on Computer Vision, pp. 73–89. Springer (2022)
Zhou, Y., Hu, L., Xing, J., Chen, W., Kung, H.W., Tong, X., Li, H.: Hairnet: single-view hair reconstruction using convolutional neural networks. In: Proceedings of the European Conference on Computer Vision (ECCV), pp. 235–251 (2018)
Rotger, G., Moreno-Noguer, F., Lumbreras, F., Agudo, A.: Single view facial hair 3D reconstruction. In: Pattern Recognition and Image Analysis: 9th Iberian Conference, IbPRIA 2019, Madrid, Spain, July 1–4, 2019, Proceedings, Part I, vol. 9, pp. 423–436. Springer (2019)
Yang, L., Shi, Z., Zheng, Y., Zhou, K.: Dynamic hair modeling from monocular videos using deep neural networks. ACM Trans. Graph. (TOG) 38(6), 1–12 (2019)
Sun, T., Nam, G., Aliaga, C., Hery, C., Ramamoorthi, R.: Human hair inverse rendering using multi-view photometric data (2021)
Fyffe, G.: High fidelity facial hair capture. In: ACM SIGGRAPH 2012 Talks, p. 1 (2012)
Beeler, T., Bickel, B., Noris, G., Beardsley, P., Marschner, S., Sumner, R.W., Gross, M.: Coupled 3D reconstruction of sparse facial hair and skin. ACM Trans. Graph. (ToG) 31(4), 1–10 (2012)
Winberg, S., Zoss, G., Chandran, P., Gotardo, P., Bradley, D.: Facial hair tracking for high fidelity performance capture. ACM Trans. Graph. (TOG) 41(4), 1–12 (2022)
Olszewski, K., Ceylan, D., Xing, J., Echevarria, J., Chen, Z., Chen, W., Li, H.: Intuitive, interactive beard and hair synthesis with generative models. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7446–7456 (2020)
Xiao, Q., Zhang, H., Zhang, Z., Wu, Y., Wang, L., Jin, X., Jiang, X., Yang, Y.L., Shao, T., Zhou, K.: Eyelashnet: a dataset and a baseline method for eyelash matting. ACM Trans. Graph. (TOG) 40(6), 1–17 (2021)
Fyffe, G., Graham, P., Tunwattanapong, B., Ghosh, A., Debevec, P.: Near-instant capture of high-resolution facial geometry and reflectance. In: Computer Graphics Forum, vol. 35, pp. 353–363. Wiley Online Library (2016)
Yang, H., Zhu, H., Wang, Y., Huang, M., Shen, Q., Yang, R., Cao, X.: Facescape: a large-scale high quality 3d face dataset and detailed riggable 3d face prediction. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 601–610 (2020)
Knapitsch, A., Park, J., Zhou, Q.Y., Koltun, V.: Tanks and temples: benchmarking large-scale scene reconstruction. ACM Trans. Graph. (ToG) 36(4), 1–13 (2017)
Acknowledgements
This research was supported by the National Natural Science Foundation of China (No. 62032011).
Author information
Authors and Affiliations
Contributions
HL helped in conceptualization, methodology, experiments, writing—original draft; XL contributed to conceptualization, funding acquisition, supervision, review and editing.
Corresponding author
Ethics declarations
Conflict of interest
The authors declare that they have no conflict of interest.
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Supplementary Information
Below is the link to the electronic supplementary material.
Supplementary file 1 (mp4 42551 KB)
Rights and permissions
Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.
About this article
Cite this article
Li, H., Liu, X. Strand-accurate multi-view facial hair reconstruction and tracking. Vis Comput 40, 4713–4724 (2024). https://doi.org/10.1007/s00371-024-03465-5
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s00371-024-03465-5