Abstract
A complete 3D face reconstruction requires to explicitly model the eyeglasses on the face, which is less investigated in the literature. In this paper, we present an automatic system that recovers the 3D shape of eyeglasses from a single face image with an arbitrary head pose. To achieve this goal, we first trains a neural network to jointly perform glasses landmark detection and segmentation, which carry the sparse and dense glasses shape information respectively for 3D glasses pose estimation and shape recovery. To solve the ambiguity in 2D to 3D reconstruction, our system fully explores the prior knowledge including the relative motion constraint between face and glasses and the planar and symmetric shape prior feature of glasses. From the qualitative and quantitative experiments, we see that our system reconstructs promising 3D shapes of eyeglasses for various poses.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Beeler, T., Bickel, B., Beardsley, P., Sumner, B., Gross, M.: High-quality single-shot capture of facial geometry. ACM Trans. Graph. (ToG), 1–9 (2010)
Cao, C., Weng, Y., Zhou, S., Tong, Y., Zhou, K.: FaceWarehouse: a 3D facial expression database for visual computing. IEEE Trans. Vis. Comput. Graph. (TVCG) 20(3), 413–425 (2013)
De Smet, M., Fransens, R., Van Gool, L.: A generalized EM approach for 3D model based face recognition under occlusions. In: Proceedings of the IEEE Computer Conference on Computer Vision and Pattern Recognition (CVPR), vol. 2, pp. 1423–1430 (2006)
Egger, B., Schneider, A., Blumer, C., Forster, A., Schönborn, S., Vetter, T.: Occlusion-aware 3D morphable face models. In: Proceedings of the British Machine Vision Conference (BMVC), vol. 2, p. 4 (2016)
Feng, Y., Wu, F., Shao, X., Wang, Y., Zhou, X.: Joint 3D face reconstruction and dense alignment with position map regression network. In: Proceedings of the European Conference on Computer Vision (ECCV), pp. 534–551 (2018)
Genova, K., Cole, F., Maschinot, A., Sarna, A., Vlasic, D., Freeman, W.T.: Unsupervised training for 3D morphable model regression. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 8377–8386 (2018)
Huynh, L., et al.: Mesoscopic facial geometry inference using deep neural networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 8407–8416 (2018)
Jiang, L., Zhang, J., Deng, B., Li, H., Liu, L.: 3D face reconstruction with geometry details from a single image. IEEE Trans. Image Process. (TIP) 27(10), 4756–4770 (2018)
Kemelmacher-Shlizerman, I., Basri, R.: 3D face reconstruction from a single image using a single reference face shape. IEEE Trans. Pattern Anal. Mach. Intell. (TPAMI) 33(2), 394–405 (2010)
Kingma, D., Ba, J.: Adam: a method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014)
Lepetit, V., Moreno-Noguer, F., Fua, P.: EPnP: an accurate O(n) solution to the PnP problem. Int. J. Comput. Vis. (IJCV) 81(2), 155–166 (2009)
Li, J., Yang, J.: Eyeglasses try-on based on improved Poisson equations. In: Proceedings of the International Conference on Multimedia Technology (ICMT), pp. 3058–3061 (2011)
Maninchedda, F., Oswald, M.R., Pollefeys, M.: Fast 3D reconstruction of faces with glasses. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 6599–6608 (2017)
Niswar, A., Khan, I.R., Farbiz, F.: Virtual try-on of eyeglasses using 3D model of the head. In: Proceedings of the International Conference on Virtual Reality Continuum and its Applications in Industry (VRCAI), pp. 435–438 (2011)
Park, J.S., Oh, Y.H., Ahn, S.C., Lee, S.W.: Glasses removal from facial image using recursive error compensation. IEEE Trans. Pattern Anal. Mach. Intell. (TPAMI) 27(5), 805–811 (2005)
Richardson, E., Sela, M., Or-El, R., Kimmel, R.: Learning detailed face reconstruction from a single image. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1259–1268 (2017)
Ronneberger, O., Fischer, P., Brox, T.: U-Net: convolutional networks for biomedical image segmentation. In: Proceedings of the International Conference on Medical Image Computing and Computer-Assisted Intervention (MICCAI) (2015)
Roth, J., Tong, Y., Liu, X.: Unconstrained 3D face reconstruction. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2606–2615 (2015)
Smith, W.A., Hancock, E.R.: Recovering facial shape using a statistical model of surface normal direction. IEEE Trans. Patt. Anal. Mach. Intell. (TPAMI) 28(12), 1914–1930 (2006)
Sorkine, O.: Differential representations for mesh processing. Comput. Graph. Forum 4, 789–807 (2006)
Tang, D., Zhang, J., Tang, K., Xu, L., Fang, L.: Making 3D eyeglasses try-on practical. In: Proceedings of the IEEE International Conference on Multimedia and Expo Workshops (ICMEW), pp. 1–6 (2014)
Tewari, A., et al.: Self-supervised multi-level face model learning for monocular reconstruction at over 250 hz. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2549–2559 (2018)
Tran, A.T., Hassner, T., Masi, I., Paz, E., Nirkin, Y., Medioni, G.G.: Extreme 3D face reconstruction: seeing through occlusions. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3935–3944 (2018)
Tran, L., Liu, X.: Nonlinear 3D face morphable model. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 7346–7355 (2018)
Wu, C., Liu, C., Shum, H.Y., Xy, Y.Q., Zhang, Z.: Automatic eyeglasses removal from face images. IEEE Trans. Pattern Anal. Mach. Intell. (TPAMI) 26(3), 322–336 (2004)
Wu, W., Qian, C., Yang, S., Wang, Q., Cai, Y., Zhou, Q.: Look at boundary: a boundary-aware face alignment algorithm. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2018)
Yuan, M., Khan, I., Farbiz, F., Niswar, A., Huang, Z.: A mixed reality system for virtual glasses try-on. In: Proceedings of the International Conference on Virtual Reality Continuum and Its Applications in Industry (VRCAI), December 2011
Yuan, X., Tang, D., Liu, Y., Ling, Q., Fang, L.: Magic glasses: from 2D to 3D. IEEE Trans. Circ. Syst. Video Technol. (TCSVT) 27(4), 843–854 (2016)
Zeng, X., Peng, X., Qiao, Y.: DF2Net: a dense-fine-finer network for detailed 3D face reconstruction. In: Proceedings of the IEEE International Conference on Computer Vision (ICCV), pp. 2315–2324 (2019)
Acknowledgements
This work was supported by the National Key R&D Program of China 2018YFA0704000, the NSFC (No. 61822111, 61727808, 61671268) and Beijing Natural Science Foundation (JQ19015, L182052).
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
1 Electronic supplementary material
Below is the link to the electronic supplementary material.
Rights and permissions
Copyright information
© 2020 Springer Nature Switzerland AG
About this paper
Cite this paper
Wang, Y., Wang, Q., Xu, F. (2020). Eyeglasses 3D Shape Reconstruction from a Single Face Image. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, JM. (eds) Computer Vision – ECCV 2020. ECCV 2020. Lecture Notes in Computer Science(), vol 12370. Springer, Cham. https://doi.org/10.1007/978-3-030-58595-2_23
Download citation
DOI: https://doi.org/10.1007/978-3-030-58595-2_23
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-58594-5
Online ISBN: 978-3-030-58595-2
eBook Packages: Computer ScienceComputer Science (R0)