Skip to main content
SpringerLink
Log in

Annotated face model-based alignment: a robust landmark-free pose estimation approach for 3D model registration

  • Original Paper
  • Published:
Machine Vision and Applications Aims and scope Submit manuscript

Abstract

Registering a 3D facial model onto a 2D image is important for constructing pixel-wise correspondences between different facial images. The registration is based on a 3 \(\times \) 4 dimensional projection matrix, which is obtained from pose estimation. Conventional pose estimation approaches employ facial landmarks to determine the coefficients inside the projection matrix and are sensitive to missing or incorrect landmarks. In this paper, a landmark-free pose estimation method is presented. The method can be used to estimate the matrix when facial landmarks are not available. Experimental results show that the proposed method outperforms several landmark-free pose estimation methods and achieves competitive accuracy in terms of estimating pose parameters. The method is also demonstrated to be effective as part of a 3D-aided face recognition pipeline (UR2D), whose rank-1 identification rate is competitive to the methods that use landmarks to estimate head pose.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10

Similar content being viewed by others

Notes

  1. In the description of RDD, index \(\mu \) is omitted for clarity.

  2. A few imprecise landmark annotations were rectified in these model/database in this journal version.

Abbreviations

GIS:

Geometry image space

AFM:

Annotated face model

T-AFM:

Texture of annotated face model

RDD:

Rotation determined decomposition

TBB:

Target bounding box

SDM:

Supervised descent method

GSDM:

Global supervised descent method

RSSDM:

Random subspace supervised descent method

2dSC:

Two-dimensional sparse coding

G3D:

Generic 3D model

PS3D:

Personalized 3D model

E-AFMA:

Ex-annotated face model-based alignment

AFMA:

Annotated face model-based alignment

References

  1. 3dMD 3dMD: 3D imaging systems and software (2012). http://www.3dmd.com/

  2. Abiantun, R., Prabhu, U., Savvides, M.: Sparse feature extraction for pose-tolerant face recognition. IEEE Trans. Pattern Anal. Mach. Intell. 36(10), 2061–2073 (2014)

    Article  Google Scholar 

  3. Asthana, A., Zafeiriou, S., Cheng, S., Pantic, M.: Robust discriminative response map fitting with constrained local models. In: Proceedings of IEEE conference on computer vision and pattern recognition, Portland, OR, pp. 3444–3451 (2013)

  4. Asthana, A., Zafeiriou, S., Cheng, S., Pantic, M.: Incremental face alignment in the wild. In: Proceedings of IEEE conference on computer vision and pattern recognition, Columbus, OH, pp. 1859–1866 (2014)

  5. Asthana, A., Zafeiriou, S., Tzimiropoulos, G., Cheng, S., Pantic, M.: From pixels to response maps: discriminative image filtering for face alignment in the wild. IEEE Trans. Pattern Anal. Mach. Intell. 37(6), 1312–1320 (2015)

    Article  Google Scholar 

  6. Ba, S.O., Odobez, J.M.: Recognizing visual focus of attention from head pose in natural meetings. IEEE Int. Conf. Syst. Man Cybern. 39(1), 16–33 (2009)

    Article  Google Scholar 

  7. Balasubramanian, V., Ye, J., Panchanathan, S.: Biased manifold embedding: a framework for person-independent head pose estimation. In: Proceedings of IEEE conference on computer vision and pattern recognition, Minneapolis, MN, pp. 1–7 (2007)

  8. Bouaziz, S., Wang, Y., Pauly, M.: Online modeling for realtime facial animation. ACM Trans. Graph. 32(4), 40 (2013)

    Article  MATH  Google Scholar 

  9. Dalal, N., Triggs, B.: Histograms of oriented gradients for human detection. In: Proceedings of IEEE conference on computer vision and pattern recognition, San Diego, CA, pp. 886–893 (2005)

  10. Divvala, SK., Efros, AA., Hebert, M.: How important are deformable parts in the deformable parts model? In: Proceedings of European conference on computer vision (workshop), Florence, Italy, pp. 31–40 (2012)

  11. Dou, P., Wu, Y., Shishir, S.K., Kakadiaris, I.A.: Benchmarking 3D pose estimation for face recognition. In: Proceedings of IEEE international conference on pattern recognition, Stockholm, Sweden, pp. 190–195 (2014)

  12. Dou, P., Zhang, L., Wu, Y., Shah, S.K., Kakadiaris, I.A.: Pose-robust face signature for multi-view face recognition. In: Proceedings of IEEE international conference on biometrics: theory, applications and systems, Arlington, VA, pp. 1–8 (2015)

  13. Drucker, H., Burges, C., Kaufman, L., Smola, A., Vapnik, V.: Support vector regression machines. In: Advances in neural information processing systems, Denver, CO, pp. 155–161 (1997)

  14. Geng, X., Xia, Y.: Head pose estimation based on multivariate label distribution. In: Proceedings of IEEE conference on computer vision and pattern recognition, Columbus, OH, pp. 1837–1842 (2014)

  15. Gourier, N., Hall, D., Crowley, J.: Estimating face orientation from robust detection of salient facial structures. In: Proceedings of international workshop on visual observation of deictic gestures, Cambridge, UK, pp. 1–9 (2004)

  16. Guo, G., Fu, Y., Dyer, CR., Huang, T.: Head pose estimation: classification or regression? In: Proceedings of international conference on pattern recognition, Tampa, FL, pp. 1–4 (2008)

  17. Hara, K., Chellappa, R.: Growing regression forests by classification: applications to object pose estimation. In: Proceedings of European conference on computer vision, Zurich, Switzerland (2014)

  18. Hartley, R., Zisserman, A.: Multiple View Geometry in Computer Vision. Cambridge University Press, Cambridge (2000)

    MATH  Google Scholar 

  19. Hsu, G., Peng, H.: Face recognition across poses using a single 3D reference model. In: Proceedings of IEEE conference on computer vision and pattern recognition, Portland, OR, pp. 869–874 (2013)

  20. Huang, D., Storer, M., De la Torre, F., Bischof, H.: Supervised local subspace learning for continuous head pose estimation. In: Proceedings of IEEE conference on computer vision and pattern recognition, Colorado Springs, CO, pp. 2921–2928 (2011)

  21. Jeni, L., Cohn, J., Kanade, T.: Dense 3D face alignment from 2D videos in real-time. In: Proceedings of IEEE international conference and workshops on automatic face and gesture recognition, Ljubljana, Slovenia, vol. 1, pp. 1–8 (2015)

  22. Kakadiaris, I.A., Passalis, G., Toderici, G., Murtuza, M., Lu, Y., Karampatziakis, N., Theoharis, T.: Three-dimensional face recognition in the presence of facial expressions: an annotated deformable model approach. IEEE Trans. Pattern Anal. Mach. Intell. 29(4), 640–649 (2007)

    Article  Google Scholar 

  23. Kakadiaris, I.A., Toderici, G., Evangelopoulos, G., Passalis, G., Zhao, X., Shah, S.K., Theoharis, T.: 3D–2D face recognition with pose and illumination normalization. Comput. Vis. Image Underst. 154, 137–151 (2017)

    Article  Google Scholar 

  24. Kemelmacher-Shlizerman, I., Seitz, S.: Face reconstruction in the wild. In: Proceedings of IEEE international conference on computer vision, Barcelona, Spain, pp. 1746–1753 (2011)

  25. Ma, B., Li, A., Chai, X., Shan, S.: CovGa: a novel descriptor based on symmetry of regions for head pose estimation. Neurocomputing 143, 97–108 (2014)

    Article  Google Scholar 

  26. Masi, I., Lisanti, G., Bagdanov, A., Pala, P., Bimbo, A.: Using 3D models to recognize 2D faces in the wild. In: Proceedings of IEEE conference on computer vision and pattern recognition, Portland, OR, pp. 775–780 (2013)

  27. Phillips, P.J., Scruggs, W.T., O’Toole, A.J., Flynn, P.J., Bowyer, K.W., Schott, C.L., Sharpe, M.: FRVT 2006 and ICE 2006 large-scale experimental results. IEEE Trans. Pattern Anal. Mach. Intell. 32(5), 831–846 (2010)

    Article  Google Scholar 

  28. Taigman, Y., Yang, M., Ranzato, M., Wolf, L.: DeepFace: closing the gap to human-level performance in face verification. In: Proceedings of IEEE conference on computer vision and pattern recognition, Columbus, OH, pp. 1701–1708 (2014)

  29. Tan, X., Triggs, B.: Enhanced local texture feature sets for face recognition under difficult lighting conditions. IEEE Trans. Image Process. 19(6), 1635–1650 (2010)

    Article  MathSciNet  MATH  Google Scholar 

  30. Tibshirani, R.: Regression shrinkage and selection via the lasso. J. R. Stat. Soc. B. 58(1), 267–288 (1996)

  31. Toderici, G., Passalis, G., Zafeiriou, S., Tzimiropoulos, G., Petrou, M., Theoharis, T., Kakadiaris, I.A.: Bidirectional relighting for 3D-aided 2D face recognition. In: Proceedings of IEEE conference on computer vision and pattern recognition, San Francisco, CA, pp. 2721–2728 (2010)

  32. Toderici, G., Evangelopoulos, G., Fang, T., Theoharis, T., Kakadiaris, I.A.: UHDB11 database for 3D–2D face recognition. In: Proceedings of Pacific-Rim symposium on image and video technology, Guanajuato, Mexico, pp. 73–86 (2013)

  33. Vu, P.V., Chandler, D.: A fast wavelet-based algorithm for global and local image sharpness estimation. IEEE Signal Process. Lett. 19(7), 423–426 (2012)

    Article  Google Scholar 

  34. Wagner, A., Wright, J., Ganesh, A., Zhou, Z., Mobahi, H., Ma, Y.: Toward a practical face recognition system: robust alignment and illumination by sparse representation. IEEE Trans. Pattern Anal. Mach. Intell. 34(2), 372–386 (2012)

    Article  Google Scholar 

  35. Wang, C., Song, X.: Robust head pose estimation via supervised manifold learning. Neural Netw. 53, 15–25 (2014)

    Article  MATH  Google Scholar 

  36. Weise, T., Bouaziz, S., Li, H., Pauly, M.: Realtime performance-based facial animation. ACM Trans. Graph. 30(4), 77 (2011)

    Article  Google Scholar 

  37. Wu, Y., Xu, X., Shah, S.K., Kakadiaris, I.A.: Towards fitting a 3D dense facial model to a 2D image: a landmark-free approach. In: Proceedings of international conference on biometrics: theory, applications and systems, Arlington, VA, pp. 1–8 (2015)

  38. Wu, Y., Shah, S.K., Kakadiaris, I.A.: Rendering or normalization? An analysis of the 3D-aided pose-invariant face recognition. In: Proceedings of IEEE international conference on identity, security and behavior analysis, Sendai, Japan, pp. 1–8 (2016)

  39. Xiong, X., De la Torre, F.: Supervised descent method and its applications to face alignment. In: Proceedings of IEEE conference on computer vision and pattern recognition, Portland, OR, pp. 532–539 (2013)

  40. Xiong, X., De la Torre, F.: Global supervised descent method. In: Proceedings of IEEE conference on computer vision and pattern recognition, Boston, MA, pp. 2664–2673 (2015)

  41. Yang, H., Jia, X., Patras, I., Chan, K.P.: Random subspace supervised descent method for regression problems in computer vision. IEEE Trans. Signal Process. Lett. 22(10), 1816–1820 (2015)

    Article  Google Scholar 

  42. Yang, H., Mou, W., Zhang, Y., Patras, I., Gunes, H., Robinson, P.: Face alignment assisted by head pose estimation. In: Proceedings of British machine vision conference, Swansea, UK, pp. 1–13 (2015)

  43. Zhao, X., Kim, T.K., Luo, W.: Unified face analysis by iterative multi-output random forests. In: Proceedings of IEEE conference on computer vision and pattern recognition, IEEE, Columbus, OH, pp. 1765–1772 (2014)

  44. Zhen, X., Wang, Z., Yu, M., Li, S.: Supervised descriptor learning for multi-output regression. In: Proceedings of IEEE conference on computer vision and pattern recognition, Boston, MA, pp. 1211–1218 (2015)

  45. Zhu, X., Ramanan, D.: Face detection, pose estimation, and landmark localization in the wild. In: Proceedings of IEEE conference on computer vision and pattern recognition, Providence, RI, pp. 2879–2886 (2012)

  46. Zhu, X., Lei, Z., Yan, J., Yi, D., Li, S.: High-fidelity pose and expression normalization for face recognition in the wild. In: Proceedings of IEEE conference on computer vision and pattern recognition, Boston, MA, pp. 787–796 (2015)

Download references

Author information

Authors and Affiliations

  1. Computational Biomedicine Lab, Department of Computer Science, University of Houston, 4849 Calhoun Road, Houston, TX, 77004, USA

    Yuhang Wu, Shishir K. Shah & Ioannis A. Kakadiaris

Authors
  1. Yuhang Wu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Shishir K. Shah
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Ioannis A. Kakadiaris
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Ioannis A. Kakadiaris.

Additional information

This material is based upon work supported by the U.S. Department of Homeland Security under Grant Award Number 2015-ST-061-BSH001. This grant is awarded to the Borders, Trade, and Immigration (BTI) Institute: A DHS Center of Excellence led by the University of Houston, and includes support for the project “Image and Video Person Identification in an Operational Environment: Phase I” awarded to the University of Houston. The views and conclusions contained in this document are those of the authors and should not be interpreted as necessarily representing the official policies, either expressed or implied, of the U.S. Department of Homeland Security.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Wu, Y., Shah, S.K. & Kakadiaris, I.A. Annotated face model-based alignment: a robust landmark-free pose estimation approach for 3D model registration. Machine Vision and Applications 29, 375–391 (2018). https://doi.org/10.1007/s00138-017-0887-6

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00138-017-0887-6

Keywords

Search

Navigation