Skip to main content
SpringerLink
Log in

Locality-constrained framework for face alignment

  • Research Article
  • Published:
Frontiers of Computer Science Aims and scope Submit manuscript

Abstract

Although the conventional active appearance model (AAM) has achieved some success for face alignment, it still suffers from the generalization problem when be applied to unseen subjects and images. To deal with the generalization problem of AAM, we first reformulate the original AAM as sparsity-regularized AAM, which can achieve more compact/better shape and appearance priors by selecting nearest neighbors as the bases of the shape and appearance model. To speed up the fitting procedure, the sparsity in sparsity-regularized AAM is approximated by using the locality (i.e., K-nearest neighbor), and thus inducing the locality-constrained active appearance-model (LC-AAM). The LC-AAM solves a constrained AAM-like fitting problem with the K-nearest neighbors as the bases of shape and appearance model. To alleviate the adverse influence of inaccurate K-nearest neighbor results, the locality constraint is further embedded in the discriminative fitting method denoted as LC-DFM, which can find better K-nearest neighbor results by employing shape-indexed feature, and can also tolerate some inaccurate neighbors benefited from the regression model rather than the generative model in AAM. Extensive experiments on several datasets demonstrate that our methods outperform the state-of-the-arts in both detection accuracy and generalization ability.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. Wiskott L, Fellous J M, Kuiger N, Malsburg C. Face recognition by elastic bunch graph matching. IEEE Transactions on Pattern Analysis and Machine Intelligence, 1997, 19(7): 775–779

    Article  Google Scholar 

  2. Liu X, Kan M, Wu W, Shan S, Chen X. Viplfacenet: an open source deep face recognition SDK. Frontiers of Computer Science, 2017, 11(2): 208–218

    Article  Google Scholar 

  3. Jiang D, Hu Y, Yan S, Zhang L, Zhang H, Gao W. Efficient 3d reconstruction for face recognition. Pattern Recognition, 2005, 38(6): 787–798

    Article  Google Scholar 

  4. Fasel B, Luettin J. Automatic facial expression analysis: a survey. Pattern Recognition, 2003, 36(1): 259–275

    Article  MATH  Google Scholar 

  5. Zhang F, Yu Y, Mao Q, Gou J, Zhan Y. Pose-robust feature learning for facial expression recognition. Frontiers of Computer Science, 2016, 10(5): 832–844

    Article  Google Scholar 

  6. Zheng H, Geng X. Facial expression recognition via weighted group sparsity. Frontiers of Computer Science, 2017, 11(2): 266–275

    Article  Google Scholar 

  7. Cootes T, Taylor C, Cooper D, Graham J. Active shape models-their training and application. Computer Vision and Image Understanding, 1995, 61(1): 38–59

    Article  Google Scholar 

  8. Cootes T, Edwards G, Taylor C. Active appearance models. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2001, 23(6): 681–685

    Article  Google Scholar 

  9. Gross R, Matthews I, Baker S. Generic vs. person specific active appearance models. Image and Vision Computing, 2005, 23(12): 1080–1093

    Google Scholar 

  10. Liu X. Generic face alignment using boosted appearance model. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2007, 1–8

    Google Scholar 

  11. Wu H, Liu X, Doretto G. Face alignment via boosted ranking model. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2008, 1–8

    Google Scholar 

  12. Saragih J, Goecke R. A nonlinear discriminative approach to AAM fitting. In: Proceedings of the IEEE International Conference on Computer Vision. 2007, 1–8

    Google Scholar 

  13. Xiong X, Torre F. Supervised descent method and its applications to face alignment. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2013, 532–539

    Google Scholar 

  14. Asthana A, Zafeiriou S, Cheng S, Pantic M. Robust discriminative response map fitting with constrained local models. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2013, 3444–3451

    Google Scholar 

  15. Lowe D. Distinctive image features from scale-invariant key points. International Journal of Computer Vision, 2004, 60(2): 91–110

    Article  Google Scholar 

  16. Dollár P, Welinder P, Perona P. Cascaded pose regression. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2010, 1078–1085

    Google Scholar 

  17. Tzimiropoulos G. Project-out cascaded regression with an application to face alignment. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2015, 3659–3667

    Google Scholar 

  18. Lee D, Park H, Yoo C. Face alignment using cascade gaussian process regression trees. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2015, 4204–4212

    Google Scholar 

  19. Kazemi V, Sullivan J. One millisecond face alignment with an ensemble of regression trees. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2014, 1867–1874

    Google Scholar 

  20. Cao X, Wei Y, Wen F, Sun J. Face alignment by explicit shape regression. International Journal of Computer Vision, 2014, 107(2): 177–190

    Article  MathSciNet  Google Scholar 

  21. Cootes T, Taylor C. A mixture model for representing shape variation. Image and Vision Computing, 1999, 17(8): 567–573

    Article  Google Scholar 

  22. Maaten L, Hendriks E. Capturing appearance variation in active appearance models. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops. 2010, 34–41

    Google Scholar 

  23. Tipping M E, Bishop C M. Mixtures of probabilistic principal component analyzers. Neural Computation, 1999, 11(2): 443–482

    Article  Google Scholar 

  24. Etyngier P, Segonne F, Keriven R. Shape priors using manifold learning techniques. In: Proceedings of the IEEE International Conference on Computer Vision. 2007, 1–8

    Google Scholar 

  25. Zhang S, Zhan Y, Dewan M, Huang J,Metaxas D, Zhou X. Towards robust and effective shape modeling: sparse shape composition. Medical Image Analysis, 2012, 16(1): 265–277

    Article  Google Scholar 

  26. Ciregan D, Meier U, Schmidhuber J. Multi-column deep neural networks for image classification. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2012, 3642–3649

    Google Scholar 

  27. Krizhevsky A, Sutskever I, Hinton G. Imagenet classification with deep convolutional neural networks. In: Proceedings of the Advances in Neural Information Processing Systems Conference. 2012, 1097–1105

    Google Scholar 

  28. Szegedy C, Toshev A, Erhan D. Deep neural networks for object detection. In: Proceedings of the Advances in Neural Information Processing Systems Conference. 2013, 2553–2561

    Google Scholar 

  29. Girshick R. Fast R-CNN. In: Proceedings of the IEEE International Conference on Computer Vision. 2015, 1440–1448

    Google Scholar 

  30. Long J, Shelhamer E, Darrell T. Fully convolutional networks for semantic segmentation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2015, 3431–3440

    Google Scholar 

  31. Sun Y, Wang X, Tang X. Deep convolutional network cascade for facial point detection. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2013, 3476–3483

    Google Scholar 

  32. Wu Y, Wang Z, Ji Q. Facial feature tracking under varying facial expressions and face poses based on restricted boltzmann machines. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2013, 3452–3459

    Google Scholar 

  33. Zhang J, Shan S, Kan M, Chen X. Coarse-to-fine auto-encoder networks (CFAN) for real-time face alignment. In: Proceedings of the European Conference on Computer Vision. 2014, 1–16

    Google Scholar 

  34. Zhang Z, Luo P, Loy C, Tang X. Learning and transferring multitask deep representation for face alignment. 2014, arXiv preprint arXiv:1408.3967

    Google Scholar 

  35. Trigeorgis G, Snape P, Nicolaou M, Antonakos E, Zafeiriou S. Mnemonic descent method: a recurrent process applied for end-to-end face alignment. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2016, 4177–4187

    Google Scholar 

  36. Jourabloo A, Liu X. Large-pose face alignment via CNN-based dense 3d model fitting. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2016, 4188–4196

    Google Scholar 

  37. Yang Y, Ma Z, Nie F, Chang X, Hauptmann A. Multi-class active learning by uncertainty sampling with diversity maximization. International Journal of Computer Vision, 2015, 113(2): 113–127

    Article  MathSciNet  Google Scholar 

  38. Gao N, Huang S, Chen S. Multi-label active learning by model guided distribution matching. Frontiers of Computer Science, 2016, 10(5): 845–855

    Article  Google Scholar 

  39. Matthews I, Baker S. Active appearance models revisited. International Journal of Computer Vision, 2004, 60(2): 135–164

    Article  Google Scholar 

  40. Zhao X, Shan S, Chai X, Chen X. Locality-constrained active appearance model. In: Proceedings of the Asian Conference on Computer Vision. 2013, 636–647

    Google Scholar 

  41. Dalal N, Triggs B. Histograms of oriented gradients for human detection. In: Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition. 2005, 886–893

    Google Scholar 

  42. Yu K, Zhang T, Gong Y. Nonlinear learning using local coordinate coding. In: Proceedings of the 22nd International Conference on Advances in Neural Information Processing Systems. 2009, 2223–2231

    Google Scholar 

  43. Zhao X, Chai X, Niu Z, Heng C, Shan S. Context constrained facial landmark localization based on discontinuous haar-like feature. In: Proceedings of the IEEE International Conference on Automatic Face and Gesture Recognition. 2011, 673–678

    Google Scholar 

  44. Zhao X, Chai X, Niu Z, Heng C, Shan S. Context modeling for facial landmark detection based on non-adjacent rectangle (NAR) haar-like feature. Image and Vision Computing, 2012, 30(3): 136–146

    Article  Google Scholar 

  45. Sim T, Baker S, Bsat M. The CMU pose, illumination, and expression (PIE) database. In: Proceedings of the IEEE International Conference on Automatic Face and Gesture Recognition. 2002, 46–51

    Google Scholar 

  46. Phillips P, Flynn P, Scruggs T, Bowyer K, Chang J, Hoffman K, Marques J, Min J, Worek W. Overview of the face recognition grand challenge. In: Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition. 2005, 947–954

    Google Scholar 

  47. Phillips P, Wechsler H, Huang J, Rauss P. The feret database and evaluation procedure for face recognition algorithms. Image and Vision Computing, 1998, 16(5): 295–306

    Article  Google Scholar 

  48. Gao W, Cao B, Shan S, Chen X, Zhou D, Zhang X, Zhao D. The CASPEAL large-scale chinese face database and baseline evaluations. IEEE Transactions on Systems, Man and Cybernetics, Part A: Systems and Humans, 2008, 38(1): 149–161

    Article  Google Scholar 

  49. Kumar N, Berg A, Belhumeur P, Nayar S. Attribute and simile classifiers for face verification. In: Proceedings of the IEEE International Conference on Computer Vision. 2009, 365–372

    Google Scholar 

  50. Tian Y, Kanade T, Cohn J. Recognizing action units for facial expression analysis. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2001, 23(2): 97–115

    Article  Google Scholar 

  51. Gu L, Kanade T. A generative shape regularization model for robust face alignment. In: Proceedings of European Conference on Computer Vision. 2008, 413–426

    Google Scholar 

  52. Milborrow S, Nicolls F. Locating facial features with an extended active shape model. In: Proceedings of European Conference on Computer Vision. 2008, 504–513

    Google Scholar 

  53. Saragih J, Lucey S, Cohn J. Deformable model fitting by regularized landmark mean-shifts. International Journal of Computer Vision, 2011, 91(2): 200–215

    Article  MathSciNet  MATH  Google Scholar 

  54. Norouzi M, Punjani A, Fleet D. Fast search in hamming space with multi-index hashing. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2012, 3108–3115

    Google Scholar 

  55. Liu X, Deng C, Lang B, Tao D, Li X. Query-adaptive reciprocal hash tables for nearest neighbor search. IEEE Transactions on Image Processing, 2016, 25(2): 907–919

    Article  MathSciNet  MATH  Google Scholar 

Download references

Acknowledgements

This work was partially supported by the National Natural Science Foundation of China (Grant Nos. 61650202, 61402443, 61672496), and the Strategic Priority Research Program of the CAS (XDB02070004).

Author information

Authors and Affiliations

  1. Key Laboratory of Intelligent Information Processing of Chinese Academy of Sciences, Institute of Computing Technology, Chinese Academy of Sciences, Beijing, 100190, China

    Jie Zhang, Meina Kan, Shiguang Shan, Xiujuan Chai & Xilin Chen

  2. University of Chinese Academy of Sciences, Beijing, 100049, China

    Jie Zhang

  3. Alibaba Group, Hangzhou, 311121, China

    Xiaowei Zhao

Authors
  1. Jie Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Xiaowei Zhao
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Meina Kan
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Shiguang Shan
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Xiujuan Chai
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Xilin Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Shiguang Shan.

Additional information

Jie Zhang received the BS degree at China University of Geosciences, China in 2011. Currently, he is a PhD candidate at the Institute of Computing Technology, Chinese Academy of Sciences. His research interests include deep learning and its application in face alignment, face recognition, object detection and localization.

Xiaowei Zhao received the PhD degree in computer science from the Institute of Computing Technology (ICT), Chinese Academy of Sciences (CAS), China in 2013. He is currently a research engineer with Alibaba Group. His research interests include computer vision, pattern recognition. He especially focuses on face detection and face alignment, image and video analysis, etc.

Meina Kan is an Associate Professor with the Institute of Computing Technology (ICT), Chinese Academy of Sciences (CAS). She received the PhD degree from the University of Chinese Academy of Sciences (CAS), China. Her research mainly focuses on Computer Vision especially face recognition, transfer learning, and deep learning.

Shiguang Shan received MS degree in computer science from the Harbin Institute of Technology, China in 1999, and PhD degree in computer science from the Institute of Computing Technology (ICT), Chinese Academy of Sciences (CAS), China in 2004. He joined ICT, CAS in 2002 and has been a professor since 2010. He is now the Deputy Director of the Key Lab of Intelligent Information Processing of CAS. His research interests cover computer vision, pattern recognition, and machine learning. He especially focuses on face recognition related research topics. He has published more than 200 papers in refereed journals and proceedings.

Xiujuan Chai received the BS, MS, and PhD degrees in computer science from the Harbin Institute of Technology, China in 2000, 2002, and 2007, respectively. She was a Post-doctorial researcher in Nokia Research Center(Beijing), from 2007 to 2009. She joined the Institute of Computing Technology, Chinese Academy Sciences, China in July 2009 and now she is an associate professor. Her research interests cover computer vision, pattern recognition, and multimodal human-computer interaction. She especially focuses on sign language recognition related research topics.

Xilin Chen received the BS, MS, and PhD degrees in computer science from the Harbin Institute of Technology, China in 1988, 1991, and 1994, respectively. He is a professor with the Institute of Computing Technology, Chinese Academy of Sciences (CAS). He has authored one book and over 200 papers in refereed journals and proceedings in the areas of computer vision, pattern recognition, image processing, and multimodal interfaces.

Electronic supplementary material

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Zhang, J., Zhao, X., Kan, M. et al. Locality-constrained framework for face alignment. Front. Comput. Sci. 13, 789–801 (2019). https://doi.org/10.1007/s11704-018-6617-z

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11704-018-6617-z

Keywords

Search

Navigation