Skip to main content
SpringerLink
Log in

A low-cost photorealistic CG dataset rendering pipeline for facial landmark localization

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

Face analysis has been a hot research field in computer vision for decades. The dataset is of vital importance for modern machine learning methods. The paper proposes a flexible CG (Computer Graphics) rendering pipe-line for creating facial image datasets together with automatic ground truth labelling. The proposed pipe-line could produce a huge amount of labelled data fast and in low cost compared to traditional dataset creation methods which need high cost hardware and longtime manual ground truth labelling. The paper also proposes a data capture setup in the CG environment for creating the dataset for facial landmark localization. The effectiveness of the proposed method is verified by cross validation with Multi-PIE dataset. For creating a high quality training dataset, some of the varying factors of the dataset should be considered. The paper analyzes a few varying factors for accurate eye landmark localization, such as eye closure levels, eye and eyebrow shapes and wearing glasses. Based on the benefits of the proposed CG rendering pipe-line, the paper implemented a facial landmark localization system across large face rotation by integrating off-the-shelves algorithms. The experiments on Multi-PIE and real persons show that the implemented system could localize facial landmarks accurately across [−90°, +90°] in yaw rotation in real time.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16
Fig. 17
Fig. 18
Fig. 19
Fig. 20

Similar content being viewed by others

References

  1. Blender. Available: https://www.blender.org/

  2. Cao C, Weng Y, Zhou S, Tong Y, Zhou K (2014) Facewarehouse: a 3d facial expression database for visual computing. IEEE Trans Vis Comput Graph 20:413–425

    Article  Google Scholar 

  3. Cao X, Wei Y, Wen F, Sun J (2014) Face alignment by explicit shape regression. Int J Comput Vis 107:177–190

    Article  MathSciNet  Google Scholar 

  4. Cao Z, Simon T, Wei S-E, Sheikh Y (2017) Realtime multi-person 2d pose estimation using part affinity fields. CVPR: 7

  5. Criminisi A, Shotton J (2013) Decision forests for computer vision and medical image analysis: Springer Science & Business Media

  6. Cycles. Available: https://www.cycles-renderer.org/

  7. Deng W, Fang Y, Xu Z, Hu J (2018) Facial landmark localization by enhanced convolutional neural network. Neurocomputing 273:222–229

    Article  Google Scholar 

  8. Dong Y, Wang Y, Yue J, Hu Z (2016) Real time 3D facial movement tracking using a monocular camera. Sensors 16:1157

    Article  Google Scholar 

  9. Dong Y, Zhang Y, Yue J, Hu Z (2016) Comparison of random forest, random ferns and support vector machine for eye state classification. Multimed Tools Appl 75:11763–11783

    Article  Google Scholar 

  10. Fan X, Liu R, Luo Z, Li Y, Feng Y (2018) Explicit shape regression with characteristic number for facial landmark localization. IEEE Transactions on Multimedia 20:567–579

    Article  Google Scholar 

  11. Gao W, Cao B, Shan S, Chen X, Zhou D, Zhang X et al (2008) The CAS-PEAL large-scale Chinese face database and baseline evaluations. IEEE Transactions on Systems, Man, and Cybernetics-Part A: Systems and Humans 38:149–161

    Article  Google Scholar 

  12. Gross R, Matthews I, Cohn J, Kanade T, Baker S (2010) Multi-pie. Image Vis Comput 28:807–813

    Article  Google Scholar 

  13. Guo J-M, Markoni H (2018) Driver drowsiness detection using hybrid convolutional neural network and long short-term memory. Multimed Tools Appl: 1–29

  14. Huang GB, Ramesh M, Berg T, Learned-Miller E (2007) Labeled faces in the wild: a database for studying face recognition in unconstrained environments. Technical Report 07-49, University of Massachusetts, Amherst

  15. H. Joo, H. Liu, L. Tan, L. Gui, B. Nabbe, I. Matthews, et al., (2015) Panoptic studio: a massively multiview system for social motion capture. Proc IEEE Int Conf Comput Vision: 3334–3342

  16. Kasinski A, Florek A, Schmidt A (2008) The PUT face database. Image Process Commun 13:59–64

    Google Scholar 

  17. Kazemi V, Josephine S (2014) One millisecond face alignment with an ensemble of regression trees. 27th IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2014, Columbus, United States, 23 June 2014 through 28 June 2014: 1867-1874

  18. Kendrick C, Tan K, Walker K, Yap M (2018) Towards real-time facial landmark detection in depth data using auxiliary information. Symmetry 10

  19. Koestinger M, Wohlhart P, Roth PM, Bischof H (2011) Annotated facial landmarks in the wild: a large-scale, real-world database for facial landmark localization. Computer vision workshops (ICCV workshops), 2011 IEEE international conference on: 2144–2151.

  20. Le V, Brandt J, Lin Z, Bourdev L, Huang TS (2012) Interactive facial feature localization. European Conference on Computer Vision: 679–692

  21. Liao S, Jain AK, Li SZ (2016) A fast and accurate unconstrained face detector. IEEE Trans Pattern Anal Mach Intell 38:211–223

    Article  Google Scholar 

  22. Ma DS, Correll J, Wittenbrink B (2015) The Chicago face database: a free stimulus set of faces and norming data. Behav Res Methods 47:1122–1135

    Article  Google Scholar 

  23. Marks RJ (2012) Advanced topics in Shannon sampling and interpolation theory. Springer Texts in Electrical Engineering 1

  24. Milborrow S, Morkel J, Nicolls F (2010) The MUCT landmarked face database. Pattern Recognition Association of South Africa 201

  25. Pan Y, Zhou J, Gao Y, Xiong SJAPA (2018) Robust facial landmark localization based on texture and pose correlated initialization

  26. Paysan P, Knothe R, Amberg B, Romdhani S, Vetter T (2009) A 3D face model for pose and illumination invariant face recognition. Advanced video and signal based surveillance, 2009. AVSS'09. Sixth IEEE international conference on: 296–301

  27. Ranjan R, Patel VM, Chellappa R (2017) Hyperface: a deep multi-task learning framework for face detection, landmark localization, pose estimation, and gender recognition. IEEE Trans Pattern Anal Mach Intell

  28. Ranjan R, Sankaranarayanan S, Castillo CD, Chellappa R (2017) An all-in-one convolutional neural network for face analysis. Automatic Face & Gesture Recognition (FG 2017), 2017 12th IEEE international conference on: 17–24

  29. Ren S, Cao X, Wei Y, Sun J (2014) Face alignment at 3000 fps via regressing local binary features. Proc IEEE Conf Comput Vision Pattern Recogn: 1685–1692

  30. Siddiqi MH, Ali R, Khan AM, Kim ES, Kim GJ, Lee S (2015) Facial expression recognition using active contour-based face detection, facial movement-based feature extraction, and non-linear feature selection. Multimedia Systems 21:541–555

    Article  Google Scholar 

  31. Wang Y, Yue J, Dong Y, Hu Z (2016) Robust discriminative regression for facial landmark localization under occlusion. Neurocomputing 214:881–893

    Article  Google Scholar 

  32. Weng R, Lu J, Tan YP, Zhou J (2016) Learning cascaded deep auto-encoder networks for face alignment. IEEE Trans Multimed 18:2066–2078

    Article  Google Scholar 

  33. Xiong X, De la Torre F (2013) Supervised descent method and its applications to face alignment. Computer vision and pattern recognition (CVPR), 2013 IEEE conference on: 532–539

  34. Yu J, Luo C, Yu L, Li L, Wang Z (2016) Facial video coding/decoding at ultra-low bit-rate: a 2D/3D model-based approach. Multimed Tools Appl 75:12021–12041

    Article  Google Scholar 

  35. Zhou E, Fan H, Cao Z, Jiang Y, Yin Q (2013) Extensive facial landmark localization with coarse-to-fine convolutional network Cascade. Presented at the 2013 IEEE international conference on computer vision workshops

Download references

Acknowledgements

The work was partially supported by the National Natural Science Foundation of China under Grant No. 61873189, the Natural Science Foundation of Shanghai under Grant No. 18ZR1442500 and the Fundamental Research Funds for the Central Universities.

Author information

Authors and Affiliations

  1. Tongji University, Shanghai, China

    Yanchao Dong, Minjing Lin, Jiguang Yue & Liang Shi

Authors
  1. Yanchao Dong
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Minjing Lin
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Jiguang Yue
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Liang Shi
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Yanchao Dong.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Dong, Y., Lin, M., Yue, J. et al. A low-cost photorealistic CG dataset rendering pipeline for facial landmark localization. Multimed Tools Appl 78, 22397–22420 (2019). https://doi.org/10.1007/s11042-019-7516-5

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-019-7516-5

Keywords

Search

Navigation