Skip to main content
SpringerLink
Log in

Incremental Clustering-Based Facial Feature Tracking Using Bayesian ART

  • Published:
Neural Processing Letters Aims and scope Submit manuscript

Abstract

Person-independent, emotion specific facial feature tracking have been of interest in the machine vision society for decades. Among various methods, the constrained local model (CLM) has shown significant results in person-independent feature tracking. In 63this paper, we propose an automatic, efficient, and robust method for emotion specific facial feature detection and tracking from image sequences. Considering a 17-point feature model on the frontal face region, the proposed tracking framework incorporates CLM with two incremental clustering algorithms to increase robustness and minimize tracking error during feature tracking. The Patch Clustering algorithm is applied to build an appearance model of face frames by organizing previously encountered similar patches into clusters while the shape Clustering algorithm is applied to build a structure model of face shapes by organizing previously encountered similar shapes into clusters followed by Bayesian adaptive resonance theory (ART). Both models are used to explore the similar features/shapes in the successive images. The clusters in each model are built and updated incrementally and online, controlled by amount of facial muscle movement. The overall performance of the proposed incremental clustering-based facial feature tracking (ICFFT) is evaluated using the FGnet database and the extended Cohn-Kanade (CK+) database. ICFFT demonstrates better results than baseline-method CLM and provides robust tracking as well as improved localization accuracy of emotion specific facial features tracking.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12

Similar content being viewed by others

References

  1. Pantic M, Rothkrantz LJ (2003) Toward an affect-sensitive multimodal human-computer interaction. Proceedings of the IEEE 91(9):1370–1390

    Article  Google Scholar 

  2. Frantzidis CA et al (2010) Toward emotion aware computing: an integrated approach using multichannel neurophysiological recordings and affective visual stimuli. Information Technology in Biomedicine, IEEE Transactions on 14(3):589–597

    Article  Google Scholar 

  3. Moreno-Garcia J et al (2010) Video sequence motion tracking by fuzzification techniques. Appl Soft Comput 10(1):318–331

    Article  Google Scholar 

  4. Lisetti C et al (2003) Developing multimodal intelligent affective interfaces for tele-home health care. Int J Hum-Comput Stud 59(1):245–255

    Article  Google Scholar 

  5. Luneski A et al (2008) Affective computer-aided learning for autistic children. In: WOCCI

  6. Tie Y, Guan L (2013) Automatic landmark point detection and tracking for human facial expressions. EURASIP J Image Video Process 2013(1):1–15

    Article  MathSciNet  Google Scholar 

  7. Friedman HS (1979) Nonverbal communication between patients and medical practitioners. J Soc Issues 35(1):82–99

    Article  Google Scholar 

  8. Ekman P, Friesen WV (1978) Facial action coding system: a technique for the measurement of facial movement. Consulting Psychologists Press, Palo Alto, CA

    Google Scholar 

  9. Hamm J et al (2011) Automated facial action coding system for dynamic analysis of facial expressions in neuropsychiatric disorders. J Neurosci Methods 200(2):237–256

    Article  Google Scholar 

  10. Asadpour V, Homayounpour MM, Towhidkhah F (2011) Audio-visual speaker identification using dynamic facial movements and utterance phonetic content. Appl Soft Comput 11(2):2083–2093

    Article  Google Scholar 

  11. Azim T, Jaffar MA, Mirza AM (2014) Fully automated real time fatigue detection of drivers through fuzzy expert systems. Appl Soft Comput 18:25–38

    Article  Google Scholar 

  12. Chien J-C et al (2013) An integrated driver warning system for driver and pedestrian safety. Appl Soft Comput 13(11):4413–4427

    Article  Google Scholar 

  13. Shan C, Gong S, McOwan PW (2009) Facial expression recognition based on local binary patterns: a comprehensive study. Image Vision Comput 27(6):803–816

    Article  Google Scholar 

  14. Cristinacce D, Cootes T (2008) Automatic feature localisation with constrained local models. Pattern Recognit 41(10):3054–3067

    Article  MATH  Google Scholar 

  15. Islam MN, Loo CK (2014) Geometric feature-based facial emotion recognition using two-stage fuzzy reasoning model. In: Neural information processing, Springer

  16. Nuevo J, Bergasa LM, Jiménez P (2010) RSMAT: robust simultaneous modeling and tracking. Pattern Recognit Lett 31(16):2455–2463

    Article  Google Scholar 

  17. Vigdor B, Lerner B (2007) The bayesian artmap. IEEE Trans Neural Netw 18(6):1628–1644

    Article  Google Scholar 

  18. Picard RW (2003) Affective computing: challenges. Int J Hum-Comput Stud 59(1):55–64

    Article  Google Scholar 

  19. Vezzetti E, Marcolin F (2012) 3D human face description: iandmarks measures and geometrical features. Image Vision Comput 30(10):698–712

    Article  Google Scholar 

  20. Tian Y-L, Kanade T, Cohn JF (2001) Recognizing action units for facial expression analysis. IEEE Trans Pattern Anal Mach Intell 23(2):97–115

    Article  Google Scholar 

  21. Draper BA et al (2003) Recognizing faces with PCA and ICA. Comput Vis Image Underst 91(1):115–137

    Article  Google Scholar 

  22. Zhao G, Pietikainen M (2007) Dynamic texture recognition using local binary patterns with an application to facial expressions. IEEE Trans Pattern Anal Mach Intell 29(6):915–928

    Article  Google Scholar 

  23. Krisshna AN (2014) Face recognition using transform domain feature extraction and PSO-based feature selection. Appl Soft Comput 22:141–161

    Article  Google Scholar 

  24. Tong Y et al (2007) Robust facial feature tracking under varying face pose and facial expression. Pattern Recognit 40(11):3195–3208

    Article  MATH  Google Scholar 

  25. Lien JJ-J et al (2000) Detection, tracking, and classification of action units in facial expression. Robot Auton Syst 31(3):131–146

    Article  Google Scholar 

  26. Tsalakanidou F, Malassiotis S (2010) Real-time 2D+ 3D facial action and expression recognition. Pattern Recognit 43(5):1763–1775

    Article  Google Scholar 

  27. Dryden IL, Mardia KV (1998) Statistical shape analysis, vol 4. Wiley, New York

    MATH  Google Scholar 

  28. Cootes TF et al (1995) Active shape models-their training and application. Comput Vis Image Underst 61(1):38–59

    Article  Google Scholar 

  29. Seshadri K, Savvides M (2012) An analysis of the sensitivity of active shape models to initialization when applied to automatic facial landmarking. IEEE Trans Inf Forensics Secur 7(4):1255–1269

    Article  Google Scholar 

  30. Milborrow S, Nicolls F (2008) Locating facial features with an extended active shape model. In: Computer vision–ECCV 2008. Springer, pp 504–513

  31. Xiong X, Torre F (2013) Supervised descent method and its applications to face alignment. In: Proceedings of the IEEE conference on computer vision and pattern recognition. IEEE

  32. Cao X, Wei Y, Wen F, Sun J (2014) Face alignment by explicit shape regression. Int J Comput Vis 107(2):177–190

    Article  MathSciNet  Google Scholar 

  33. Yang J, Deng J, Zhang K, Liu Q (2015) Facial shape tracking via spatio-temporal cascade shape regression. In: Proceedings of the IEEE international conference on comput vision workshops. IEEE

  34. Liu Q, Deng J, Tao D (2015) Dual sparse constrained cascade regression for robust face alignment. IEEE Trans Image Process 25(2):700–712

    Article  MathSciNet  Google Scholar 

  35. Deng J, Liu Q, Yang J, Tao D (2015) M 3 CSR: multi-view, multi-scale and multi-component cascade shape regression. Image Vis Comput 47:19–26

    Article  Google Scholar 

  36. Ding C, Choi J, Tao D, Davis L (2014) Multi-directional multi-level dual-cross patterns for robust face recognition. IEEE Trans Pattern Anal Mach Intell 38(3):518–531

    Article  Google Scholar 

  37. Ding C, Xu C, Tao D (2015) Multi-task pose-invariant face recognition. IEEE Trans Image Process 24(3):980–993

    Article  MathSciNet  Google Scholar 

  38. Ding C, Tao D (2015) A comprehensive survey on pose-invariant face recognition. ACM Transactions on intelligent systems and technology (TIST), 7(3): Article No. 37, arXiv preprint arXiv:1502.04383

  39. Solomon CJ, Gibson SJ, Mist JJ (2013) Interactive evolutionary generation of facial composites for locating suspects in criminal investigations. Appl Soft Comput 13(7):3298–3306

    Article  Google Scholar 

  40. Cootes TF, Edwards GJ, Taylor CJ (2001) Active appearance models. IEEE Trans Pattern Anal Mach Intell 23(6):681–685

    Article  Google Scholar 

  41. Cho K-S, Kim Y-G, Lee Y-B (2006) Real-time expression recognition system using active appearance model and EFM. In: 2006 International conference on computational intelligence and security. IEEE

  42. Chen Y, Hua C, Bai R (2014) Regression-based active appearance model initialization for facial feature tracking with missing frames. Pattern Recognit Lett 38:113–119

    Article  Google Scholar 

  43. Haslam J, Taylor CJ, Cootes T (1994) A probabilistic fitness measure for deformable template models. In: BMVC, Citeseer

  44. Burl MC, Leung TK, Perona P (1995) Face localization via shape statistics. In: International workshop on automatic face and gesture recognition

  45. Lucey S et al (2010) Non-rigid face tracking with enforced convexity and local appearance consistency constraint. Image Vis Comput 28(5):781–789

    Article  Google Scholar 

  46. Saragih JM, Lucey S, Cohn JF (2009) Face alignment through subspace constrained mean-shifts. In: IEEE 12th international conference on computer vision. IEEE

  47. Lucas BD, Kanade T (1981) An iterative image registration technique with an application to stereo vision. In: IJCAI

  48. Matthews I, Ishikawa T, Baker S (2004) The template update problem. IEEE Trans Pattern Anal Mach Intell 26(6):810–815

    Article  Google Scholar 

  49. Xu C, Tao D, Xu C (2015) Multi-view self-paced learning for clustering. In: Proceedings of the 24th international conference on Artificial Intelligence

  50. Dowson ND, Bowden R (2005) Simultaneous modeling and tracking (smat) of feature sets. In: CVPR 2005, IEEE computer society conference on computer vision and pattern recognition. IEEE

  51. Gross R et al (2010) Multi-pie. Image Vis Comput 28(5):807–813

    Article  Google Scholar 

  52. Nuevo J et al (2011) Face tracking with automatic model construction. Image Vis Comput 29(4):209–218

    Article  Google Scholar 

  53. Viola P, Jones M (2001) Rapid object detection using a boosted cascade of simple features. In: CVPR 2001, Proceedings of the 2001 IEEE computer society conference on computer vision and pattern recognition. IEEE

  54. Saragih JM, Lucey S, Cohn JF (2011) Deformable model fitting by regularized landmark mean-shift. Int J Comput Vis 91(2):200–215

    Article  MathSciNet  MATH  Google Scholar 

  55. Wang Y, Lucey S, Cohn JF (2008) Enforcing convexity for improved alignment with constrained local models. In: CVPR 2008, IEEE conference on computer vision and pattern recognition. IEEE

  56. Cootes TF, Taylor CJ (1992) Active shape models—‘smart snakes’. In: BMVC92, Springer, pp 266–275

  57. Di Stefano L, Mattoccia S, Tombari F (2005) ZNCC-based template matching using bounded partial correlation. Pattern Recognit Lett 26(14):2129–2134

    Article  Google Scholar 

  58. Kanaujia A, Huang Y, Metaxas D (2006) Tracking facial features using mixture of point distribution models. In: Computer vision graphics and image processing. Springer, pp 492–503

  59. Jain LC et al (2013) A review of online learning in supervised neural networks. Neural Comput Appl 25(3—-4):491–509

    Google Scholar 

  60. FGNet (2004) Fgnet talking face video. http://www-prima.inrialpes.fr/FGnet/data/01-TalkingFace/talking_face.html

  61. Lucey P et al (2010) The extended cohn-kanade dataset (CK+): a complete dataset for action unit and emotion-specified expression. In: IEEE computer society conference on computer vision and pattern recognition workshops (CVPRW). IEEE

  62. Sagonas C, Tzimiropoulos G, Zafeiriou S, Pantic M (2013) 300 faces in-the-wild challenge: the first facial landmark localization challenge. In: Proceedings of the IEEE international conference on computer vision workshops. IEEE

  63. Li Y et al (2013) Simultaneous facial feature tracking and facial expression recognition. IEEE Trans Image Process 22(7):2559–2573

    Article  Google Scholar 

  64. Wu Y, Wang Z, Ji Q (2013) Facial feature tracking under varying facial expressions and face poses based on restricted boltzmann machines. In: IEEE conference on computer vision and pattern recognition (CVPR). IEEE

Download references

Acknowledgments

This research is supported by University of Malaya Grand Challenge Project GC003A-14HTM.

Author information

Authors and Affiliations

  1. Faculty of Computer Science and Information Technology, University of Malaya, Kuala Lumpur, Malaysia

    Md. Nazrul Islam & Chu Kiong Loo

  2. Faculty of Engineering, Computing and Science, Swinburne University of Technology Sarawak Campus, Kuching, Malaysia

    Manjeevan Seera

Authors
  1. Md. Nazrul Islam
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Chu Kiong Loo
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Manjeevan Seera
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Chu Kiong Loo.

Ethics declarations

Conflict of interest

The authors declare that they have no conflict of interest.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Islam, M.N., Loo, C.K. & Seera, M. Incremental Clustering-Based Facial Feature Tracking Using Bayesian ART. Neural Process Lett 45, 887–911 (2017). https://doi.org/10.1007/s11063-016-9554-6

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11063-016-9554-6

Keywords

Search

Navigation