Skip to main content
SpringerLink
Log in

Lip as biometric and beyond: a survey

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

Owing to the recent fuel in investigation of lip as a biometrics trait both in physiological and behavioral aspect, and in healthcare application, it is imperative to comprehensively study and document the literature. Hence, in this work we enlist the works related to the lip as a biometirc trait and other applications-based on its behavior and physiology. Consequently, in this survey we first discuss the lip anatomy that permits to identify the physiological and behavioral properties. Followed by the list of challenges to characterize lip behavior and physiology. Next, we proceed to provide an insight of several inspiring ideas proposed in the literature to establish lip behavior and physiology for identity and other applications such as health monitoring and forensic. Whilst, it can be concluded at the end of the survey that their are several open research area and unsolved issues, which we discuss in details at the end of this survey to attract the attention of researchers.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6

Similar content being viewed by others

Notes

  1. https://pubmed.ncbi.nlm.nih.gov/11987663/

  2. http://www.jfds.org/text.asp?2009/1/1/28/50885

References

  1. Abel A, Hussain A, Nguyen QD, Ringeval F, Chetouani M, Milgram M (2009) Maximising audiovisual correlation with automatic lip tracking and vowel based segmentation. In: European workshop on biometrics and identity management. pp 65–72 . https://doi.org/10.1007/978-3-642-04391-8_9

  2. Adamu L, Taura M, Hamman W, Ojo S, Dahiru A, Sadeeq A, Umar K (2015) Study of lip print types among nigerians. Homo 66(6):561–569

    Article  Google Scholar 

  3. Aleix M, Robert B (1998) The AR face database. CVC Tech. Report #24

  4. Almajai I, Cox S, Harvey R, Lan Y (2016) Improved speaker independent lip reading using speaker adaptive training and deep neural networks. In: International conference on acoustics, speech and signal processing. pp 2722–2726. https://doi.org/10.1109/ICASSP.2016.7472172

  5. Anina I, Zhou Z, Zhao G, Pietikäinen M (2015) Ouluvs2: A multi-view audiovisual database for non-rigid mouth motion analysis. In: 11th IEEE International conference and workshops on automatic face and gesture recognition (FG), vol 1. pp 1–5. https://doi.org/10.1109/FG.2015.7163155

  6. AR Face Database: Available:http://www2.ece.ohio-state.edu/ aleix/ARdatabase.html

  7. Aravabhumi VR, Chenna RR, Reddy KU (2010) Robust method to identify the speaker using lip motion features. In: International conference on mechanical and electrical technology. pp 125–129. https://doi.org/10.1109/ICMET.2010.5598333

  8. Bakhshali MA, Shamsi M (2014) Segmentation of color lip images by optimal thresholding using bacterial foraging optimization (BFO). Journal of Computational Science 5(2):251–257. https://doi.org/10.1016/j.jocs.2013.07.001

    Article  Google Scholar 

  9. Bakshi S, Raman R, Sa PK (2011) Lip pattern recognition based on local feature extraction. In: Annual IEEE India conference. pp 1–4. https://doi.org/10.1109/INDCON.2011.6139357

  10. Bakshi S, Raman R, Sa PK (2016) Nitrlipv1: a constrained lip database captured in visible spectrum. ACM SIGBioinformatics Record 6(1):2. https://doi.org/10.1145/2921555.2921557

    Article  Google Scholar 

  11. Bhattacharjee S, Arunkumar S, Bandyopadhyay SK (2013) Personal identification from lip-print features using a statistical model. arXiv:arXiv:1310.0036

  12. Bijjargi SC, Malligere SB, Sangle VA, Saraswathi F, Majid IA (2015) A new attempt in comparision between 3 racial groups in india-based on lip prints (cheiloscopy)

  13. Briceño JC, Travieso CM, Alonso JB, Ferrer MA (2010) Robust identification of persons by lips contour using shape transformation. In: 14th international conference on intelligent engineering systems. pp 203–207. https://doi.org/10.1109/INES.2010.5483848

  14. Çetingul HE, Erzin E, Yemez, Y, Tekalp AM (2004) On optimal selection of lip-motion features for speaker identification. In: 6th workshop on multimedia signal processing. pp 7–10. https://doi.org/10.1109/MMSP.2004.1436400

  15. Cetingul HE, Yemez Y, Erzin E, Tekalp AM (2005) Robust lip-motion features for speaker identification. International conference on acoustics, speech, and signal processing 1:509–512. https://doi.org/10.1109/ICASSP.2005.1415162

    Article  MATH  Google Scholar 

  16. Cetingul HE, Yemez Y, Erzin E, Tekalp AM (2006) Discriminative analysis of lip motion features for speaker identification and speech-reading. IEEE Transactions on Image Processing 15(10):2879–2891. https://doi.org/10.1109/TIP.2006.877528

    Article  MATH  Google Scholar 

  17. Chan CH, Goswami B, Kittler J, Christmas W (2011) Local ordinal contrast pattern histograms for spatiotemporal, lip-based speaker authentication. IEEE Transactions on Information Forensics and Security 7(2):602–612. https://doi.org/10.1109/TIFS.2011.2175920

    Article  Google Scholar 

  18. Chan CH, Goswami B, Kittler J, Christmas WJ (2011) Kernel-based speaker verification using spatiotemporal lip information. In: MVA. pp 422–425

  19. Chan MT (1999) Automatic lip model extraction for constrained contour-based tracking. International conference on image processing 2:848–851. https://doi.org/10.1109/ICIP.1999.823017

    Article  Google Scholar 

  20. Cheng F, Wang SL, Liew AWC (2018) Visual speaker authentication with random prompt texts by a dual-task CNN framework. Pattern Recognition 83:340–352. https://doi.org/10.1016/j.patcog.2018.06.005

    Article  Google Scholar 

  21. Chetty G, Wagner M (2004) Automated lip feature extraction for liveness verification in audio-video authentication. Proc. Image and Vision Computing :17–22

  22. Cheung YM, Li M, Cao X, You X (2014) Lip segmentation under map-mrf framework with automatic selection of local observation scale and number of segments. IEEE Transactions on Image Processing 23(8):3397–3411. https://doi.org/10.1109/TIP.2014.2331137

    Article  MathSciNet  MATH  Google Scholar 

  23. Cheung YM, Liu X, You X (2012) A local region based approach to lip tracking. Pattern Recognition 45(9):3336–3347

    Article  Google Scholar 

  24. Chin SW, Seng KP, Ang LM (2012) Audio-visual speech processing for human computer interaction. In: Advances in robotics and virtual reality. pp 135–165. https://doi.org/10.1007/978-3-642-23363-0_6

  25. Chindaro S, Deravi F (2001) Directional properties of colour co-occurrence features for lip location and segmentation. In: International conference on audio-and video-based biometric person authentication. pp 84–89. https://doi.org/10.1007/3-540-45344-X_13

  26. Choraś M (2007) Human lips recognition. Computer recognition systems 2:838–843. https://doi.org/10.1007/978-3-540-75175-5_104

    Article  Google Scholar 

  27. Choraś M (2010) The lip as a biometric. Pattern Analysis and Applications 13(1):105–112. https://doi.org/10.1007/s10044-008-0144-8

    Article  MathSciNet  Google Scholar 

  28. Choraś M, Kozik R (2012) Contactless palmprint and knuckle biometrics for mobile devices. Pattern Analysis and Applications 15(1):73–85. https://doi.org/10.1007/s10044-011-0248-4

    Article  MathSciNet  Google Scholar 

  29. Choraś RS (2011) Lip-prints feature extraction and recognition. In: Image processing and communications challenges 3, pp 33–42. https://doi.org/10.1007/978-3-642-23154-4_4

  30. Cooke M, Barker J, Cunningham S, Shao X (2006) An audio-visual corpus for speech perception and automatic speech recognition. The Journal of the Acoustical Society of America 120(5):2421–2424. https://doi.org/10.1121/1.2229005

    Article  Google Scholar 

  31. Coward R (2007) The stability of lip pattern characteristics over time. J Forensic Odontostomatol 25:40–56

    Google Scholar 

  32. de la Cuesta AG, Zhang J, Miller P (2008) Biometric identification using motion history images of a speaker’s lip movements. In: International machine vision and image processing conference. pp 83–88. https://doi.org/10.1109/IMVIP.2008.13

  33. Das A, Dantcheva A, Bremond F (2018) Mitigating bias in gender, age and ethnicity classification: a multi-task convolution neural network approach. In: Proceedings of the European conference on computer vision (ECCV). pp 0–0

  34. Das A, Galdi C, Han H, Ramachandra R, Dugelay JL, Dantcheva A (2018) Recent advances in biometric technology for mobile devices. In: 2018 IEEE 9th international conference on biometrics theory, applications and systems (BTAS). IEEE, pp 1–11

  35. Das S, Muhammad K, Bakshi S, Mukherjee I, Sa PK, Sangaiah AK, Bruno A (2018) Lip biometric template security framework using spatial steganography. Pattern Recognition Letters 126:102–110. https://doi.org/10.1016/j.patrec.2018.06.026

    Article  Google Scholar 

  36. Delmas P, Coulon PY, Fristot V (1999) Automatic snakes for robust lip boundaries extraction. International conference on acoustics, speech, and signal processing 6:3069–3072. https://doi.org/10.1109/ICASSP.1999.757489

    Article  Google Scholar 

  37. Dineshshankar J, Ganapathi N, Yoithapprabhunath TR, Maheswaran T, Kumar MS, Aravindhan R (2013) Lip prints: Role in forensic odontology. Journal of Pharmacy and Bioallied Sciences 5(Suppl 1):S95

    Article  Google Scholar 

  38. Erzin E, Yemez Y, Tekalp A (2004) Dsp in mobile and vehicular systems

  39. Ezz M, Mostafa AM, Nasr AA (2020) A silent password recognition framework based on lip analysis. IEEE Access 8:55354–55371

    Article  Google Scholar 

  40. Faraj MI, Bigun J (2007) Audio-visual person authentication using lip-motion from orientation maps. Pattern recognition letters 28(11):1368–1382. https://doi.org/10.1016/j.patrec.2007.02.017

    Article  Google Scholar 

  41. Foong OM, Hong KW, Yong SP (2016) Droopy mouth detection model in stroke warning. In: 2016 3rd international conference on computer and information sciences (ICCOINS). IEEE, pp 616–621

  42. Fox NA, O’Mullane BA, Reilly RB (2005) VALID: a new practical audio-visual database, and comparative results. In: International conference on audio-and video-based biometric person authentication. pp 777–786

  43. Franzgrote M, Borg C, Ries BJT, Bussemaker S, Jiang X, Fieleser M, Zhang L (2011) Palmprint verification on mobile phones using accelerated competitive code. In: International conference on hand-based biometrics. pp 1–6. https://doi.org/10.1109/ICHB.2011.6094309

  44. Fu JW, Wang SL, Lin X (2016) Robust lip region segmentation based on competitive fcm clustering. In: International conference on digital image computing: techniques and applications. pp 1–8. https://doi.org/10.1109/DICTA.2016.7797077

  45. George R, Afandi NSBN, Abidin SNHBZ, Ishak NIB, Soe HHK, Ismail ARH (2016) Inheritance pattern of lip prints among malay population: a pilot study. Journal of Forensic and Legal Medicine 39:156–160

    Article  Google Scholar 

  46. Ghaleh VEC, Behrad A (2010) Lip contour extraction using rgb color space and fuzzy c-means clustering. In: 9th international conference on cybernetic intelligent systems. pp 1–4. https://doi.org/10.1109/UKRICIS.2010.5898135

  47. Gofman MI, Mitra S, Cheng THK, Smith NT (2016) Multimodal biometrics for enhanced mobile device security. Communications of the ACM 59(4):58–65. https://doi.org/10.1145/2818990

    Article  Google Scholar 

  48. Gomez E, Travieso CM, Briceno J, Ferrer M (2002) Biometric identification system by lip shape. In: International carnahan conference on security technology. pp 39–42. https://doi.org/10.1109/CCST.2002.1049223

  49. Guan C, Wang S, Liew AWC (2019) Lip image segmentation based on a fuzzy convolutional neural network. IEEE Transactions on Fuzzy Systems

  50. Guan C, Wang S, Liu G, Liew AWC (2019) Lip image segmentation in mobile devices based on alternative knowledge distillation. In: 2019 IEEE international conference on image processing (ICIP). IEEE, pp 1540–1544

  51. Guan YP (2006) Automatic extraction of lip based on wavelet edge detection. In: Eighth international symposium on symbolic and numeric algorithms for scientific computing. pp 125–132. https://doi.org/10.1109/SYNASC.2006.19

  52. Hamzah NH, Seliman AFFM, Osman K, GABRIEL GF (2020) Lip print analysis in malaysian chinese population (klang valley): Lipstick-cellophane tape technique. Jurnal Sains Kesihatan Malaysia (Malaysian Journal of Health Sciences) 18(2)

  53. Happy S, Dantcheva A, Das A, Zeghari R, Robert P, Bremond F (2019) Characterizing the state of apathy with facial expression and motion analysis. In: 2019 14th IEEE international conference on automatic face & gesture recognition (FG 2019). IEEE, pp 1–8

  54. Ichino M (2014) Lip-movement based speaker recognition using fusion of canonical angles. In: 13th international conference on control automation robotics & vision (ICARCV). pp 958–963. https://doi.org/10.1109/ICARCV.2014.7064435

  55. Ichino M, Sakano H, Komatsu N (2006) Multimodal biometrics of lip movements and voice using kernel fisher discriminant analysis. In: 9th international conference on control, automation, robotics and vision. pp 1–6 . https://doi.org/10.1109/ICARCV.2006.345473

  56. Ichino M, Yamazaki Y, Jian-Gang W, Yun YW (2012) Text independent speaker gender recognition using lip movement. In: 12th international conference on control automation robotics & vision. pp 176–181. https://doi.org/10.1109/ICARCV.2012.6485154

  57. Kanade T, Cohn JF, Tian, Y (2000) Comprehensive database for facial expression analysis. In: Fourth IEEE international conference on automatic face and gesture recognition. pp 46–53. https://doi.org/10.1109/AFGR.2000.840611

  58. Kapoor N, Badiye A (2017) A study of distribution, sex differences and stability of lip print patterns in an indian population. Saudi journal of biological sciences 24(6):1149–1154

    Article  Google Scholar 

  59. Kasinski A, Florek A, Schmidt A (2008) The put face database. Image Processing and Communications 13(3–4):59–64

    Google Scholar 

  60. Kim JO, Lee W, Hwang J, Baik KS, Chung CH (2004) Lip print recognition for security systems by multi-resolution architecture. Future Generation Computer Systems 20(2):295–301

    Article  Google Scholar 

  61. Lai JY, Wang SL, Liew AWC, Shi XJ (2016) Visual speaker identification and authentication by joint spatiotemporal sparse coding and hierarchical pooling. Information Sciences 373:219–232. https://doi.org/10.1016/j.ins.2016.09.015

    Article  Google Scholar 

  62. Lai JY, Wang SL, Shi XJ, Liew AWC (2014) Sparse coding based lip texture representation for visual speaker identification. In: 19th international conference on digital signal processing. pp 607–610. https://doi.org/10.1109/ICDSP.2014.6900736

  63. Langner O, Dotsch R, Bijlstra G, Wigboldus DH, Hawk ST, Van Knippenberg A (2010) Presentation and validation of the radboud faces database. Cognition and emotion 24(8):1377–1388. https://doi.org/10.1080/02699930903485076

    Article  Google Scholar 

  64. Lee D, Myung K (2017) Read my lips, login to the virtual world. In: International conference on consumer electronics. pp 434–435. https://doi.org/10.1109/ICCE.2017.7889386

  65. Leung SH, Wang SL, Lau WH (2004) Lip image segmentation using fuzzy clustering incorporating an elliptic shape function. IEEE transactions on image processing 13(1):51–62. https://doi.org/10.1109/TIP.2003.818116

    Article  Google Scholar 

  66. Li F, Zhao C, Xia Z, Wang Y, Zhou X, Li GZ (2012) Computer-assisted lip diagnosis on traditional chinese medicine using multi-class support vector machines. BMC complementary and alternative medicine 12(1):127

    Article  Google Scholar 

  67. Li H, Jones KL, Hooper JE, Williams T (2019) The molecular anatomy of mammalian upper lip and primary palate fusion at single cell resolution. Development 146(12)

  68. Li M, Cheung YM (2010) Automatic segmentation of color lip images based on morphological filter. In: International conference on artificial neural networks. pp 384–387. https://doi.org/10.1007/978-3-642-15819-3_51

  69. Liao CW, Lin WY, Lin CW (2008) Video-based person authetication with random passwords. In: International conference on multimedia and expo. pp 581–584. https://doi.org/10.1109/ICME.2008.4607501

  70. Liévin M, Delmas P, Coulon PY, Luthon F, Fristol V (1999) Automatic lip tracking: Bayesian segmentation and active contours in a cooperative scheme. International conference on multimedia computing and systems 1:691–696. https://doi.org/10.1109/MMCS.1999.779283

    Article  Google Scholar 

  71. Lievin M, Luthon F (1998) Lip features automatic extraction. In: International conference on image processing. pp 168–172. https://doi.org/10.1109/ICIP.1998.727160

  72. Liew AWC, Leung SH, Lau WH (2002) Lip contour extraction from color images using a deformable model. Pattern Recognition 35(12):2949–2962. https://doi.org/10.1016/S0031-3203(01)00231-X

    Article  MATH  Google Scholar 

  73. Liu X, Cheung YM (2013) Learning multi-boosted HMMs for lip-password based speaker verification. IEEE Transactions on Information Forensics and Security 9(2):233–246. https://doi.org/10.1109/TIFS.2013.2293025

    Article  Google Scholar 

  74. Liu YF, Lin CY, Guo JM (2012) Impact of the lips for biometrics. IEEE Transactions on Image Processing 21(6):3092–3101. https://doi.org/10.1109/TIP.2012.2186310

    Article  MathSciNet  MATH  Google Scholar 

  75. Liu YF, Lin CY, Guo JM (2012) Limitation investigation toward lips recognition. In: 2012 IEEE international conference on acoustics, speech and signal processing (ICASSP). IEEE, pp 1857–1860

  76. Lu L, Yu J, Chen Y, Liu H, Zhu Y, Liu Y, Li M (2018) Lippass: Lip reading-based user authentication on smartphones leveraging acoustic signals. In: IEEE INFOCOM 2018-IEEE conference on computer communications. pp 1466–1474. https://doi.org/10.1109/INFOCOM.2018.8486283

  77. Lu Y (2018) Liu, Q (2018) Lip segmentation using automatic selected initial contours based on localized active contour model. EURASIP Journal on Image and Video Processing 1:7. https://doi.org/10.1186/s13640-017-0243-9

    Article  Google Scholar 

  78. Lu Y, Yang S, Xu Z, Wang J (2020) Speech training system for hearing impaired individuals based on automatic lip-reading recognition. In: International conference on applied human factors and ergonomics. Springer, pp 250–258

  79. Lu Y, Zhu X, Xiao K (2019) Unsupervised lip segmentation based on quad-tree mrf framework in wavelet domain. Measurement 141:95–101

    Article  Google Scholar 

  80. Lu Z, Wu X, He R (2016) Person identification from lip texture analysis. In: International conference on digital signal processing. pp 472–476. https://doi.org/10.1109/ICDSP.2016.7868602

  81. Luettin J, Thacker NA, Beet SW (1996) Speechreading using shape and intensity information. Fourth international conference on spoken language 1:58–61. https://doi.org/10.1109/ICSLP.1996.607024

    Article  Google Scholar 

  82. Luettin J, Thacker NA, Beet SW (1996) Speaker identification by lipreading. Fourth international conference on spoken language 1:62–65. https://doi.org/10.1109/ICSLP.1996.607030

    Article  Google Scholar 

  83. Ma X, Zhang H, Li Y (2017) A lip localization algorithm under variant light conditions. In: Proceedings of the 9th international conference on machine learning and computing. pp 305–309

  84. Malek M, Aïcha B, et al (2019) Automatic lip segmentation with level set method. In: 2019 international conference on control, automation and diagnosis (ICCAD). IEEE, pp 1–4

  85. Mathulaprangsan S, Wang CY, Kusum AZ, Tai TC, Wang JC (2015) A survey of visual lip reading and lip-password verification. In: 2015 international conference on orange technologies (ICOT). IEEE, pp 22–25

  86. Matthews I, Cootes TF, Bangham JA, Cox S, Harvey R (2002) Extraction of visual features for lipreading. pp 198–213. https://doi.org/10.1109/34.982900

  87. Mehra H, Das A, Ranjan R, Pandey B, Ranjan S, Shukla A, Tiwari R (2010) Expert system for speaker identification using lip features with PCA. In: 2nd international workshop on intelligent systems and applications. pp 1–4. https://doi.org/10.1109/IWISA.2010.5473241

  88. Messer K, Matas J, Kittler J, Luettin J, Maitre G (1999) Xm2vtsdb: The extended m2vts database. Second international conference on audio and video-based biometric person authentication 964:965–966

    Google Scholar 

  89. Mir SA, Qurat-ul Ain SK, Bhat MA, Mehraj H (2018) Person identification by lips using sgldm and support vector machine. International Journal of Scientific Research in Computer Science, Engineering and Information Technology :152–157

  90. Mok LL, Lau WH, Leung S, Wang S, Yan H (2004) Person authentication using asm based lip shape and intensity information. International conference on image processing 1:561–564. https://doi.org/10.1109/ICIP.2004.1418816

    Article  Google Scholar 

  91. Movellan JR (1995) Visual speech recognition with stochastic networks. In: Advances in neural information processing systems. pp 851–858

  92. Nagrani A, Chung JS, Zisserman A (2017) Voxceleb: a large-scale speaker identification dataset. arXiv:arXiv:1706.08612

  93. Nguyen QD, Milgram M (2008) Lip contours detection and tracking with multi features. In: Biometrics symposium. pp 35–40. https://doi.org/10.1109/BSYM.2008.4655520

  94. Nicolaidis C, Raymaker D, McDonald K, Dern S, Boisclair WC, Ashkenazy E, Baggs A (2013) Comparison of healthcare experiences in autistic and non-autistic adults: a cross-sectional online survey facilitated by an academic-community partnership. Journal of general internal medicine 28(6):761–769

  95. Niu X, Zhao X, Han H, Das A, Dantcheva A, Shan S, Chen X (2019) Robust remote heart rate estimation from face utilizing spatial-temporal attention. In: 2019 14th IEEE international conference on automatic face & gesture recognition (FG 2019). IEEE, pp 1–8

  96. Norhikmah MK, Angriawan SKH (2019) Implementation of 2dpca and som algorithms to determine sex according to lip shapes. In: 2019 4th international conference on information technology, information systems and electrical engineering (ICITISEE). IEEE, pp 101–106

  97. Omata M, Hamamoto T, Hangai S (2001) Lip recognition using morphological pattern spectrum. In: International conference on audio-and video-based biometric person authentication. pp 108–114. https://doi.org/10.1007/3-540-45344-X_17

  98. OULUVS2: A multi-view audiovisual database: Available: http://www.ee.oulu.fi/research/imag/OuluVS2/

  99. Pass A, Zhang J, Stewart D (2010) Feature selection for pose invariant lip biometrics. In: Eleventh annual conference of the international speech communication association. pp 1165–1168

  100. Patterson EK, Gurbuz S, Tufekci Z, Gowdy JN (2002) CUAVE: A new audio-visual database for multimodal human-computer interface research. International conference on acoustics, speech, and signal processing 2:2017–2020. https://doi.org/10.1109/ICASSP.2002.5745028

    Article  Google Scholar 

  101. Peer P (2005) Cvl face database. Computer vision lab, faculty of computer and information science, University of Ljubljana, Slovenia. Available at http://www.lrv.fri.uni-lj.si/facedb.html

  102. Pérez JFG, Frangi AF, Solano EL, Lukas K (2005) Lip reading for robust speech recognition on embedded devices. International conference on acoustics, speech, and signal processing 1:473–476. https://doi.org/10.1109/ICASSP.2005.1415153

    Article  Google Scholar 

  103. Petajan ED (1984) Automatic lipreading to enhance speech recognition (speech reading)

  104. Petridis S, Wang Y, Li Z, Pantic M (2017) End-to-end audiovisual fusion with LSTMs. arXiv:arXiv:1709.04343

  105. Pocovnicu A (2009) Biometric security for cell phones. Informatica Economica 13(1):57–63

    Google Scholar 

  106. Porwik P, Doroz R, Wrobel K (2019) An ensemble learning approach to lip-based biometric verification, with a dynamic selection of classifiers. Expert Systems with Applications 115:673–683

    Article  Google Scholar 

  107. Porwik P, Orczyk T (2012) Dtw and voting-based lip print recognition system. In: IFIP international conference on computer information systems and industrial management. Springer, pp 191–202

  108. Raman R, Sa P, Majhi B, Bakshi S (2017) Fusion of shape and texture features for lip biometry in mobile devices. Mobile Biometrics 3:155. https://doi.org/10.1049/PBSE003E_ch

    Article  Google Scholar 

  109. Raman R, Sa PK, Majhi B, Bakshi S (2017) Acquisition and corpus description of a constrained lip database captured from handheld devices: NITRLipV2 (MobioLip). ACM SIGBioinformatics Record 7(1):2. https://doi.org/10.1145/3056351.3056353

    Article  Google Scholar 

  110. Ramli D, Samad S, Hussain A (2008) A UMACE filter approach to lipreading in biometric authentication system. Journal of Applied Sciences 8(2):280–287. https://doi.org/10.3923/jas.2008.280.287

    Article  Google Scholar 

  111. Ranjan V, Sunil MK, Kumar R et al (2014) Study of lip prints: A forensic study. Journal of Indian Academy of Oral Medicine and Radiology 26(1):50

    Article  Google Scholar 

  112. Rojas AM, Travieso CM, Alonso JB, Ferrer MA (2012) Automatic lip identification applied under soft facial emotion conditions. In: International carnahan conference on security technology. pp 218–223. https://doi.org/10.1109/CCST.2012.6393562

  113. Ross A, Jain A, Reisman J (2003) A hybrid fingerprint matcher. Pattern Recognition 36(7):1661–1673

    Article  Google Scholar 

  114. Roy A, Marcel S (2010) Crossmodal matching of speakers using lip and voice features in temporally non-overlapping audio and video streams. In: 20th International conference on pattern recognition. pp 4504–4507. https://doi.org/10.1109/ICPR.2010.1094

  115. Sim T, Baker S, Bsat M (2003) The cmu pose, illumination, and expression database. IEEE Transactions on Pattern Analysis and Machine Intelligence 25(12):1615–1618

    Article  Google Scholar 

  116. Saeed U (2010) Comparative analysis of lip features for person identification. In: Proceedings of the 8th international conference on frontiers of information technology. https://doi.org/10.1145/1943628.1943648

  117. Saeed U, Dugelay JL (2009) Temporal normalization of videos using visual speech. In: Proceedings of the first ACM workshop on multimedia in forensics. pp 7–12. https://doi.org/10.1145/1631081.1631084

  118. Saeed U, Dugelay JL (2010) Combining edge detection and region segmentation for lip contour extraction. In: International conference on articulated motion and deformable objects. pp 11–20. https://doi.org/10.1007/978-3-642-14061-7_2

  119. Salehghaffari H (2018) Speaker verification using convolutional neural networks. arXiv:arXiv:1803.05427

  120. Sanderson C, Lovell BC (2009) Multi-region probabilistic histograms for robust and scalable identity inference. In: International conference on biometrics. pp 199–208. https://doi.org/10.1007/978-3-642-01793-3_21

  121. Sandhya S, Fernandes R, Sapna S, Rodrigues AP (2021) Segmentation of lip print images using clustering and thresholding techniques. In: Advances in artificial intelligence and data engineering. Springer, pp 1023–1034

  122. Saxena A, Anand A, Mukerjee A (2004) Robust facial expression recognition using spatially localized geometric model. International conference on systemics, Cybernetics and Informatics 1:124–129

    Google Scholar 

  123. Sayo A, Kajikawa Y, Muneyasu M (2011) Biometrics authentication method using lip motion in utterance. In: 8th International conference on information, communications & signal processing. pp 1–5. https://doi.org/10.1109/ICICS.2011.6173131

  124. Shabeer HA, Suganthi P (2007) Mobile phones security using biometrics. International conference on conference on computational intelligence and multimedia applications 4:270–274. https://doi.org/10.1109/ICCIMA.2007.182

    Article  Google Scholar 

  125. Sharma P, Deo S, Venkateshan S, Vaish A (2011) Lip print recognition for security systems: an up-coming biometric solution. In: Intelligent interactive multimedia systems and services. Springer, pp 347–359

  126. Shi XX, Wang SL, Lai JY (2016) Visual speaker authentication by ensemble learning over static and dynamic lip details. In: International conference on image processing. pp 3942–3946. https://doi.org/10.1109/ICIP.2016.7533099

  127. Shirgahi H, Motameni H, Valipour P (2008) A new approach for detection by movement of lips base on image processing and fuzzy decision. World Applied Sciences Journal 3(2):323–329

    Google Scholar 

  128. Singh P, Laxmi V, Gaur MS (2012) Speaker identification using optimal lip biometrics. In: 5th IAPR international conference on biometrics (ICB). pp 472–477. https://doi.org/10.1109/ICB.2012.6199795

  129. Smacki L, Luczak J, Wrobel Z (2016) Lip print pattern extraction using top-hat transform. In: Proceedings of the 9th international conference on computer recognition systems CORES 2015. Springer, pp 337–346

  130. Smacki L, Wrobel K, Porwik P (2011) Lip print recognition based on dtw algorithm. In: 2011 third world congress on nature and biologically inspired computing. IEEE, pp 594–599

  131. Spyridonos P, Saint AF, Likas A, Gaitanis G, Bassukas I (2018) Multi-threshold lip contour detection. In: International conference on image processing. pp 1912–1916. https://doi.org/10.1109/ICIP.2018.8451680

  132. Stafylakis T, Tzimiropoulos G (2017) Combining residual networks with LSTMs for lipreading. arXiv:arXiv:1703.04105(2017)

  133. Sukno FM, Ordas S, Butakoff C, Cruz S, Frangi AF (2007) Active shape models with invariant optimal features: Application to facial analysis. Transactions on Pattern Analysis and Machine Intelligence 29(7):1105–1117. https://doi.org/10.1109/TPAMI.2007.1041

    Article  Google Scholar 

  134. Sun K, Yu C, Shi W, Liu L, Shi Y (2018) Lip-interact: Improving mobile device interaction with silent speech commands. In: 31st annual ACM symposium on user interface software and technology. pp 581–593. https://doi.org/10.1145/3242587.3242599

  135. Szeliski R (2010) Computer vision: algorithms and applications. Springer Science & Business Media, Berlin

    MATH  Google Scholar 

  136. Tan J, Wang X, Nguyen CT, Shi Y (2018) Silentkey A new authentication framework through ultrasonic-based lip reading. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 2(1):36. https://doi.org/10.1145/3191768

    Article  Google Scholar 

  137. Tatulli E, Hueber T (2017) Feature extraction using multimodal convolutional neural networks for visual speech recognition. In: International conference on acoustics, speech and signal processing (ICASSP). pp 2971–2975. https://doi.org/10.1109/ICASSP.2017.7952701

  138. Thabet Z, Nabih A, Azmi K, Samy Y, Khoriba G, Elshehaly M (2018) Lipreading using a comparative machine learning approach. In: First international workshop on deep and representation learning. pp 19–25. https://doi.org/10.1109/IWDRL.2018.8358210

  139. Thangthai K, Harvey RW, Cox SJ, Theobald BJ (2015) Improving lip-reading performance for robust audiovisual speech recognition using DNNs. In: 1st joint conference on facial analysis, animation, and auditory-visual speech processing Vienna, Austria. pp 127–131

  140. Thein T, San KM (2018) Lip localization technique towards an automatic lip reading approach for myanmar consonants recognition. In: 2018 international conference on information and computer technologies (ICICT). IEEE, pp 123–127

  141. Torfi A, Iranmanesh SM, Nasrabadi N, Dawson J (2017) 3d convolutional neural networks for cross audio-visual matching recognition. IEEE Access 5:22081–22091. https://doi.org/10.1109/ACCESS.2017.2761539

    Article  Google Scholar 

  142. Travieso CM, Ravelo-García AG, Alonso JB, Canino-Rodríguez JM, Dutta MK (2019) Improving the performance of the lip identification through the use of shape correction. Applied Intelligence 49(5):1823–1840

    Article  Google Scholar 

  143. Travieso CM, Zhang J, Miller P, Alonso JB (2014) Using a discrete hidden markov model kernel for lip-based biometric identification. Image and Vision Computing 32(12):1080–1089. https://doi.org/10.1016/j.imavis.2014.10.001

    Article  Google Scholar 

  144. Travieso CM, Zhang J, Miller P, Alonso JB, Ferrer MA (2011) Bimodal biometric verification based on face and lips. Neurocomputing 74(14–15):2407–2410. https://doi.org/10.1016/j.neucom.2011.03.012

    Article  Google Scholar 

  145. Tresadern P, McCool C, Poh N, Matejka P, Hadid A, Levy C, Cootes T, Marcel S (2013) Mobile biometrics: Combined face and voice verification for a mobile platform. IEEE pervasive computing 12(1):79–87. https://doi.org/10.1109/MPRV.2012.54

    Article  Google Scholar 

  146. Tsuchihashi Y (1974) Studies on personal identification by means of lip prints. Forensic Science 3:233–248. https://doi.org/10.1016/0300-9432(74)90034-X

    Article  Google Scholar 

  147. VidTIMIT Audio-Video Dataset: Available: http://conradsanderson.id.au/vidtimit/

  148. Wang H, Roussel P, Denby B (2021) Improving ultrasound-based multimodal speech recognition with predictive features from representation learning. JASA Express Letters 1(1):015205

    Article  Google Scholar 

  149. Wang J, Wang Y, Liu A, Xiao J (2017) Assistance of speech recognition in noisy environment with sentence level lip-reading. In: Chinese conference on biometric recognition. pp 593–601. https://doi.org/10.1007/978-3-319-69923-3_64

  150. Wang SL, Lau WH, Leung SH, Liew AWC (2004) Lip segmentation with the presence of beards. International conference on acoustics, speech, and signal processing 3:529–532. https://doi.org/10.1109/ICASSP.2004.1326598

    Article  Google Scholar 

  151. Wang SL, Lau WH, Leung SH, Yan H (2004) A real-time automatic lipreading system. International Symposium on Circuits and Systems 2:101–104. https://doi.org/10.1109/ISCAS.2004.1329218

    Article  Google Scholar 

  152. Wang SL, Lau WH, Liew AWC, Leung SH (2007) Robust lip region segmentation for lip images with complex background. Pattern Recognition 40(12):3481–3491. https://doi.org/10.1016/j.patcog.2007.03.016

    Article  MATH  Google Scholar 

  153. Wang SL, Leung SH, Lau WH (2002) Lip segmentation by fuzzy clustering incorporating with shape function. In: International conference on acoustics, speech and signal processing. pp 1077–1080. https://doi.org/10.1109/ICASSP.2002.5743982

  154. Wang SL, Liew AWC (2007) ICA-based lip feature representation for speaker authentication. In: Third international IEEE conference on signal-image technologies and internet-based system. pp 763–767. https://doi.org/10.1109/SITIS.2007.37

  155. Wang SL, Liew AWC (2012) Physiological and behavioral lip biometrics: A comprehensive study of their discriminative power. Pattern Recognition 45(9):3328–3335. https://doi.org/10.1016/j.patcog.2012.02.016

    Article  Google Scholar 

  156. Wark T, Sridharan S (1998) A syntactic approach to automatic lip feature extraction for speaker identification. International conference on acoustics, speech and signal processing 6:3693–3696. https://doi.org/10.1109/ICASSP.1998.679685

    Article  Google Scholar 

  157. Wark T, Sridharan S, Chandran V (2000) The use of temporal speech and lip information for multi-modal speaker identification via multi-stream hmms. International conference on acoustics, speech, and signal processing 4:2389–2392. https://doi.org/10.1109/ICASSP.2000.859322

    Article  Google Scholar 

  158. Wright C, Stewart D (2019) One-shot-learning for visual lip-based biometric authentication. In: International symposium on visual computing. Springer, pp 405–417

  159. Wright C, Stewart DW (2020) Understanding visual lip-based biometric authentication for mobile devices. EURASIP Journal on Information Security 2020(1):1–16

    Article  Google Scholar 

  160. Wrobel K, Doroz R, Palys M (2013) A method of lip print recognition based on sections comparison. In: International conference on biometrics and kansei engineering. pp 47–52. https://doi.org/10.1109/ICBAKE.2013.10

  161. Wrobel K, Doroz R, Palys M (2015) Lip print recognition method using bifurcations analysis. In: Asian conference on intelligent information and database systems. Springer, pp 72–81

  162. Wrobel K, Doroz R, Porwik P, Bernas M (2018) Personal identification utilizing lip print furrow based patterns. a new approach. Pattern Recognition 81:585–600. https://doi.org/10.1016/j.patcog.2018.04.030

    Article  Google Scholar 

  163. Wrobel K, Doroz R, Porwik P, Naruniec J, Kowalski M (2017) Using a probabilistic neural network for lip-based biometric verification. Engineering Applications of Artificial Intelligence 64:112–127. https://doi.org/10.1016/j.engappai.2017.06.003

    Article  Google Scholar 

  164. XM2VTSDB: Available:http://www.ee.surrey.ac.uk/CVSSP/xm2vtsdb/

  165. Yazdi MZ (2019) Depth-based lip localization and identification of open or closed mouth, using kinect 2. In: Multidisciplinary digital publishing institute proceedings, vol 27. p 22

  166. Zhang J, Roussel P, Denby B (2021) Creating song from lip and tongue videos with a convolutional vocoder. IEEE Access

  167. Zhang X, Mersereau RM (2000) Lip feature extraction towards an automatic speechreading system. International conference on image processing 3:226–229. https://doi.org/10.1109/ICIP.2000.899336

    Article  Google Scholar 

  168. Zhao G, Barnard M, Pietikainen M (2009) Lipreading with local spatiotemporal descriptors. IEEE Transactions on Multimedia 11(7):1254–1265. https://doi.org/10.1109/TMM.2009.2030637

    Article  Google Scholar 

  169. Zheng L, Li X, Yan X, Li F, Zheng X, Li W (2010) Lip color classification based on support vector machine and histogram. In: 2010 3rd international congress on image and signal processing, vol 4. IEEE, pp 1883–1886

  170. Zhu ZY, He QH, Feng XH, Li YX, Wang ZF (2013) Liveness detection using time drift between lip movement and voice. International conference on machine learning and cybernetics 2:973–978. https://doi.org/10.1109/ICMLC.2013.6890423

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of Computer Science and Engineering, National Institute of Technology, Rourkela, India

    Debbrota P. Chowdhury, Ritu Kumari, Sambit Bakshi & Manmath N. Sahoo

  2. Inria Sophia Antipolis - Méditerranée, Biot, France

    Abhijit Das

  3. Thapar University, Patiala, Punjab, India

    Abhijit Das

Authors
  1. Debbrota P. Chowdhury
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Ritu Kumari
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Sambit Bakshi
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Manmath N. Sahoo
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Abhijit Das
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Abhijit Das.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Chowdhury, D.P., Kumari, R., Bakshi, S. et al. Lip as biometric and beyond: a survey. Multimed Tools Appl 81, 3831–3865 (2022). https://doi.org/10.1007/s11042-021-11613-5

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-021-11613-5

Keywords

Search

Navigation