Skip to main content
SpringerLink
Log in

Occlusion detection and restoration techniques for 3D face recognition: a literature review

  • Original Paper
  • Published:
Machine Vision and Applications Aims and scope Submit manuscript

Abstract

Methodologies for 3D face recognition which work in the presence of occlusions are core for the current needs in the field of identification of suspects, as criminals try to take advantage of the weaknesses among the implemented security systems by camouflaging themselves and occluding their face with eyeglasses, hair, hands, or covering their face with scarves and hats. Recent occlusion detection and restoration strategies for recognition purposes of 3D partially occluded faces with unforeseen objects are here presented in a literature review. The research community has worked on face recognition systems under controlled environments, but uncontrolled conditions have been investigated in a lesser extent. The paper details the experiments and databases used to handle the problem of occlusion and the results obtained by different authors. Lastly, a comparison of various techniques is presented and some conclusions are drawn referring to the best outcomes.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5

Similar content being viewed by others

References

  1. Saini, R., Rana, N.: Comparison of various biometric methods. Int. J. Adv. Sci. Technol. (IJAST) 2(1), 24–30 (2014)

    Google Scholar 

  2. Pato, J., Millett, L.: Biometric Recognition: Challenges and Opportunities. National Academies Press, Washington, D.C. (2010)

    Google Scholar 

  3. Jain, A., Ross, A., Prabhakar, S.: An introduction to biometric recognition. IEEE Trans. Circuits Syst. Video Technol. 14(1), 4–20 (2004)

    Article  Google Scholar 

  4. Zhao, W., Chellappa, R., Phillips, P., Rosenfeld, A.: Face recognition: a literature survey. ACM Comput. Surv. 35, 399–458 (2003)

    Article  Google Scholar 

  5. Zhao, W., Chellappa, R.: 3D Model Enhanced Face Recognition. In: Proceedings 2000 International Conference on Image Processing 3, 50–53 (2000)

  6. Bowyer, K.W., Chang, K.I., Flynn, P.J.: A survey of approaches to three-dimensional face recognition. Int. Conf. Pattern Recognit. 1, 358–361 (2004)

    Google Scholar 

  7. Kakadiaris, I.A., Toderici, G., Evangelopoulos, G., Passalis, G., Chu, D., Zhao, K., Shah, S.K., Theoharis, T.: 3D–2D face recognition with pose and illumination normalization. Comput. Vis. Image Underst. 154, 137–151 (2017)

    Article  Google Scholar 

  8. Ayyavoo, T., Suseela, J.J.: Illumination pre-processing method for face recognition using 2D DWT and CLAHE. IET Biom. (2017)

  9. Bowyer, K., Chang, K., Flynn, P.: A survey of approaches and challenges in 3D and multi-modal 3D+2D face recognition. Comput. Vis. Image Underst. 101(1), 1–15 (2006)

    Article  Google Scholar 

  10. Pal, R.: Innovative Research in Attention Modelling and Computer Vision Applications. IGI Global, Hershey, PA (2015)

    Google Scholar 

  11. Ekenel, H., Stiefelhagen, R.: Why Is Facial Occlusion a Challenging Problem? Lecture Notes in Computer Science 5558 (2009)

  12. Li, H., Huang, D., Morvan, J.M., Chen, L., Wang, Y.: Expression-robust 3D face recognition via weighted sparse representation of multi-scale and multi-component local normal patterns. Neurocomputing 133, 179–193 (2014)

    Article  Google Scholar 

  13. Chellappa, R., Wilson, C., Sirohey, S.: Human and machine recognition of faces: a survey. Proc. IEEE 83(5), 705–741 (1995)

    Article  Google Scholar 

  14. Samal, A., Iyengar, P.A.: Automatic recognition and analysis of human faces and facial expressions: a survey. Pattern Recogn. 25, 65–7 (1992)

    Article  Google Scholar 

  15. Azeem, A., Raza, M., Murtaza, M.: A survey: face recognition techniques under partial occlusion. Int. Arab J. Inf. Technol. 11, 1–10 (2014)

    Google Scholar 

  16. Savran, A., Alyüz, N., Dibeklioğlu, H., Çeliktutan, O., Gökberk, B., Sankur, B., Akarun, L.: Bosphorus Database for 3D Face Analysis. Lecture Notes in Computer Science 5372 (2008)

  17. Alyuz, N., Gokberk, B., Akarun, L.: A 3D face recognition system for expression and occlusion invariance. Biometrics: Theory, Applications and Systems 29 (2008)

  18. Colombo, A., Cusano, C., Schettini, R.: UMB-DB: A database of partially occluded 3D faces. Computer Vision Workshops (ICCV Workshops) (2011)

  19. Moreno, A.B., Sanchez, A.: GavabDB: a 3D face database. Workshop on Biometrics on the Internet, 77–85 (2004)

  20. Zohra, F.T., Rahman, W., Gavrilova, M.: Occlusion Detection and Localization from Kinect Depth Images. In: International Conference on Cyberworlds (2016)

  21. Min, R., Kose, N., Dugelay, J.: KinectFaceDB: a kinect database for face recognition. IEEE Trans. Syst. Man Cybern. 44(11), 1534–1548 (2014)

    Article  Google Scholar 

  22. Phillips, P., Flynn, P., Scruggs, T., Bowyer, K., Chang, J., Hoffman, K., Marques, J., Min, J., Worek, W.: Overview of the face recognition grand challenge. IEEE Comput. Soc. Conf. Comput. Vis. Pattern Recognit. 1, 947–954 (2005)

    Google Scholar 

  23. Colombo, A., Cusano, C., Schettin, R.: Gappy PCA classification for occlusion tolerant 3D face detection. J. Math. Imaging Vision 35, 193 (2009)

    Article  MathSciNet  Google Scholar 

  24. Alyuz, N., Gokberk, B., Akarun, L.: 3-D face recognition under occlusion using masked projection. IEEE Trans. Inf. Forensics Secur. 8, 789–802 (2013)

    Article  Google Scholar 

  25. Alyuz, N., Gokberk, B., Akarun, L.: Detection of realistic facial occlusions for robust 3D face recognition. In: 22nd International Conference on Pattern Recognition (2014)

  26. Bagchi, P., Bhattacharjee, D., Nasipuri, M.: Robust 3D face recognition in presence of pose and partial occlusions or missing parts. Int. J. Found. Comput. Sci. Technol. (IJFCST) 4(4), 21–35 (2014)

    Article  Google Scholar 

  27. Bellil, W., Brahim, H., Ben Amar, C.: Gappy wavelet neural network for 3D occluded faces: detection and recognition. Moltimed. Tools Appl. 75, 36–380 (2016)

    Google Scholar 

  28. Drira, H., Ben Amor, B., Srivastava, A., Daoudi, M., Slama, R.: 3D face recognition under expressions, occlusions and pose variations. IEEE Trans. Pattern Anal. Mach. Intell. 35, 2270–2283 (2013)

    Article  Google Scholar 

  29. Drira, H., Ben Amor, B., Srivastava, A., Daoudi, M., Slama, R.: 3D face recognition using geodesic facial curves to handle expression, occlusions and pose variations. Int. J. Comput. Sci. Inf. Technol. (IJCSIT) 53, 4284–4287 (2014)

    Google Scholar 

  30. Ganguly, S., Bhattacharjee, D., Nasipuri, M.: Depth based occlusion detection and localization from 3D face image. Int. J. Image Graph. Signal Process. 5, 20–31 (2015)

    Article  Google Scholar 

  31. Liu, P., Wang, Y., Huang, D., Zhaoxiang, Z.: Recognizing occluded 3D faces usign an efficient ICP variant. In: IEEE International Conference on Multimedia and Expo (2012)

  32. Liu, R., Hu, R., Yu, H.: Nose detection on 3D face images by depth-based template matching. In: The 2014 7th international congress on image and signal processing (2014)

  33. Yu, X., Gao, Y., Zhou, J.: Boosting Radial String for 3D Face Recognition with Expressions and Occlusions. In: International Conference on Digital Image Computing: Tecniques and Applications (DICTA) (2016)

  34. Zhao, X., Dellandrea, E., Chen, L.: Accurate Landmarking of Three-dimensional facial data in the presence of facial expressions and occlusions using a three-dimensional statistical facial feature model. IEEE Trans. Syst. Man Cybern. 41(5), 1417–1428 (2011)

    Article  Google Scholar 

  35. Alyuz, N., Gokberk, B., Akarun, L.: Adaptive registration for occlusion robust 3D face recognition. In: European Conference on Computer Vision 7585 (2012)

  36. Srinivasan, A., Balamurugan, V.: Occlusion detection and image restoration in 3D face image. In: IEEE Region Conference TENCON (2014)

  37. Li, X., Da, F.: Efficient 3D face recognition handling facial expression and hair occlusion. Image Vis. Comput. 30(9), 668–679 (2012)

    Article  Google Scholar 

  38. Yin, L., Wei, X., Sun, Y., Wang, J., Rosato, M.J.: A 3D facial expression database for facial behavior research. In: FGR ’06: Proceedings of the 7th International Conference on Automatic Face and Gesture Recognition, pp. 211–216 (2006)

  39. Min, R., Hadid, A., Dugelay, J.L.: Efficient detection of occlusion prior to robust face recognition. Sci. World J. (2014)

  40. Li, H., Huang, D., Morvan, J.M., Wang, Y., Chen, L.: Towards 3D face recognition in the real: a registration-free approach using fine-grained matching of 3D keypoint descriptors. Int. J. Comput. Vision 113(2), 128–142 (2015)

    Article  MathSciNet  Google Scholar 

  41. Srivastava, A., Klassen, E., Joshi, S.H., Jermyn, I.H.: Shape analysis of elastic curves in euclidean spaces. IEEE Trans. Pattern Anal. Mach. Intell. 33(7), 1415–1428 (2011)

    Article  Google Scholar 

  42. Younes, L., Michor, P.W., Shah, J., Mumford, D.: A metric on shape space with explicit geodesics. Rendiconti Lincei—Matematica e Applicazioni 19(1), 25–57 (2008)

    Article  MathSciNet  MATH  Google Scholar 

  43. Samir, C., Srivastava, A., Daoudi, M.: Three-dimensional face recognition using shapes of facial curves. IEEE Trans. Pattern Anal. Mach. Intell. 28, 1858–1863 (2006)

    Article  Google Scholar 

  44. Samir, C., Srivastava, A., Daoudi, M., Klassen, E.: An intrinsic framework for analysis of facial surfaces. Int. J. Comput. Vision 82(1), 80–95 (2009)

    Article  Google Scholar 

  45. Samir, C., Srivastava, A., Daoudi, M.: Automatic 3D face recognition using shapes of facial curves. In: IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) (2006)

  46. Raed, M., Zou, B., Hiyam, H.: Nose tip detection in 3D face image based on maximum intensity algorithm. Int. J. Multimed. Ubiquitous Eng. 10(5), 373–382 (2015)

    Article  Google Scholar 

  47. Drira, H., Ben Amor, B., Daoudi, M., Srivastava, A.: Pose and expression invariant 3D face recognition using elastic radial curves. In: British Machine Vision Conference, pp. 1–11 (2010)

  48. Yu, X., Gao, Y., Zhou, J.: 3D face recognition under partial occlusions using radial strings. In: IEEE International Conference on Image Processing (ICIP) (2016)

  49. Gao, Y., Leung, M.: Human face profile recognition using attributed string. Pattern Recogn. 35, 353–360 (2002)

    Article  MATH  Google Scholar 

  50. Bhave, D., Choudhary, R., Gavali, R., Gholap, P.: 3D face recognition under expressions, occlusions, and pose variations. Int. J. Adv. Res. Comput. Commun. Eng. 5(3), 2270–2283 (2016)

    Google Scholar 

  51. Pan, G., Zheng, L., Wu, v.: Robust metric and alignment for profile-based face recognition: an experimental comparison. In: Proceedings of Workshop Applications of Computer Vision, pp. 1–6 (2005)

  52. Veltkamp, R.C., Haar, F.B.: A 3D face matching framework for facial curves. Graph. Models 71(2), 77–91 (2009)

    Article  Google Scholar 

  53. Li, C., Barreto, A., Zhai, J., Chin, C.: Exploring face recognition by combining 3D profiles and contours. In: IEEE Proceedings SoutheastCon 576–579 (2005)

  54. Samir, C., Srivastava, A., Daoudi, M.: Three-dimensional face recognition using shapes of facial curves. IEEE Trans. Pattern Anal. Mach. Intell. 28(11), 1858–1863 (2006)

    Article  Google Scholar 

  55. Kakadiaris, I.A., Passalis, G., Toderici, G., Murtuza, M.N., Lu, Y., Karampatziakis, N., Theoharis, T.: Three-dimensional face recognition in the presence of facial expressions: an annotated deformable model approach. IEEE Trans. Pattern Anal. Mach. Intell. 29(4), 640–649 (2007)

    Article  Google Scholar 

  56. Chang, K.I., Bowyer, K.W., Flynn, P.J.: Multiple nose region matching for 3D face recognition under varying facial expression. IEEE Trans. Pattern Anal. Mach. Intell. 28, 1695–1700 (2006)

    Article  Google Scholar 

  57. Faltemier, T., Bowyer, K.W., Flynn, P. J.: 3D face recognition with region committee voting. In: Third International Symposium on 3D Data Processing, Visualization, and Transmission 318–325 (2006)

  58. Mian, A.S., Bennamoun, M., Owens, R.: An efficient multimodal 2D–3D hybrid approach to automatic face recognition. IEEE Trans. Pattern Anal. Mach. Intell. 29(11), 1927–1943 (2007)

    Article  Google Scholar 

  59. Alyuz, N., Gokberk, B., Spreeuwers, L., Veldhuis, R., Akarun, L.: Robust 3D face recognition in the presence of realistic occlusions. In: 5th IAPR international conference on biometrics (ICB) (2012)

  60. Gonzalez, R.C., Woods, R.E.: Digital Image Processing (2007)

  61. Srinivas, T., Mohan, P., Shankar, R., Reddy, C.S., Naganjaneyulu, P.V.: Face recognition using PCA and bit-plane slicing. Lect. Notes Electr. Eng. 150, 515–523 (2013)

    Article  Google Scholar 

  62. Otsu, N.: In a threshold selection method from gray-level histograms. IEEE Trans. Syst. Man Cybern. 9(1), 62–66 (1979)

    Article  MathSciNet  Google Scholar 

  63. Nirmala Priya, G., Wahida Banu, R.S.D.: Occlusion invariant face recognition using mean based weight matrix and support vector machine. Sadhana 39(2), 303–315 (2014)

    Article  MATH  Google Scholar 

  64. Saxena, A., Chung, S.H.: 3D depth reconsrtuction from a single still image. Int. J. Comput. Vision 76(1), 53–69 (2008)

    Article  Google Scholar 

  65. Lu, X., Jain, A.: Automatic feature extraction for multiview 3D face recognition. In: Proceedings of the 7th International Conference on Automatic Face and Gesture Recognition, pp. 585–590 (2006)

  66. Passalis, G., Perakis, P., Theoharis, T.: Using facial symmetry to handle pose variations in real-world 3d face recognition. IEEE Trans. Pattern Anal. Mach. Intell. 33(10), 1938–1951 (2011)

    Article  Google Scholar 

  67. Passalis, G., Perakis, P., Kakadiaris, I.: 3D facial landmark detection under large yaw and expression variations. IEEE Trans. Pattern Anal. Mach. Intell. 35, 1552–1564 (2013)

    Article  Google Scholar 

  68. Peng, X., Bennamoun, M., Mian, A.: A training-free nose tip detection method from face range images. Pattern Recogn. 44, 544–558 (2011)

    Article  MATH  Google Scholar 

  69. Faltemier, T., Bowyer, K., Flynn, P.: Rotated profile signatures for robust 3d feature detection. In: Proceedings of the 8th IEEE International Conference on Automatic Face and Gesture Recognition, pp. 1–7 (2008)

  70. Spreeuwers, L.: Fast and accurate 3d face recognition. Int. J. Comput. Visi. IJCV11 93, 389–414 (2011)

    Article  MATH  Google Scholar 

  71. Gokberk, B., Irfanoglu, M., Akarun, L.: 3D shape-based face representation and feature extraction for face recognition. J. Image Vis. Comput. 24, 857–869 (2006)

    Article  Google Scholar 

  72. Salah, A., Alyuz, N., Akarun, L.: Registration of three-dimensional face scans with average face model. J. Electron. Imaging 17, 1–14 (2008)

    Google Scholar 

  73. Blanken, H., De Vries, A.P., Blok, H.E., Feng, L.: Multimedia Retrieval, pp. 163–165. Springer-Verlag, Berlin (2007)

    Book  MATH  Google Scholar 

  74. Briechle, K., Hanebeck, U.D.: Template matching using fast normalized cross correlation. Proc. SPIE Opt. Pattern Matching XII 4387, 95–102 (2001)

    Google Scholar 

  75. Maurer, T., Guigonis, D., Malov, I., Presenti, B., Tsaregorodtsev, A., West, D., Medioni, G.: Preformance of Geometrix ActiveID 3D Face Recognition engine on the FRGC data. IEEE CVPR (2005)

  76. Chang, K.I., Bowyer, K.W., Flynn, P.J.: Multiple nose egion matching for 3D face recogntion under varying facial expression. IEEE Trans. Pattern Anal. Mach. Intell. 28, 1695–1700 (2006)

    Article  Google Scholar 

  77. Calvo, A.R., Ruiz, F.T., Rurainsky, J., Eisert, P.: 2D-3D Mixed Face Recognition Schemes. In: Recent Advances in Face Recognition, pp. 125–149 (2008)

  78. Tsapatsoulis, N., Doulamis, N., Doulamis, A., Kollias, S.: Face extraction from non-uniform background and recognition in compressed domain. Proc. IEEE Int. Conf. Acoust. Speech Signal 5, 2701–2704 (1998)

    Google Scholar 

  79. Chang, K.I., Bowyer, K.W., Flynn, P.J.: An evaluation of multimodal 2D+3D face biometrics. IEEE Trans Pattern Anal. Mach. Intell. 27, 619–624 (2005)

    Article  Google Scholar 

  80. Samani, A., Winkler, J., Niranjan, M.: Automatic face recognition using stereo images. In: IEEE International Conference on Acoustics, Speech and Signal Processing (2006)

  81. Colombo, A., Cusano, C., Schettini, R.: 3D face detection using curvature analysis. Pattern Recogn. 39(3), 444–455 (2006)

    Article  MATH  Google Scholar 

  82. Boykov, Y.Y., Jolly, M.P.: Interactive graph cuts for optimal boundary & region segmentation of objects in nd images. Int. Conf. Comput. Vis. 1, 105–112 (2001)

    Google Scholar 

  83. Boykov, Y., Funka-Lea, G.: Graph cuts and efficient ND image segmentation. Int. J. Comput. Vision 70(2), 109–131 (2006)

    Article  Google Scholar 

  84. Colombo, A., Cusano, C., Schettini, R.: Detection and restoration of occlusions for 3D face recognition. In: IEEE International Conference on Multimedia and Expo, pp. 1541–1544 (2006)

  85. Everson, R., Sirovich, L.: Karhunen–Loeve procedure for gappy data. J. Opt. Soc. Am. A 12(8), 1657–1664 (1995)

    Article  Google Scholar 

  86. Xiaoli Li, F.D.: Efficient 3D face recognition handling facial expression and hair occlusion. Image Vis. Comput. 30, 668–679 (2012)

    Article  Google Scholar 

  87. Drira, H., Ben Amor, B., Srivastava, A., Daoudi, M., Slama, R.: 3D Face recognition under expressions, occlusions and pose variations. In: IEEE Transactions on Pattern Analysis and Machine Intelligence, Institute of Electrical and Electronics Engineers, pp. 2270–2283 (2013)

  88. Othmani, M., Bellil, W., Amar, C., Alimi, M.: A New structure and training procedure for multimother wavelet networks. IJWMIP 8(1), 149–175 (2010)

    MATH  Google Scholar 

  89. Gawali, S., Deshmukh, R.R.: 3D face recognition using geodesic facial curves to handle expression, occlusion and pose variations. Int. J. Comput. Sci. Inf. Technol. 5(3), 4284–4287 (2014)

    Google Scholar 

  90. Colombo, A., Cusano, C., Schettini, R.: Three-dimensional occlusion detection and restoration of partially occluded faces. J. Math. Imaging Vis. 40(1), 105–119 (2010)

    Article  MathSciNet  MATH  Google Scholar 

  91. U. D. R. C. i. S. Processing, FG2018 special session: is deep learning always the best solution for face recognition? https://udrc.eng.ed.ac.uk/about/news/20170725/fg2018-special-session-deep-learning-always-best-solution-face-recognition

  92. Kim, D., Hernandez, M., Choi, J., Medioni, G.: Deep 3D face identification. Computer Vision and Pattern Recognition (2017)

Download references

Author information

Authors and Affiliations

  1. Department of Management and Production Engineering, Politecnico di Torino, Corso Duca degli Abruzzi 24, 10129, Turin, Italy

    Nicole Dagnes, Enrico Vezzetti, Federica Marcolin & Stefano Tornincasa

Authors
  1. Nicole Dagnes
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Enrico Vezzetti
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Federica Marcolin
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Stefano Tornincasa
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Federica Marcolin.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Dagnes, N., Vezzetti, E., Marcolin, F. et al. Occlusion detection and restoration techniques for 3D face recognition: a literature review. Machine Vision and Applications 29, 789–813 (2018). https://doi.org/10.1007/s00138-018-0933-z

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00138-018-0933-z

Keywords

Search

Navigation