Elsevier

Pattern Recognition Letters

Volume 107, 1 May 2018, Pages 33-40
Pattern Recognition Letters

Fast and robust occluded face detection in ATM surveillance

https://doi.org/10.1016/j.patrec.2017.09.011Get rights and content

Highlights

  • We propose a novel potential energy function to approximate the head-shoulders Omega shape with a Gaussian function.

  • We develop a fast and robust head tracking algorithm, which utilizes gradient and shape cues in a Bayesian framework.

  • A novel face occlusion verification method is proposed.

  • We tested on, and published, our real-world data to demonstrate its performance.

Abstract

Crimes with respect to ATMs (Automatic Teller Machines) have attracted more and more attention, where criminals deliberately cover their faces in order to avoid being identified. This paper proposes a fast and robust face occlusion detection algorithm for ATM surveillance, which is demonstrated to be effective and efficient to handle arbitrarily occluded faces. In this algorithm, we innovatively propose to make use of the Omega shape formed by the head and shoulder of the person for head localization to tackle severe face occlusion. For this purpose, we first construct a novel energy function for elliptical head contour detection. Then, we develop a fast and robust head tracking algorithm, which utilizes the gradient and shape cues in a Bayesian framework. Lastly, to verify whether a detected head is occluded or not, we propose to fuse information from both skin color and face structure using the AdaBoost algorithm. Experimental results on real world data show that our proposed algorithm can achieve 98.64% accuracy on face detection and 98.56% accuracy on face occlusion detection, even though there are severe occlusions in faces, at a speed of up to 12 frames per second.

Introduction

As a kind of popular financial terminals, ATMs are widely distributed all over cities. When using the ATM, a customer is asked to insert a bank card and input password for personal identification. ATM crimes such as peeping passwords and beating machine maliciously [19] occur constantly. Usually, ATM criminals wear sunglasses, masks or helmets to cover their faces in order to avoid being identified by video surveillance systems. If a surveillance system can detect these abnormal activities automatically and alarm security staff promptly, it will greatly improve the financial security of banks and facilitate the detective work of the police. Up till now, there have been many developmental systems about ATM surveillance. A more comprehensive survey of existing works is presented in Section 2.

Among various existing approaches [1], [5], [11], [30], [37], [38], [39], [40], detecting users’ facial images are widely used in practice, where the acquired facial images are crucial for verifying occlusion. Some recent methods are proposed related to face recognition and classification [8], [9], [10], [26]. However, the performance of existing systems degrades significantly when dealing with faces with severe occlusions.

In ATM related crimes, in order to avoid their face picture being captured, suspects often occlude their eyes and mouths. This has become an obstacle for many existing approaches. Therefore, this paper targets the severe occlusion problem, and proposes a fast and robust face occlusion detection algorithm by innovatively making use of the Omega shape formed by the head and shoulder of the person. Note that, this work simply targets to justify whether the face in front of an ATM has been occluded or not. Whether a person is a criminal or not will need to be further investigated by security or police. Our contributions are mainly three folded:

1) First, we propose to make use of the Omega shape formed by the head and shoulder of the person for accurate localization of heads to tackle severe face occlusion. This overcomes the problems with most existing approaches that rely on the presence of facial organs, which are not available under severe occlusion. For this purpose, we construct a new energy function for elliptical head contour detection.

2) Secondly, we develop a fast and robust head tracking algorithm, which utilizes gradient and shape cues in a Bayesian framework. This not only avoids unnecessary computation and thus speeds up the system but also contributes to a high detection rate.

3) Lastly, to verify whether a detected face is occluded, we propose to combine a skin color classifier and a face template matching classifier using the AdaBoost algorithm. This helps to address the problems associated with skin color variation due to different races, sexes, and ages as well as uneven illumination.

The rest of this paper is organized as follows. Section 2 reviews related existing works. Section 3 describes our proposed energy-based head localization and elliptical head tracking algorithm. Section 4 describes our classifier based face occlusion verification method. In Section 5, experimental results and analysis are presented. Finally, conclusions are drawn in Section 6.

Section snippets

Related works

The work described in this paper targets the problem of face occlusion detection, which aims to determine whether a captured head image is with occlusion or not in ATM surveillance. This problem constitutes the steps of many computer vision and surveillance applications, such as face recognition, face retrieval, face tracking, etc. Verifying whether there is face occlusion from an image is generally done in two steps, i.e., head region detection which is to locate the head regions from images,

Proposed occluded face detection scheme

In this section, we first present our proposed solution for detecting faces with severe occlusions. We create a potential energy function and convert the problem of extracting head contour to an energy minimization problem.

Face occlusion verification

In order to determine whether a face is occluded or not, we build two weak classifiers, each considering one type of face features only, i.e., the skin color and face shape respectively. Then, the AdaBoost learning algorithm is employed to learn a strong classifier with a higher accuracy from the cascaded weak classifiers.

Pre-processing

Pan et al. [33] proposed a content similarity based fast reference frame selection algorithm for reducing the computational complexity of the multiple reference frames based interframe prediction. Based on the best motion vector selection correlation among the different size prediction modes, Pan et al. [34], [35] proposed a fast motion estimation (ME) method to reduce the encoding complexity. The proposed algorithm can yield a quite promising coding performance in terms of RD performance and

Conclusion

In this paper, we have constructed a simple but fast and robust face occlusion detection algorithm for ATM surveillance applications. We innovatively made use of the Omega shape formed by head and shoulder and proposed a novel potential energy function for head contour detection. We also developed a fast and robust head tracking algorithm, which utilizes the gradient and shape cues in a Bayesian framework. The newly developed head detection module has been demonstrated to be robust for its

Acknowledgments

This research was partly supported by National Natural Science Foundation, China (Nos. 61702226, 61672263, 21365008), the Natural Science Foundation of Jiangsu Province (Grants nos. BK20170200, BK20161135).

References (43)

  • S. Birchfield

    Elliptical head tracking using intensity gradients and color histograms

    Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition

    (1998)
  • I. Choi et al.

    Facial fraud discrimination using detection and classification

    Int. Conf. Adv. Vis. Comput.

    (2010)
  • D. Costa et al.

    Face recognition using support vector machine and multiscale directional image representation methods: a comparative study

    Int. Joint. Conf. Neural Netw.

    (2015)
  • A. Dhall et al.

    Finding happiest moments in a social context

    ACCV

    (2012)
  • C.X. Ding et al.

    Robust face recognition via multimodal deep face representation

    ACM Trans Intell Syst Technol

    (2015)
  • C.X. Ding et al.

    A comprehensive survey on pose-invariant face recognition

    ACM Trans. Intell. Syst. Technol.

    (2016)
  • W. Dong et al.

    Image-based fraud detection in automatic teller machine

    Int. J. Comput. Sci. Netw. Security

    (2006)
  • V. Ferrari et al.

    Progressive search space reduction for human pose estimation

    Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition

    (2008)
  • S. Fidler et al.

    Combining reconstructive and discriminative subspace methods for robust classification and regression by subsampling

    IEEE Trans. Pattern. Anal. Mach. Intell.

    (2006)
  • G.D. Finlayson et al.

    On the removal of shadows from images

    IEEE Trans. Pattern. Anal. Mach. Intell.

    (2006)
  • R.M. Gurav et al.

    Real time finger tracking and contour detection for gesture recognition using opencv

    Int. Conf. Ind. Instrument. Cont.

    (2015)
  • Cited by (45)

    • Collaborative algorithms that combine AI with IoT towards monitoring and control system

      2021, Future Generation Computer Systems
      Citation Excerpt :

      We conclude the paper and discuss the direction of our future work in Section 6. Designing an appropriate system detection framework for IoT is difficult because IoT is heterogeneous [2–5], concurrent by nature, and sensitive to timing. Some difficulties happen all the time due to the difficult fusion problem of hardware and software when IoT uses some pattern recognition algorithms for computation [6–9].

    • Self-learning soft computing algorithms for prediction machines of estimating crowd density

      2021, Applied Soft Computing
      Citation Excerpt :

      Each function module describes a state, implemented by one or more machine learning algorithms [69–77]. Mode changes characterize the relationship of each functional module of the system, and several approaches were explored to solve deadlines misses problems [78–80]. The proposed IoT-based prediction machine in this paper uses those function modes to keep the security, robustness and reliability of designed system; in addition, the proposed prediction machine switches to stop mode immediately to prevent damage to the system in the event of a failure or unexpected event.

    View all citing articles on Scopus
    View full text