Tracking regions of human skin through illumination changes

https://doi.org/10.1016/S0167-8655(02)00327-6Get rights and content

Abstract

New human computer interfaces are using computer vision systems to track faces and hands. A critical task in such systems is the segmentation. An often used approach is colour based segmentation, approximating the skin chromaticities with a statistical model, e.g. with mean value and covariance matrix. The advantage of this approach is that it is invariant to size and orientation and fast to compute. A disadvantage is that it is sensitive to changes of the illumination, and in particular to changes in the illumination colour.

This paper investigates (1) how accurately the covariance matrix of skin chromaticities might be modelled for different illumination colours using a physics-based approach, (2) how this may be used as a feature to classify between skin and other materials. Results are presented using real image data taken under different illumination colours and from subjects with different shades of skin. The eigenvectors of the modelled and measured covariances deviate in orientation about 4°. The feature to distinguish skin from other surfaces is tested on sequences with changing illumination conditions containing hands and other materials. In most cases it is possible to distinguish between skin and other objects.

Introduction

Computer vision based tracking of human faces and hands has many applications, e.g. in human computer interfaces and surveillance systems. Different cues may be used for tracking, such as motion, shape, and colour. Each of these cues may fail under certain conditions, e.g. nothing is moving, rapid changes in shape, or changing illumination, respectively. In order to get more robust systems it has been suggested to fuse the information of multiple cues (Crowley and Berard, 1997; Triesch and von der Malsburg, 2000; Spengler and Schiele, 2001).

An often used cue is skin-colour segmentation because it is invariant to size and orientation and fast to compute. Several approaches have been proposed, some statistically based, e.g. (McKenna et al., 1999; Yang et al., 1998), and some physics based, e.g. (Soriano et al., 2000; Störring et al., 2001). A problem when using skin colour as a feature arises under varying lighting conditions. In particular changes in the spectral composition of the scene illumination may result in failures of colour segmentation methods (Funt et al., 1998).

Yang et al. (1998) and Störring and Granum (2002) showed that the facial skin chromaticity distribution of an individual under a single light source may be approximated by a multivariate normal distribution in the red–green chromaticity plane. Yang et al. (1998) proposed an adaptive statistical skin-colour model updating the mean vector and the covariance matrix of the red–green chromaticities as the lighting conditions change. The model is used in a real-time face tracker and works under slightly changing indoor illumination conditions.

McKenna et al. (1999) use Gaussian mixtures to model the skin-colour distribution in Hue-Saturation space. The model parameters are updated over time in order to adapt to changes in illumination and viewing direction.

A problem with adapting the parameters of a statistical colour model during tracking is the lack of ground-truth of the region of interest (McKenna et al., 1999), i.e. the colour model might adapt to image regions which do not belong to the skin-coloured object and, hence, result in false positives and/or false negatives. Particularly under rapid illumination changes, e.g. from indoor to outdoor illumination when opening the curtains or blinds of a window, the skin chromaticities are changing significantly.

In (Störring et al., 2001) skin chromaticities for different illuminations are modelled with a good approximation by a physics-based approach. The model uses knowledge about the camera parameters and assumes that commonly used in- and outdoor light sources can be modelled by blackbody radiators (Finlayson and Schaefer, 2001). The skin chromaticities for a variety of illuminations with different correlated colour temperatures (CCTs) form a ‘skin locus’ which follows the curvature of the Planckian locus of blackbody radiators, Fig. 2. This might be used to constrain the search area for skin colour in the chromaticity plane.

Soriano et al. (2000) presented a face tracking system working outdoors under changing illumination conditions. They constrained the search area by the skin locus. Inside the skin locus a non-parametric skin-colour model is learned and updated by histogram backprojection. Histogram backprojection has the same drawback as adapting the statistical model, i.e. the histogram might adapt to non-skin-coloured objects in the background.

Adapting to non-skin objects might be avoided if the statistical skin-colour model would be constrained by physics-based knowledge about possible skin distributions. This is investigated in this paper, and in particular,

  • (1)

    how accurately may the eigenspace of a skin chromaticity covariance matrix be modelled for illuminations with arbitrary CCTs by using a physics-based approach;

  • (2)

    using the modelled eigenspace, how well may a skin area be distinguished from other skin-like objects under varying illumination, given a reference image of the skin area is available.

The paper is organized as follows: Section 2 gives an overview on statistical and physics-based modelling of skin colour. Section 3 proposes a method to model the covariance matrix of skin chromaticities under arbitrary illumination, which is used in Section 4 to derive a feature to classify between skin and other objects. In Section 5 experimental results are given which is followed by a discussion and conclusions.

Section snippets

Theory and background

This section provides a brief overview of the reflection properties of human skin and how they might be approximated with a statistical and a physics-based model.

Adapting statistical models to changing illumination

The method proposed in this paper uses physics-based knowledge to estimate how the statistical model will change as the illumination changes. A necessary condition for this is to find how these types of models can be related.

Firstly, we consider how the distributions of the skin chromaticities change as illustrated in Fig. 1 (see also Fig. 4, Fig. 5). They change position along the skin locus and the major and minor axes (eigenspace of the covariance matrix) change in orientation and aspect

Skin-colour feature for segmented regions

This section proposes a feature that indicates which of i=1,…,K (pre-segmented) image areas Di corresponds to a skin model that had been initialised using a reference skin area in some previous image frame. Such a feature is of interest in tracking applications to classify a segmentation result, in particular after rapid illumination changes when the camera needs some frames to adapt the shutter speed and iris before providing useful image data.

In the previous section we have suggested a

Experiments

In the following two sets of experiments will be presented. First the method to model the covariance matrix (Section 3) is tested using one reference CCT and three test CCTs. Images of faces of eight subjects having different ethnic backgrounds (China, Iran, Cameroun, Latvia, Greece, Spain, Denmark, and India) were captured, so that altogether 32 images were used. For each reference there are three test images used to compare the estimated statistical model with the measurements.

Secondly, the

Discussion

This paper proposed (1) a method to estimate the covariance matrices and eigenspaces of an individual’s skin chromaticity distribution for illuminations with arbitrary CCTs, and (2) its application as a skin-colour feature for segmented regions. We have tested (1) for eight very different skin types, and with three illuminations for each. The average of the absolute deviation between measured and estimated orientations of the eigenspaces is about 4° and the maximum about 10°. The average

Conclusions

In this paper the linking of a statistical with a physics-based skin-colour model was investigated. It was demonstrated that the covariance matrix and the eigenspace of skin chromaticities can be modelled for different illuminations using a physics-based approach. The average orientation error is about 4°, the average deviation between the lengths of the vectors is about 6%.

Furthermore, using the proposed method a feature was suggested indicating which area in an image is most likely to be

References (17)

  • S.J McKenna et al.

    Tracking colour objects using adaptive mixture models

    Image Vision Comput.

    (1999)
  • M Störring et al.

    Physics-based modelling of human skin under mixed illuminants

    J. Robot. Autonom. Syst.

    (2001)
  • H.J Andersen et al.

    Classifying illumination condition from two light sources by colour histogram assessment

    J. Opt. Soc. Amer. A

    (2000)
  • R.R Anderson et al.

    Optical radiation transfer in the human skin and applications in in vivo remittance spectroscopy

  • Crowley, J.L., Berard, F., 1997. Multi-modal tracking of faces for video communications. In: IEEE Conf. on Computer...
  • G.D Finlayson et al.

    Solving for colour constancy using a constrained dichromatic reflection model

    Internat. J. Comput. Vision

    (2001)
  • Funt, B., Barnard, K., Martin, L., 1998. Is machine colour constancy good enough? In: Burkhardt, H., Neumann, B....
  • T Ohtsuki et al.

    Using color and geometric models for extracting facial features

    J. Imag. Sci. Technol.

    (1998)
There are more references available in the full text version of this article.

Cited by (29)

  • Human skin detection: An unsupervised machine learning way

    2024, Journal of Visual Communication and Image Representation
  • A survey of skin-color modeling and detection methods

    2007, Pattern Recognition
    Citation Excerpt :

    Skin-color modeling and detection is then applied on these color corrected images. Störring et al. [21] have used a physics-based model to solve the problem of color constancy for skin detection. The physics-based model describes an expected area for the skin chromaticities under certain illuminations.

  • Significance of Color Spaces and Their Selection for Image Processing: A Survey

    2022, Recent Advances in Computer Science and Communications
View all citing articles on Scopus
View full text