Deceiving faces: When plastic surgery challenges face recognition

Highlights

• A survey about face recognition after plastic surgery is presented.

• Approaches to the problem and to related topics are resumed and discussed.

• Results reported in literature are compared and analyzed.

Abstract

An exponential growth of the number of plastic surgery treatments specific to face (from the minimally-invasive ones to the real surgical procedures) has characterized the last two decades and it seems likely that this phenomenon, that has social and cultural meanings and implications, could spread even further in the next years as the average cost of these treatments is lowering and the wish for “beautification” is becoming part of the global esthetics sense. For these reasons, face recognition as an established research topic has a new major challenge: delivering methods capable of high recognition accuracy even in case probe and gallery differ by a surgical alteration of face shape. To this aim is of fundamental importance understanding the range and the extent of the modification produced by the various types of treatments or by a combination of them. We present a survey of the state of the art on this topic, starting by an analysis of the diffusion of the facial plastic surgery and describing the key aspects of each of the most statistically relevant treatments available, resumed by a synthetic table. The paper includes a brief description of all the approaches proposed in the field so far to the best of authors' knowledge and a comparison of the performance reported by the existing methods when applied to the most referenced plastic surgery face dataset to date. A critical discussion of the results achieved so far and an insight about the challenges that still have to be addressed concludes this work.

Introduction

In the scientific literature on face recognition, an introduction is often found reporting the positive features of this biometric trait (universality, acceptability and collectability, resistance to circumvention and recognition accuracy) as well as its peculiar weakness to environmental variations such as lighting, pose and occlusion, and a wide range of intra-class variations related to expressions, aging and other voluntary (piercing, tattoos, make-up, etc.) and involuntary (scars, moles, skin diseases, facial traumas, etc.) modifications of the face appearance. However, it is worth noting that in the battle to improve methods robustness to the aforementioned challenges, there is an implicit assumption of an “overall consistence of face shape” between the enrolled template and the probe image. In other terms, the type and the level of intra-class variations should not alter too deeply the overall physiognomy. In this sense, while a wide range of expression or lighting variations represent a typical addressed issue, it is obvious (when considering age variations) that nobody would expect a high recognition accuracy by comparing a child to a boy or a boy to an adult or even an adult to an old man, as during these stages of life facial features are subject to dramatic changes often undermining overall face's shape consistence.

Though this example might seem too extreme, when it comes to facial plastic surgery, the aforementioned assumption can possibly become not true anymore even within the same age group, depending on the extent and on the type of the procedure performed. It is also important to highlight that, in its very nature, facial plastic surgery aims to improving facial appearance or restoring the original one, either according to a deliberate choice of the patient, motivated by esthetic or psychological reasons, or due to functional needs, the former being largely the most diffused case.

The wish and sometimes the urge for a surgical improvement of face aspect is surely not new and it is related to a complex set of social-cultural factors strongly fostered by mass-media, but it is constantly increased in the last years as reported by the American Society of Plastic Surgeons (ASPS), representing 94% of all board-certified plastic surgeons in the U.S. and among the largest plastic surgery specialty organizations in the world. According to this source [1], indeed, on a total of 14.6 million cosmetic plastic surgery procedures performed in the United States in 2012, more than 10 million pertained to face, of which more than 9 million were minimally-invasive procedures including botulinum toxin injections (6.1 million), soft-tissue fillers (2 million), chemical peel (1.1 million), microdermabrasion (1 million) while the rest were actual surgical procedures including nose reshaping (0.25 million), eyelid surgery (0.2 million), facelift (0.13 million) and a number of statistically less relevant procedures like chin augmentation, cheekbone reshaping or ear reshaping. Another interesting statistic concerns the distribution of cosmetic procedures (both surgical and minimally invasive) among genders and ethnic groups. Indeed, while not surprisingly females account for almost 91% of total cosmetic procedures, males undergoing plastic surgery are reaching 10% with a constant increase year after year. With regard to ethnic group distribution, while Hispanics are almost stable, there is a 6–7% increase in Caucasians and African Americans and a 21% increase in Asian Americans. This wish for face “beautification” has inspired researches like the ones by Eisenthal et al. [2] and Leyvand et al. [3], exploring aspects such as facial attractiveness and virtual face enhancement. However, as the figures reported above concern only the United States and considering that globalization applies to this field as well, it is easy to understand the potential relevance on a worldwide scale of this scenario to the topic of face recognition in each of its applications. The impact of cosmetic surgery may vary from a light surface remodeling to a deep change in subject's main physiognomic traits, representing a serious challenge for recognition algorithms and, sometimes, for human recognition capability as well, as witnessed by cases of individuals undergoing multiple procedures and resulting in a completely remodeled face geometry. Facial plastic surgery basically operates through reduction, augmentation or reshaping of face at a local or global level. From a more geometrical point of view the resulting visible effects can be subtractive, additive or both, while skin level (epidermis, dermis) changes can also lead to a modified surface appearance (color, texture). Consequently, face recognition after plastic surgery represents a topic that, besides being already relevant in terms of statistics, push many of the most diffused and recognized techniques for facial features extraction and matching to their limits, as it forces to focus on the very essence of what makes a face that particular face to the aim of defining selected features possibly invariant to surgical procedures.

In the attempt to answer to these open questions, this paper presents a comprehensive survey on the state of the art about this interesting and stimulating topic. Besides resuming the key aspects of all the approaches to face recognition specifically aimed to plastic surgery available to date, we report about the main research trends emerging through a comparison of the methodologies proposed so far which range from classical descriptors like Principal Component Analysis (PCA), Fisher Discriminant Analysis (FDA), Geometric Features (GF), Local Feature Analysis (LFA), Local Binary Patterns (LBP), Speeded Up Robust Features (SURF) and Neural Network based Gabor Transform (GNN), to less common approaches based on Evolutionary-Genetic algorithm (GA), Particle Swarm Optimization (PSO), Partitioned Iterated Function System (PIFS) and Structural Similarity Image Maps (SSIM).

As the specificity of each plastic surgery procedure has an impact to specific face regions that could be worth to consider for the research, we describe extensively each of the available procedures by means of a textual and graphical profile, also resumed in a comparative table. We also include a detailed description of the only face dataset specific to facial plastic surgery publicly available to date providing before–after shots. Finally we discuss the results provided in the field trying to analyze the potential and the limits of the main classes of methodologies (e.g. holistic vs region based, etc.) with the aim to draw a future path of research for improving existing techniques and developing new ones.

The reminder of the paper is organized as follows. Section 2 provides a detailed description of each of the most statistically relevant facial plastic surgery procedures performed, while Section 3 provides a summary of each of the works published so far specifically on the topic of face recognition after plastic surgery, to the best of author's knowledge and includes a comparison of the algorithms described according to a common set of objective data. Section 4 briefly recalls the features of the most cited publicly available face dataset with before/after surgery images. Section 5 presents an in depth discussion about the various aspects of the topic, trying to analyze the results achieved so far and the main open issues. Section 6 concludes the paper by summarizing our findings.