Face alive
Introduction
The idea of simulating facial expressions by 3D face model was formulated by Frederic I. Parke in 1972 [1]. Since then researchers have attempted to generate realistic facial models and animation. There are some of the areas where a good model of human face is essential for success. For example, a good 3D facial model could be used in such applications as face and facial expression recognition, computer synthetic facial surgery, video teleconferencing, and models for the analysis of facial expression by psychologists of nonverbal communication. The common challenge has been to develop facial models that not only look real, but which are also capable of synthesizing the various nuances of facial motion quickly and accurately. In facial animation, there exists two important issues: structure modeling and deformation modeling. The first one relates to the development of accurate representations of the 3D geometry of the facial structure. The second one applies to the efficient deformation of the facial skin shape to generate flexible expressions.
For the structure modeling, it is desirable to have a refined topological mesh that captures the facial surface. In the literature, most facial animation approaches were based on the surface model range from standard polygonal surface meshes [2], [3], [4], [5], [6] to parametric surfaces [7], [8], [9]. All these facial models are adapted from a template face with low resolution instead of generated directly from the underlying data set, and therefore could not represent a personalized face in an anatomically accurate and realistic way. Some image-based facial modeling methods [10], [11], [12] used shape features extracted from facial images to modify a generic model. Although this kind of technique can provide a reconstructed face model easily, the 3D shape is not completely accurate. So far, the most accurate modeling can be achieved by using range scanning technology. The automated laser range (LR) scanner allows detailed facial geometries and corresponding texture image to be extracted quickly. Based on the highly accurate range and color data, it is possible to develop a realistic personalized face model with structured information for further animation.
For the deformation modeling, early works restricted themselves to pure geometric modeling. However, with physically based modeling paradigms, more realistic facial models arose. The physically based models are based on either a particle system (see e.g. [13]) or a continuous system (see e.g. [14]). The model using the particle system is based on discretizing the body into a number of particles, whose connectivity is maintained through constraint forces. In contrast, in the model based on the continuous system, the equations of motion describe the complete (rigid and nonrigid) motion of an object in a single system of equations. For computer animation, the models based on the particle system are more popular. This is primarily due to the fact that their simple formulation is easily implementable and supports topological and geometric flexibility through the local geometric operations.
However, there are several issues with existing facial models that still lack adequate solutions:
- •
Spring approximation: The existing particle-system-based models adopted linear or piecewise linear approximation to the skin's mechanical characteristics. Though this assumption simplifies somewhat the equation of motion at each node, it is undesirable for accurate simulation of the real tissue that has a nonlinear stress–strain characteristics.
- •
Multi-layer soft tissue: Most face models were based on the surface model. Surface model is conceptually simple, providing ease of control and rapid computation. But it has little relation to physical reality of soft tissue with different layers, and therefore is inherently unable to capture the behavior of complex tissue structures.
- •
Skull base: Skull plays an important role in the articulation of the jaw and constrains the deformation of the skin. The incorporation of the skull structure into face model also facilitates the construction of contractile muscles at anatomically correct positions. However, in previous work the skin–skull interface has not been emphasized.
- •
Facial muscle construction: Automatic construction of facial muscles in the face remains a problem. In some previous work, although the muscle structures have been modeled, the correctness of the locations of facial muscles cannot be verified due to either the absence of underlying skull or the interactive muscle insertion. Particularly, the latter requires the user to go through a series of trials and corrections in the 3D space, clearly making this manual procedure the bottleneck in face modeling.
- •
Fast computational algorithm: Computational complexity is a fundamental constraint in facial animation. Research on facial animation has generally relied on explicit numerical integration (such as Euler's method or Runge–Kutta methods) to advance the simulation. Unfortunately, all these explicit methods suffer from a problem: very small time steps are required to ensure stability. Therefore, they can only be considered as conditionally stable and can explode numerically during the simulation. Moreover, for the fast computation of the underlying deformation and force model, efficient data structures and algorithms that can process high-resolution models have not been developed.
The simulation techniques described in this paper are developed to address these issues to the extent required to make an accurate individualized facial model for the realistic expression synthesis. By exploiting the laser range data obtained from scanning a subject, a facial mesh precisely representing the skin geometry is reconstructed. Based on the reconstructed facial mesh, we develop a multi-layer MSD skin model to dynamically simulate the nonhomogenous behavior of the real skin. The model takes into account the nonlinear stress–strain relationship of the skin and the fact that soft tissue is almost incompressible. Instead of approximating nonlinear behavior of soft tissue by linear springs, a kind of nonlinear spring is developed to directly simulate the dynamic skin deformation. Three kinds of muscle models are developed to simulate the distribution of muscle force applied on the skin. The 3D face model incorporates a skull structure which extends the scope of facial motion, facilitates facial muscle construction, and constrains skin deformation. The facial muscle construction is achieved by using an efficient muscle mapping approach that ensures the muscles to be semi-automatically located at the anatomically correct positions with reduced manual intervention. When muscles contract, the deformation of facial skin is computed by solving the underlying dynamic equation. For efficient simulation, we propose an adaptive simulation algorithm. It employs either a dynamic or a quasi-static simulator for the numerical simulation by taking advantage of the facts that facial deformations are local and facial soft tissues are well damped. By propagating forces in an ordered fashion through the facial mesh, the governing equation is adapted locally in terms of approximation quality and the computational load is concentrated on the facial regions that undergo significant deformations.
The outline of the paper is as follows: Section 2 reviews pertinent research on face modeling and animation. An overview of our system architecture is presented in Section 3. Section 4 describes the procedures of facial data acquisition from laser range scans. 5 Deformable facial skin model, 6 Modeling of facial muscles, 7 Skull model and facial muscle construction present our methods to modeling of three structural components of the face (the skin, muscles and skull) for construction of a physical face model with anatomically different layers. Section 8 illustrates the computational model we developed for efficient numerical simulation. Section 9 presents the simulation results and analyzes the performance of the system. Section 10 draws conclusions.
Section snippets
Previous work
Facial modeling and animation has been a problem of interest in the community of computer graphics for decades. A variety of approaches have been advocated and can be classified into two groups: geometric animation and physically based animation [15].
In the geometrical facial animation, the most intuitive approach is morphing between fixed and variable polygon topology [1], [16]. This keyframing method is easy to implement and can effectively create facial expressions in a short amount of
System overview
Our new approach has been integrated into a system for physically based animation of reconstructed hierarchical face model of a specific person. Fig. 1 shows the two-part block diagram of the system.
The hierarchical facial modeling part uses the range and reflectance data of the individual face measurements to generate a photorealistic face model combining anatomically different layers. Starting from the procedures of data acquisition and preprocessing, we reconstruct an accurate geometric face
Data acquisition and preprocessing
We use a Minolta VIVID 700 DigitizerTM to acquire the geometry and color information of the facial surface. To recover the whole facial geometry, scans are taken from several different view angles of the subject's face. When each scan is complete, the scanner captures a 200×200 laser range map (Fig. 2(a)). The range map can be transformed into a 3D mesh representing the surface of the person. In addition to the range map, a 400×400 reflectance (RGB) image (Fig. 2(b)) that registers the surface
Deformable facial skin model
The realism of facial skin deformation can be greatly enhanced by modeling its dynamic behavior physically, taking into account its mechanical characteristics. In the following subsections, we first analyze the skin physiology to determine the attributes that are necessary for realistic modeling, then a deformable multi-layer skin model that intends to model and simulate the true biological forms and functions of the human skin is presented.
Modeling of facial muscles
In the general sense, muscles are the organs of facial motion. By their contractions, they move the various parts of the face to generate expressions. The facial muscles tend to be of the flat, diffuse variety more like the smooth muscles of the gut than the cylindrical muscles used for locomotion and manipulation in the arms and legs. They can be defined according to the orientation of the individual muscle fibers that may be parallel, oblique or spiralized relative to the direction of pull at
Skull and jaw
We use a generic skull model to map general anatomical attributes to the facial surface. It consists of an immovable skull and a rotating jaw. In the skull model fitting process, affine transformations—rotation, translation and nonuniform scaling—are applied on the skull model by the user interactively. Fig. 12 shows facial skin mesh and the shape of the skull after alignment.
To distinguish between the skin part that lies over the skull or over the jaw, we have mapped the 3D facial surface and
Computational model for facial animation
We consider the deformable facial skin model and its configuration (shape) before deformation. Under the action of a field of forces, the facial skin deforms to a new configuration . Then the problem consists of determining the displacement field which associates to the position of any point of the face before deformation, its position in the final configuration.
Results
Our 3D face modeling and expression animation system is programmed with C++/OpenGL and runs on an Intergraph Z×10 with dual Pentium III , memory. After 3D data of the reconstructed facial mesh (vertices’ Euclidean positions and connections) is read into the system, the multi-layer soft tissue is assembled automatically and the integration of the soft tissue with underlying muscle and skull models to build up a face model with hierarchical structure takes only a few minutes. In the
Conclusion and future work
We have presented the techniques to producing natural looking animations of a 3D face model of a specific person. In various preprocessing steps a facial mesh that accurately captures the facial geometry is reconstructed from laser range scans. Based on the human face anatomy, we developed a physical face model with hierarchical structure, incorporating the skin, muscle and skull. The proposed multi-layer MSD skin model takes into account the nonlinear stress–strain relationship of soft tissue
References (46)
- F.I. Parke, Computer generated animation of faces, Master's Thesis, University of Utah, Salt Lake City,...
Extending the range of facial types
Journal of Visualization and Computer Animation
(1991)- I. Essa, S. Basu, Modeling, tracking and interactive animation of facial expressions and head movements using input...
Parameterized models for facial animation
IEEE Computer Graphics and Application
(1982)- S. Platt, N. Badler, Animating facial expressions, Proceedings of SIGGRAPH’81, Dallas, Texas, 15 (1981)...
- et al.
Physically-based facial modeling, analysis and animation
The Journal of Visualization and Computer Animation
(1990) - D. DeCarlo, D. Metaxas, M. Stone, An anthropometric face model using variational techniques, Proceedings of...
- et al.
Animation of a B-spline figure
The Visual Computer
(1988) - C.L.Y. Wang, D.R. Forsey, Langwidere: a new facial animation system, Proceedings of Computer Animation’94, Geneva,...
- et al.
Automatic creation of 3D facial models
IEEE Computer Graphics and Application
(1993)
Constructing a 3D individualized head model from two orthogonal views
The Visual Computer
Computer Facial Animation
Feature-point-driven facial animation using a hypothetical face
Computer Graphics Forum
Cited by (23)
Reinforcement learning coupled with finite element modeling for facial motion learning
2022, Computer Methods and Programs in BiomedicineCitation Excerpt :In addition, for investigating muscles driving facial motion problem, biomechanical models are recommended because they can be customized to reflect the true anatomy and mechanical deformations as well as imitate physical processes [39]. In fact, physics-based facial models using finite element methods have been intensively developed to explore the role of the facial muscle excitation, contraction and coordination during facial motion [9-16, 31,32]. Muscle excitation represents the neural control process, which contracts the face tissues and moves the skull to perform facial expressions and movements.
An observation based muscle model for simulation of facial expressions
2018, Signal Processing: Image CommunicationCitation Excerpt :Zhang and Prakash [30] elaborated on this idea and formally proposed a sheet muscle model. This model has been a part of the majority of mass–spring–damper (MSD) simulations in the literature [31–37]. In our implementation muscle fibers are defined on the wireframe vertices.
Modelling facial expressions: A framework for simulating nonlinear soft tissue deformations using embedded 3D muscles
2013, Finite Elements in Analysis and DesignCitation Excerpt :Analysing muscle contraction at the whole face level presents even more fascinating challenge as computer modellers must also consider the directional mechanical contribution of individual muscle fascicles that are part of a complex muscular network. Almost all early computational models of the human face treated muscular structures as series of line segments [9–11]. However, these linear 1-D models were incapable of recreating the paths of muscles with complex geometry.
Quantitative analysis of facial movement-A review of three-dimensional imaging techniques
2009, Computerized Medical Imaging and GraphicsMAUI avatars: Mirroring the user's sensed emotions via expressive multi-ethnic facial avatars
2006, Journal of Visual Languages and ComputingCitation Excerpt :We also chose Haptek software because it provides some built-in facial expressions, which we can use directly to encode expressions for sad, happy, surprised, and angry, each with varying degrees of intensity. It should be noted, however, that our approach currently focuses on using the avatar to visualize the user's perceived emotional state, rather than on building sophisticated animated models of the face as seen in Zhang [33], a type of research which could complement our approach as it reaches maturity. Our approach is also different, and complementary, to approaches developing mark-up languages to control behavior of life-like characters during interaction dialogues or during web-based interactions, as with APML [34–36]—an XML-based language allowing to specify the meaning of a communicative act then instantiated to an expressive communicative signal, or as with MPML [37]—an XML-style language enabling authors to script rich web-based interaction scenarios featuring life-like characters.
Modeling and recognition of emotions in manufacturing
2022, International Journal on Interactive Design and Manufacturing