Computational caricaturization of surfaces

doi:10.1016/j.cviu.2015.05.013

Computer Vision and Image Understanding

Volume 141, December 2015, Pages 1-17

https://doi.org/10.1016/j.cviu.2015.05.013 Get rights and content

Highlights

•
We propose a caricaturization framework that is robust to natural poses and expressions.
•
The proposed model links between the two schools of thoughts of computerized caricaturization.
•
It is simple to implement and works efficiently as part of a realtime system.
•
We used the proposed caricaturization method for generating musical video clips.

Abstract

The question whether a caricature of a 2D sketch, or an object in 3D can be generated automatically is probably as old as the attempt to answer the question of what defines art. In an attempt to provide a partial answer, we propose a computational approach for automatic caricaturization. The idea is to rely on intrinsic geometric properties of a given model that are invariant to poses, articulations, and gestures. A property of a surface that is preserved while it undergoes such deformations is self-isometry. In other words, while smiling, running, and posing, we do not change much the intrinsic geometry of our facial surface, the area of our body, or the size of our hands. The proposed method locally amplifies the area of a given surface based on its Gaussian curvature. It is shown to produce a natural comic exaggeration effect which can be efficiently computed as a solution of a Poisson equation. We demonstrate the power of the proposed method by applying it to a variety of meshes such as human faces, statues, and animals. The results demonstrate enhancement and exaggeration of the shape’s features into an artistic caricature. As most poses and postures are almost isometries, the use of the Gaussian curvature as the scaling factor allows the proposed method to handle animated sequences while preserving the identity of the animated creature.

Introduction

A caricature is an illustration of an object in which some features are exaggerated. The terms feature exaggeration usually refers to an operation that relates measures of a given object to those of a reference one. For example, a small mouth, relates the size of a given mouth to its average in a given population. The procedure we have in mind is one that emphasizes and extends such discrepancies. An interesting question is what happens when there is no reference population or an average to relate to. At the other end, it would be intriguing to design a caricaturization procedure that would have the same effect on an articulated object which is not affected by its pose, or posture (Fig. 1).

The oldest caricatures known today were drawn in ancient Egypt at about 3000 BC [2]. Later, the Greek and the Roman Empires used caricatures to intimidate people and spread their propaganda. The characters in these ancient caricatures were often monstrous hybridizations of humans and animals, see for example Fig. 2. In the fifteenth century, there was a change in the attitude towards caricatures. It has probably started with Leonardo da-Vinci, who in his study and exploration for shapes and forms searched for people with extremely deformed facial characteristics, that he used for his scientific art. Followers of da-Vinci started to emphasize properties of faces by exaggerating features in their drawings.

Drawing professional caricatures is a skill that requires long practice and innate talent. Skilled caricaturists are intimately familiar with human faces and their saddle variations. They can perceptually differentiate key features in one’s face from an average one. Before drawing, a caricaturist finds the discrepancy of common parts and memorizes them. Then, she draws the subject while amplifying the more distinct characteristics. The process is described in Fig. 3. The choice of features and the ability to exaggerate while maintaining the identity is where art comes into play. Obviously, personal touch and drawing skills would lead to different projections onto different artists perceptions of reality.

Several papers attempt to automate a process of caricature drawing. The main efforts in this area can be divided into model-based and geometry-based approaches. The model based procedures are usually designed specifically for human faces that are represented by some parametric model. The discrepancies between the representations of a given subject face and an average one are exaggerated. In the geometry-based approach, intrinsic features of a shape are exaggerated. Here, we show how the proposed framework bridges between these two models.

While designing an automatic exaggeration procedure a fundamentally important property is robustness to isometric deformations, such as rigid body transformations, as well as changes of expressions and poses. Based on the above arguments, the two axioms we would like to have for an automatic shape caricaturization procedure are thereby

•
The ability to extend discrepancies with respect to a reference model.
•
Robustness or invariance to isometric deformations.

Indeed, when exaggerating a shape in motion, a key principle is to ensure that the processed sequence of exaggerated shapes are all related in a similar fashion to the original one. This property would enable applying the algorithm to animated figures and 3D scans while preserving the identity of the modified creature. Here, we propose a caricaturization framework that obeys these guidelines. Our axiomatic construction is based on the analysis provided in [4] by which facial expressions and articulations are almost isometries. Next, the simplest differential isometry invariant measure is the Gaussian curvature. We show how to efficiently modify the geometry of a given surface such that its area is locally amplified by some scalar function of the Gaussian curvature. The proposed model links between the two schools of thoughts of computerized caricaturization. Given the correspondence between the subject shape and a reference average shape, the relation between the curvatures at corresponding points determines the local stretching of the shape. The results of the algorithm are demonstrated to be robust to almost isometric deformations. It is simple to incorporate the proposed framework as part of a real-time system using the implementation consideration we provide.

The organization of this paper is as follows. We start by reviewing caricature generation related papers in Section 2. We next introduce the proposed approach by briefly reviewing a classical surface modification technique to which we apply our axiomatically designed amplification factor in Section 3. Details about discretization and boundary conditions are given in Section 5. Section 6 discusses experimental results of the model applied to synthetic and real surfaces, as well as potential applications. We conclude with future directions in Section 7.

Section snippets

Related work

The challenge of automatically generating caricatures occupied researchers since the 1980’s. Brennan [3] introduced the idea of exaggerating the discrepancies between a sketch of a subject and an average face. She developed an interactive system in which the user picks feature points on a facial image. Simple interpolation of these points provides a sketch of the faces. Then, the sketch is aligned and registered to an average one, and the distances between matched feature points are exaggerated

Notations

Consider the boundary of a 3D shape as a surface or a two dimensional manifold $M \subset R^{3}$ given in a parametric form as $S : Ω \subset R^{2} \to M$ . Throughout this exposition, the surface M will be described by its coordinates $S (u, v) = {x (u, v), y (u, v), z (u, v)},$ where x, y, and z are the coordinate-functions of the manifold.

The first partial derivatives of the parametric form, S_u and S_v, are two vectors that are tangent to the surface and linearly independent. Thus, one can set the basis for the tangent plane T_pM, about a

Isometry preservation

We define two isometric surfaces as parametrized surfaces between which an isometric map exists (see Section 3.1). By construction, the proposed surface enhancement model is robust to isometric transformations. That is, given two isometric surfaces S₁

S₂, if we can find corresponding surfaces

{\tilde{S}}_{1}

and

{\tilde{S}}_{2}

that satisfy Eq. (20) we also have that

{\tilde{S}}_{1}

{\tilde{S}}_{2}

Theorem 4.1

If the surfaces S₁ and S₂ are isometric, and there exist two surfaces ${\tilde{S}}_{1}$ and ${\tilde{S}}_{2}$ such that $\begin{matrix} \nabla_{G_{1}} {\tilde{S}}_{1} = | K_{1} | \nabla_{G_{1}} S_{1}, \end{matrix}$ and $\begin{matrix} \nabla_{G_{2}} {\tilde{S}}_{2} = | K_{2} | \nabla_{G_{2}} S_{2}, \end{matrix}$ then, ${\tilde{S}}_{1}$ and ${\tilde{S}}_{2}$

Implementation considerations

One possible implementation of the proposed framework is when approximate continuous surfaces by triangulated meshes. In what follows, we briefly review discrete approximations of a continuous function defined on the surface, its gradient, the surface Gaussian curvature, a discrete divergence operator [24], and a discrete Laplace–Beltrami operator [25]. We conclude by combining these elements together for solving the system of equations (20), with some additional constraints that would fix, for

Results and discussion

We evaluate the proposed caricature generation framework by conducting and discussing several example use cases. First, we demonstrate its applicability by caricaturizing a number of synthetic surfaces and captured geometries. Then, we discuss the performance analysis of a real-time implementation of the proposed algorithm. Next, we examine the robustness of the method to almost isometric deformations. We conclude the section by proposing two possible applications.

Conclusions

We introduced a novel caricature generation framework for surfaces. The method scales the gradient fields of the surface coordinates by a function of the Gaussian curvature of the surface, and solves a corresponding Poisson equation for finding the exaggerated shape. When a reference shape is provided, local discrepancies are used to amplify the scaling effect, while in the absence of a reference shape, the reference is assumed to be a scaled down version of the given one thereby letting the

Acknowledgement

Model sources include: [37] Barack Obama : by Claudio Gasparini and Autodesk [36]; Athena : by Alex Baranov; Max Planck : by Christian Rssl and Jens Vorsatz of Max-Planck-Institut fr Informatik; Armadillo: Stanford Computer Graphics Laboratory; This work has been supported by grant agreement no. 267414 of the European Community’s FP7-ERC program.

References (37)

S. Osher et al.
Fronts propagating with curvature-dependent speed: algorithms based on hamilton-jacobi formulations
J. Comput. Phys.
(1988)
T. Lewiner et al.
Interactive 3d caricature from harmonic exaggeration
Comput. Graph.
(2011)
A. Baranov, <http://www.3dcadbrowser.com/download.aspx?3dmodel=76578>,...
T. Wright
A history of caricature and grotesque in literature and art
(1865)
S.E. Brennan
Caricature generator: the dynamic exaggeration of faces by computer
Leonardo
(1985)
A.M. Bronstein et al.
Expression-invariant representations of faces
Image Process. IEEE Trans.
(2007)
V. Blanz, T. Vetter, A morphable model for the synthesis of 3d faces (1999)...
E. Akleman et al.
Making extreme caricatures with a new interactive 2d deformation technique with simplicial complexes
Proceedings of Visual
(2000)
E. Akleman et al.
Modeling expressive 3d caricatures
ACM SIGGRAPH 2004 Sketches
(2004)
Y.-K. Lim et al.
3d caricature generation system on the mobile handset using a single photograph
Proceedings of the International Conference on Parallel Processing Workshops, 2007. ICPPW 2007
(2007)

L. Clarke et al.

Automatic generation of 3d caricatures based on artistic deformation styles

Vis. Comput. Graph. IEEE Trans.

(2011)

A. Steiner et al.

Planar shape enhancement and exaggeration

(1996)

M. Eigensatz et al.

Curvature-domain shape processing

Computer Graphics Forum

(2008)

B. Kim et al.

Geofilter: Geometric selection of mesh filter parameters

Computer Graphics Forum

(2005)

G. Cimen et al.

Perceptual caricaturization of 3d models

Computer and Information Sciences III

(2013)

Y. Yu et al.

Mesh editing with poisson-based gradient field manipulation

(2004)

M. Chuang et al.

Interactive and anisotropic geometry processing using the screened poisson equation

(2011)

M.P. Do Carmo et al.

Differential Geometry of Curves and Surfaces

(1976)

Cited by (32)

Landmark Detection and 3D Face Reconstruction for Caricature using a Nonlinear Parametric Model
2021, Graphical Models
Citation Excerpt :
Vieira et al. [57] proposed a method based on deformations by manipulation of moving spherical influence zones. Sela et al. [48] presented a framework to scale the gradient fields of the surface coordinates by a function of the Gaussian curvature of the surface and solve a corresponding Poisson equation to find the exaggerated shape. Besides, there are some works on modeling 3D caricatures from images.
Caricature is an artistic abstraction of the human face by distorting or exaggerating certain facial features, while still retains a likeness with the given face. Due to the large diversity of geometric and texture variations, automatic landmark detection and 3D face reconstruction for caricature is a challenging problem and has rarely been studied before. In this paper, we propose the first automatic method for this task by a novel 3D approach. To this end, we first build a dataset with various styles of 2D caricatures and their corresponding 3D shapes, and then build a parametric model on vertex based deformation space for 3D caricature face. Based on the constructed dataset and the nonlinear parametric model, we propose a neural network based method to regress the 3D face shape and orientation from the input 2D caricature image. Ablation studies and comparison with state-of-the-art methods demonstrate the effectiveness of our algorithm design. Extensive experimental results demonstrate that our method works well for various caricatures. Our constructed dataset, source code and trained model are available at https://github.com/Juyong/CaricatureFace.
Total variation diffusion and its application in shape decomposition
2020, Computers and Graphics (Pergamon)
One challenge in shape decomposition is to capture correct boundaries between different parts and get piecewise constant results. Based on the good edge-preserving and sparsity properties of total variation regularization, this paper introduces a novel diffusion model by minimizing weighted total-variation energy with Dirichlet boundary constraints. By the total variation diffusion model, we propose an edge-preserving shape decomposition optimization model, which can be solved effectively by augmented Lagrangian method with each subproblem having closed form solution. A number of experiments display that our method can produce segmentation results with piecewise constant parts and feature-preserving boundaries for both meshes and 3D point clouds, especially for shapes with sharp features. In addition, for mesh segmentation, our results compare favorably to those obtained by several existing techniques when evaluated on the Princeton Segmentation Benchmark. Furthermore, the quantitative errors show that the algorithm is robust numerically and the computational costs are reasonable.
Intrinsic and extrinsic operators for shape analysis
2019, Handbook of Numerical Analysis
Citation Excerpt :
As an example of generalizing signal processing to meshes, the spectral filtering methods proposed in Vallet and Lévy (2008) apply low-pass, high-pass, and band-exaggeration filters to eigenvalues, leading to effects of mesh smoothing, details sharpening, and band-feature enhancement, respectively. Relatedly, locally amplifying surface area according to its Gaussian curvature produces natural exaggeration effects (Sela et al., 2015). Descriptors like heat and wave kernel signatures in Section 4.3 provide powerful tools to find keypoints that can be candidates for landmark correspondences.
Geometric operators are common objects in surface-based shape analysis and geometry processing. While the intrinsic Laplace–Beltrami operator has been a ubiquitous choice thanks to its intuitive and often desirable properties, it fails to capture the spatial embedding of a shape because it discards extrinsic information; furthermore, it is not always sensitive to the geometric features relevant for a given shape analysis task. To address these challenges, several alternative operators for shape analysis have been proposed in recent work, with an emphasis on operators sensitive to extrinsic features. Many operators appearing in previous work on other problems also encode aspects of extrinsic geometry and are potentially suitable for shape analysis. In this survey, we unify discussion of operators for shape analysis, highlighting key theoretical properties as well as their numerical discretizations. Additionally, we provide numerical experiments on model tasks in the operator-based shape analysis pipeline, including computation of descriptors, distances, and segmentations, to demonstrate the effect of using different operators on the qualitative behaviour of algorithms in this space.
Variational sparse diffusion and its application in mesh processing
2024, Engineering Computations (Swansea, Wales)
An Elastic Basis for Spectral Shape Correspondence
2023, Proceedings - SIGGRAPH 2023 Conference Papers
TEXTure: Text-Guided Texturing of 3D Shapes
2023, Proceedings - SIGGRAPH 2023 Conference Papers

View all citing articles on Scopus

View full text

Computational caricaturization of surfaces

Highlights

Abstract

Introduction

Section snippets

Related work

Notations

Isometry preservation

Implementation considerations

Results and discussion

Conclusions

Acknowledgement

J. Comput. Phys.

Comput. Graph.

A history of caricature and grotesque in literature and art

Caricature generator: the dynamic exaggeration of faces by computer

Leonardo

Expression-invariant representations of faces

Image Process. IEEE Trans.

Making extreme caricatures with a new interactive 2d deformation technique with simplicial complexes

Proceedings of Visual

Modeling expressive 3d caricatures

ACM SIGGRAPH 2004 Sketches

3d caricature generation system on the mobile handset using a single photograph

Proceedings of the International Conference on Parallel Processing Workshops, 2007. ICPPW 2007

Automatic generation of 3d caricatures based on artistic deformation styles

Vis. Comput. Graph. IEEE Trans.

Planar shape enhancement and exaggeration

Curvature-domain shape processing

Computer Graphics Forum

Geofilter: Geometric selection of mesh filter parameters

Computer Graphics Forum

Perceptual caricaturization of 3d models

Computer and Information Sciences III

Mesh editing with poisson-based gradient field manipulation

Interactive and anisotropic geometry processing using the screened poisson equation

Differential Geometry of Curves and Surfaces