CAD-Deform: Deformable Fitting of CAD Models to 3D Scans

Abstract

Shape retrieval and alignment are a promising avenue towards turning 3D scans into lightweight CAD representations that can be used for content creation such as mobile or AR/VR gaming scenarios. Unfortunately, CAD model retrieval is limited by the availability of models in standard 3D shape collections (e.g., ShapeNet). In this work, we address this shortcoming by introducing CAD-Deform (The code for the project: https://github.com/alexeybokhovkin/CAD-Deform), a method which obtains more accurate CAD-to-scan fits by non-rigidly deforming retrieved CAD models. Our key contribution is a new non-rigid deformation model incorporating smooth transformations and preservation of sharp features, that simultaneously achieves very tight fits from CAD models to the 3D scan and maintains the clean, high-quality surface properties of hand-modeled CAD objects. A series of thorough experiments demonstrate that our method achieves significantly tighter scan-to-CAD fits, allowing a more accurate digital replica of the scanned real-world environment while preserving important geometric features present in synthetic CAD environments.

V. Ishimtsev and A. Bokhovkin—equal contribution.

A. Artemov—Technical lead.

Appendices

A Statistics on the Used Datasets

In Tables 4 and 5, we summarize statistical information on the number of instances and categories considered in our evaluation. As we require parts annotations as an important ingredient in our deformation, we only select instances in Scan2CAD  [5] where the associated parts annotation in PartNet  [31] is available, resulting in total in 9 categories (25%), 572 instances (18%), and 1979 annotated correspondences (14%). Note that the vast majority of cases remain within our consideration, keeping our evaluation comprehensive.

Table 4. Overall statistics on the numbers of categories, instances, and correspondences present in our datasets.

Table 5. The top 15 most frequent ShapeNet categories in Scan2CAD dataset including a detailed information on those with the availability of the corresponding parts annotations.

We further select the most well-presented six shape categories as our core evaluation set, outlined in Table 5. Note that as our method is non-learnable, we can just as easily experiment with the remaining categories, at the cost of somewhat reduced statistical power.

B Optimization Details

Our full experimental pipeline is a sequence of deformation stages with different optimization parameters, and Hessian being recomputed before each stage. Specifically, we perform one part-to-part optimization with parameters $\alpha _{\text {shape}} = 1, \alpha _{\text {smooth}} = 0, \alpha _{\text {sharp}} = 0, \alpha _{\text {data}} = 5\times 10^{4}$ for 100 iterations, then we perform 5 runs of nearest-neighbor deformation for 50 iterations with parameters $\alpha _{\text {shape}} = 1, \alpha _{\text {smooth}} = 10, \alpha _{\text {sharp}} = 10, \alpha _{\text {data}} = 10^3$. Such number of iterations was sufficient to achieve convergence with energy changes less than $10^{-1}$ in our experiments. Runtime of our method breaks into cost computation ($\mathtt {\sim }0.3$ s), backward ($\mathtt {\sim }0.2$ s), and optimization steps containing the main bottleneck (sparse matrix-vector multiplication) ($\mathtt {\sim }1.2$ s) for a typical $10^4$ vertices mesh. All operations can be easily further optimized.

C Qualitative Fitting Results

In Fig. 6, we display a series of qualitative results with a variety of shape deformations with different classes of instances. Comparing to baselines, our framework achieves accurate fit while preserving sufficient perceptual quality.

Table 6. Quantitative results of local surface quality evaluation using DAME measure  [41] (the smaller, the better, normalized to a maximum score of 100), where our CAD-Deform compares favourably to the baselines across all considered categories. Note, however, how surface quality significantly decreases when smoothness and sharp feature-related terms are dropped.

Table 7. Results of LSLP-GAN reconstruction in terms of Earth-Mover’s Distance between reconstructed and original point clouds of mesh vertices.

Table 6 reports the results of surface quality evaluation using deformations obtained with our CAD-Deform vs. the baselines, category-wise. While outperforming the baseline methods across all categories, we discover the smoothness and sharpness energy terms to be the crucial ingredients in keeping high-quality meshes.

Figure 7 displays visually the deformation results using the three distinct classes, highlighting differences in surfaces obtained using the three methods.

Table 7 reports shape abnormality evaluation results across the six considered categories. Baselines show (Fig. 8) low reconstruction quality as evidenced by a larger number of black points. In other words, comparing to CAD-Deform, the distance from these meshes to undeformed ones is mush larger.

In Fig. 9, we show a series of examples for CAD-Deform ablation study. Perceptual quality degrades when excluding every term from the energy.

Table 8. Comparative evaluation of our approach in terms of Accuracy on different levels of detail.

Table 9. Comparative evaluation of ARAP deformations w.r.t. the change of Laplacian term weight in terms of Accuracy (%).

D Morphing

In this section, we present an additional series of examples of morphing properties (Fig. 10). Every iteration of optimization process gradually increases the quality of fit. With CAD-Deform we can morph each part to imitate the structure of the target shape.

E PartNet Annotation

This set of experiments shows how quality of fitting depends on mesh vertices labelling. We can provide labels for mesh in different ways depending on the level in PartNet hierarchy [31]. We observe the increase of fitting quality with greater level of detail (Table 8). Examples presented in Fig. 11 are selected as the most distinguishable deformations on different levels. There are minor visual differences in deformation performance of part labeling level.

F Fitting Accuracy Analysis

CAD-Deform deformation framework is sensitive to Accuracy threshold $\tau$ for the distance between mesh vertices and close scan points. In Fig. 12 variation of $\tau$ threshold is presented and we selected $\tau = 0.2\text {~m}$ for fitting Accuracy metric.

G Perceptual Assessment and User Study Details

Having obtained a collection of deformed meshes, we aim to assess their visual quality in comparison to two baseline deformation methods: as-rigid-as-possible (ARAP)  [38] and Harmonic deformation  [7, 26], using a set of perceptual quality measures. The details of our user study design and visual assessment are provided in the supplementary. To this end, we use original and deformed meshes to compute DAME and reconstruction errors, as outlined in Sect. 6.1, and complement these with visual quality scores obtained with a user study (see below). These scores, presented in Table 3, demonstrate that shapes obtained using CAD-Deform have $2\times$ higher surface quality, only slightly deviate from undeformed shapes as viewed by neural autoencoders, and receive $2\times$ higher ratings in human assessment, while sacrificing only 1.1–4.5 % accuracy compared to other deformation methods.

Table 10. Comparative evaluation of Harmonic deformations w.r.t. the change of Laplacian term weight in terms of Accuracy (%).

Fig. 6.

Qualitative shape deformation results using obtained using ARAP  [38], Harmonic deformation  [7, 26], and our CAD-Deform. Mesh surface is colored according to the value of tMMD measure, with darker values corresponding to the larger distance values.

Fig. 7.

Qualitative comparison of deformations obtained using ARAP  [38], Harmonic deformation  [7, 26], and our CAD-Deform, with shapes coloured according to the value of DAME measure  [41]. Our approach results in drastic improvements in local surface quality, producing higher-quality surfaces compared to other deformations.

Fig. 8.

Qualitative comparison of reconstruction of point clouds extracted from mesh vertices. These meshes are obtained using ARAP  [38], Harmonic deformation  [7, 26], and our CAD-Deform, the first column corresponds to original undeformed meshes. The color of reconstructed point clouds is related to Earth-Mover’s Distance between reconstructed and original point clouds of mesh vertices.

Design of Our User Study. The users were requested to examine renders of shapes from four different categories: the original undeformed shapes as well as shapes deformed using ARAP, Harmonic, and CAD-Deform methods, and give a score to each shape according to the following perceptual aspects: surface quality and smoothness, mesh symmetry, visual similarity to real-world objects, and overall consistency. Ten random shapes from each of the four categories have been rendered from eight different views and scored by 100 unique users on a scale from 1 (bad) to 10 (good). The resulting visual quality scores are computed by averaging over users and shapes in each category.

In Fig. 13, we present a distribution of user scores over different deformation methods and shapes. It can be clearly seen that users prefer our deformation results to baselines for all of the cases, which is obvious from the gap between histogram of CAD-Deform and ARAP/Harmonic histograms. At the same time, shapes deformed by CAD-Deform are close to undeformed ShapeNet shapes in terms of surface quality and smoothness, mesh symmetry, visual similarity to real-world objects, and overall consistency. Besides, in Tables 9, 10 we provide numbers for evaluation of ARAP/Harmonic deformations w.r.t. the change of Laplacian term weight.

Fig. 9.

Qualitative results of ablation study usind our deformation framework, with mesh coloured according to the value of the tMMD measure.

Fig. 10.

Qualitative shape translation results, interpolating between the original mesh (left) and the target mesh (right).

Fig. 11.

Deformation performance depending on different level of labelling from the PartNet dataset [31]. Deformed mesh surfaces are colored according to the value of tMMD measure, with darker values corresponding to the larger distance values.

Fig. 12.

Fitting Accuracy vs. varying $\tau$ threshold for the distance between mesh vertices and close scan points.

Fig. 13.

Distribution of user scores averaged by ten shapes from original ShapeNet [9], meshes deformed with ARAP [38], Harmonic [7, 26] and CAD-Deform.