A Polynomial Surface Fit Algorithm for Filling Holes in Point Cloud Data

Abstract

In this paper, we propose a novel framework for detecting and filling missing regions in point cloud data. We propose to investigate the properties of point cloud data and develop a framework that can detect boundaries of intricate holes in the point cloud data of complex shapes and fill the hole consequently. The holes in point cloud data are caused owing to many reasons like reflectance, transparency, occlusions etc. Detecting holes in point cloud data is a non-trivial task, since point cloud data is unstructured and comes with no adjacency/connectivity information. We propose a Centroid-Shift algorithm that exploits the distance of cluster centroid from the member points to detect the boundaries of holes, further we propose a polynomial surface fit framework to accurately fill the missing regions/holes without losing the original shape attributes of the objects. We demonstrate our framework on popular 3D objects, we provide qualitative results and report RMS error as the evaluation metric to measure the effectiveness of the hole-filling algorithm.

1 Introduction

In this paper¹. We propose a novel framework to detect missing regions in point clouds and consequently fill them without losing the original shape attributes of the objects involved. Point cloud representations are becoming popular owing to rise of 3D scanners, which help us acquire data in an unstructured point cloud data format. These point clouds come equipped with certain properties like normals and colors, however the surface information is not explicitly known and has to be determined by inferring the relationship of these points with their neighborhood. Missing regions in point clouds are a result of infidel 3D scanners, reflectance, occlusions and/or transparency. Even with high-fidelity 3D scanners, it is possible to end up with missing regions in final integrated models because of high grazing angles, missing regions in the original model, limited number of range scans from different viewing directions etc. While point clouds are easy to represent and manipulate compared to mesh representations, certain trivial operations often encountered in geometry modeling, like determining hole boundaries and filling holes become ill-posed problems. Hole detection and filling finds important applications in 3D shape representation and analysis, surface reconstruction, 3D model completion etc. One of the most direct methods of detecting hole boundaries and then filling the holes is to convert the point clouds into meshed representations, the techniques include Delaunay Triangulation [4] which uses the point clouds as vertices for forming a triangle mesh or by using surface reconstruction methods like Poisson [8]. The problem with these methods apart from being computationally expensive is that they don’t work well in presence of large missing regions in the original point clouds, this prompts for a framework that can directly work on the original point clouds. In this work, we propose a new algorithm called Centroid-Shift that exploits the geometric properties like distance of point cloud cluster centroids from member points to detect the candidate hole boundaries in point clouds. We also propose a post-processing step that helps in filtering out the external boundary, unwanted edges and corners of the objects involved. Finally, we describe a Polynomial Surface Fit framework that is used to fill in the missing regions without the loss of original shape attributes of the objects involved. Towards this we make the following contributions:

• 1. We propose a framework, which we call the Centroid-Shift algorithm that helps in selecting the candidate boundary points of all potential holes/missing regions.

• 2. We propose a post-processing step based on underlying surface variations of the point clouds to eliminate pseudo-boundary points and unwanted edges.

• 3. Finally, we propose a Polynomial Surface Fit hole-filling algorithm that can be used to fill/complete missing regions in point clouds without losing the original shape attributes of the objects.

2 Related Work

Detection of holes and filling missing regions have been explicitly discussed in the context of mesh representations [1, 7, 10]. Recently with the rise of affordable 3D scanners, point cloud representations have become ubiquitous and various techniques to address the problems of detecting missing regions and hole-filling in point clouds have become apparent. While its easy for a human being to visually identify the boundaries of holes by just looking, the problem is non-trivial when it comes to algorithmically detecting the boundaries of missing regions in order to fill them. Several approaches that exploit various geometric properties have been proposed, in [6] the authors propose an angle criterion in which the point clouds are projected onto a tangent plane and are sorted radially and then the angle between the consecutive samples is measured, the larger the angle gap the more the probability of the points being boundary-points. Authors in [2] take this approach further and propose multiple criteria to detect the boundary points. They propose to use an angle criterion, a weighted-average criterion that assumes the boundary points are homeomorphic to half-disk and finally an ellipsoidal criterion that uses the eigenvalues which encode the shape of the ellipsoid. Finally, all these criteria are combined in a weighted average manner to detect boundary points. After detecting the boundaries of the missing regions, they have to be filled such that the original shape attributes of the object is not deformed. Since the underlying geometric shape is not apparent from just point clouds, it becomes difficult to ascertain what the missing region is. Authors in this survey paper [5] provide a detailed account of various hole-filling methods.

3 Proposed Methodology

In this section, we discuss our hole-boundary detection and hole-filling algorithm. Figure 1 outlines the proposed approach.

Fig. 1.

In a certain point cloud with a large missing region, the euclidean distance $d_i$ is calculated between a candidate boundary point $P_i$ and the centroid $C_i$ of a cluster. The points, which are farther from the centroid are possible boundary points. Such points are accumulated to detect the boundaries of the holes, later the pseudo-boundary points are discarded using a post-processing step.

3.1 Hole Boundary Detection

In this section, we describe the technique to detect the boundaries of missing regions, which we refer to as Centroid-Shift algorithm. This framework gives us a set of candidate boundary points, which could be plausible boundaries of missing regions, we further use a refining step to filter out pseudo-boundary points. Here, our main contribution lies in refining the detected boundaries and discarding the pseudo-boundaries and unwanted edges that are detected during the hole-boundary detection step. Consider a set of point clouds with a large missing region, we use the K-Nearest Neighbor (KNN) to find the neighbouring points of a query point, the K value is set heuristically based on the density of the point clouds, as a general rule we set the K value as 5% of the point cloud density, now for a query point $P_i$. We compute the centroid of these K neighbouring points. After computing the centroid of the sample, we compute the euclidean distance $d_i$ between all the K neighbouring points and the cluster centroid. Similarly, we compute for all the points. The points whose distance is greater than the quarter of the mean of the distance of the neighbor points, given by the Eq. 1 are considered the candidate boundary points [3]. The proposed method is shown in Fig. 1.

$$\begin{aligned} d_i > \frac{\sum _{1}^{k}|p_i - c_i|}{4k} \end{aligned}$$

(1)

3.2 Filtering Pseudo-Boundary Points

The centroid-shift algorithm detects the object boundaries and unwanted edges which satisfy the criteria mentioned above but are not actually holes/missing regions. The different types of pseudo-boundary holes detected are shown in Fig. 3. To discard the false boundary points, we consider surface variations. The process involves already computed k-nearest neighbors and then computing the surface variations using eigen vectors. If the surface variation is found to be greater than certain threshold t, those points are deemed as pseudo-boundary points. The proposed algorithm for both boundary detection and eliminating the pseudo points is highlighted below.

The algorithm involves computing surface variation using the eigen vectors via the co-variance matrix of the neighborhood points. The co-variance matrix is calculated as given by the equation below,

$$\begin{aligned} cov (x,y) = \frac{\sum _{1}^{K}(N_{xmean} - n_{ix}) (N_{ymean} - n_{iy})}{k-1} \end{aligned}$$

(2)

The eigen vectors are used to calculate the surface variation as follows, and a surface variation greater than some threshold t is considered as pseudo-boundary.

$$\begin{aligned} Surface Variation = \frac{e_3(P_i)}{e_1(P_i) + e_2(P_i) + e_3(P_i)} \end{aligned}$$

(3)

The value of k and t is heuristically set to achieve best possible scenario of objects whose pseudo-boundary points are not detected as actual holes.

Fig. 2.

The different cases of pseudo-boundary holes detected. In this first figure (from left) the query point is at a corner, in the second figure the point is located at a valley and in the last figure the query point is located at an edge.

In Fig. 3 we have experimented with varying k and t value to obtain object without pseudo-boundary points. In Fig. 3a, the k value and t value was set to 30 and 1.2 respectively, in Fig. 3b the values were set to k = 30 and t = 0.1, in Fig. 3c the values were k = 20 and t = 0.2 and for Fig. 3d the values were set to k = 20 and t = 0.1. By varying the k value and threshold value, we can obtain an optimum detection of all actual holes.

Fig. 3.

We have experimented with varying k and t value to obtain object without pseudo-boundary points.

Fig. 4.

Projecting the hole boundaries onto eigen-space provides a proper sense of spatial orientations of hole boundaries allowing us to efficiently fill them. In figure a, the hole boundaries are shown in euclidean space while in figure b the holes are projected onto eigen-space.

3.3 Hole Filling via Polynomial Surface Fit

In this section, we describe our proposed hole-filling algorithm. After detecting the candidate boundary points and eliminating the pseudo boundary points we aim to fill the detected missing regions to ensure better surface reconstruction. Our algorithm involves fitting an optimum polynomial surface patch and then sampling the neighborhood, most of the 3d objects can be approximated within 4th order polynomial fit. The holes in euclidean space doesn’t give a proper sense of hole orientation and allows for overlapping of closely located holes making it difficult to preserve the sampling density of the original point cloud and to ascertain the patch necessary for filling the holes. Hence, we project the individual hole on to its euclidean plane, which in turn is computed via neighbouring points of boundary and thus reducing the problem of reconstructing holes in 3D to the simpler interpolation problem. Figure 5 shows projecting the holes onto eigen space as opposed to euclidean space. The steps involved in filling the holes/missing regions are outlined below:

• 1. For an identified true hole boundary consisting of m points, $\mathbf B = \{b_1, b_2, b_3, ... b_m\}$

• 2. Compute the centroid of the hole boundary points using the equation,

  $$\begin{aligned} Centroid = (\frac{\sum _{i=1}^{m}x_i}{m},\frac{\sum _{i=1}^{m}y_i}{m}, \frac{\sum _{i=1}^{m}z_i}{m} ) \end{aligned}$$

  (4)

• 3. Compute a learning radius R consisting of learning points that will help us ascertain the order of the surface fit as follows,

  $$\begin{aligned} Learning Radius (R) = 2(\frac{\sum _{i=1}^{m}|Centroid - b_i}{m}) \end{aligned}$$

  (5)

• 4. For the learning points L = $\{L_1, L_2, L_3, . . . L_n\}$ compute the covariance matrix and calculate the eigen vectors as outlined in Eq. 2.

• 5. Project all the learning points onto the eigen space to fit a higher order surface F(x,y). Projecting of learning points onto eigen space helps in efficient interpolation of higher order surface holes even though they have complex spatial orientations.

• 6. Interpolate the new points I = $\{p_1, p_2, p_3,...p_j\}$ using regression fit F(x,y) as $P_{iz} = F(P_{ix}, P_{iy})$. $P_{ix}$ and $P_{iy}$ are considered such that the interpolated points preserve the point density of original point cloud. i.e $P_{ix}$ and $P_{iy}$ should be uniformly distributed.

• 7. Finally the eigen-space projected points are reprojected back to the original space via inverse transformation using already computed eigen vectors.

Fig. 5.

After detecting the hole boundaries and eliminating the pseudo-boundaries, we apply our polynomial surface algorithm to fill the missing region. The figure shows the region is faithfully filled without distorting the original shape attributes of the model as reported by the RMS error in Table 1.

Fig. 6.

Comparison of 3D reconstruction performed before and after hole filling.

4 Results

In this section, we present the results of our hole detection and filling framework. We perform various experiments on popular 3D models and report RMS error as a quantitative evaluation metric to assess the effectiveness of our hole filling algorithm. In Fig. 5 (left), we perform hole filling on a lady statue using our proposed framework. In Figs. 5 (right) and 6 we have compared surface reconstruction with and without hole filling, the top row in Fig. 5 shows the Sanford bunny model surface reconstructed without filling in missing regions and the below model shows surface reconstruction performed after hole filling. Clearly, filling the missing regions using our proposed framework yields better quality surface reconstruction. We report the RMS error for the filled patch in Table 1. As evident from the Table 1, the RMS error decreases as we fit higher order surfaces to fill the missing regions.

Table 1. RMS errors for Polynomial Surface Fit algorithm performed on various 3D models.

5 Conclusion

In this paper, we have proposed a novel algorithm to detect holes/large missing regions in point cloud data and consequently fill them without losing the original shape attributes of the model involved. Our proposed framework can successfully detect holes, filter out pseudo-boundaries and consequently help fill them. Experiments carried out on popular 3D models demonstrates the effectiveness of our framework.