Keywords

1 Introduction and Motivation

In the last decade 3D scanners have been employed in architecture, engineering, biology, cultural heritage as well as diagnostic medicine and reconstruction surgery [1,2,3,4,5,6,7,8,9]. These devices allow doctors to get a detailed virtual model of a human body. The opportunity to acquire body parts shape, including soft tissues like the female human breast, has motivated our conjunct study with the medical specialists in breast reconstruction.

Our main aim is to find a discriminative parametrization of female breast shape i.e., a small set of parameters to meaningfully describe it. This kind of mathematical representation gives the possibility to easily define accurate metric for breast difference evaluation. This result is very attractive for breast surgeon, since it can be used to develop new tools to assess the symmetry after a breast reconstruction. It could also be an effective strategy to create clear and well-defined breast shape categories.

Currently, the surgeons are routinely used to acquire pictures of the patients, or rather a 2D projection of them. The only way to evaluate the surgery is still based on a photographic comparison using pictures taken before and after the surgery. Nevertheless, 3D scanners capture and store more information, like volume estimation, curvature and so on. The use of these data would enable the specialists to plan and asses the surgery in a more accurate way.

The 3D scanner acquisition of human body parts requires a certain time and skills. Long scanning time, tends to increase the patient stress as well as the amount of noise due to the breath and involuntary micro-movements. Modern hand-held scanners, reduces these problems by allowing low acquisition time. Furthermore it guarantees a sufficiently high quality of the data. Actually, extremely high resolution and accuracy are pointless to capture general shape. Moreover, dense points clouds would affect the processing time. For this reasons we propose to perform dataset acquisition with a fast and low-cost hand-held 3D scanner: Structure Sensor [10]. High portability of hand-held scanners simplifies the operator job, that can easily turn around the patient.

The 3D data have to be processed and simplified to capture just the information that surgeons need for their analysis. In the proposed approach we consider normals orientation to build a compact representation of breast model. To further simplify processed 3D data, Principal Component Analysis (PCA) [11] has been employed. PCA is a popular and valuable approach to reduce the high dimensionality of the datasets and capture just the most significant features. Feature reductions through PCA has already been used in the parametrization process of the human body parts [12, 13]. Concerning the breast shapes, other authors proposed to analyse them either using linear measurements, stationary laser scanner, MRI, X-rays or thermoplastic moulding [14,15,16,17,18,19]. Compared to our previous work [9], in this paper we do not employ the planar projections. In [9] the 3D meshes are projected in 2D space and then Thin-Plate Splines [20] is used to estimate the non-linear transformation which change each breast projection in the average one. Our contribution in the field can be summarized in the following points:

  • The acquisition of 3D breast models to build a proper dataset and perform significant experiments. At the best of our knowledge there are not available dataset like this.

  • The idea to exploit 3D normals to create a compact representation of 3D breast models.

  • Time and cost optimization by employing a hand-held 3D scanner.

The remainder of this paper is structured as follows: employed devices and proposed method are described in Sect. 2. Details on the dataset are provided in Subsect. 2.1, while the proposed parametrization method is detailed in Subsect. 2.2. Experimental results are given in Sect. 3. A final discussion, with some consideration for future works, ends the paper.

2 Materials and Methods

The study we conducted is mainly focused on digital shape analysis of breast models to assist breast surgeons for medical and surgical purposes. Our idea is based on three key points: minimally invasive for the patient, use of low cost devices, easy data visualization-&-understanding for people with a medical background.

We employed a 3D scanner with structured infrared light technology that allows us to acquire the information about depth of thousands of points at the same time. The Structure Sensor (Fig. 1) is a hand-held scanner proved to be empirically able to acquire up to 12 m, although it is recommended a distance in the range 0.4 and 3.5 m. Its maximum accuracy is 0.5 mm, but worsens when the volume of the area scanned is large. Since the scanner uses infrared rays, it is recommended for indoor usage only. The device is calibrated, that means each 3D model will show its real size. The sensor itself is not able to acquire RGB colour mode information, however it is possible to plug into an iPad and uses the tablet camera to this purpose.

Fig. 1.
figure 1

A Structure Sensor clipped onto an iPad. We used the same setting in our acquisitions.

Full size image

To acquire a breast model, we propose a clinical procedure in which the female patients hold the hands behind and above the head. In this way the operator can move around the breast with the Structure Sensor (which is clipped onto the iPad). Although texture information have been acquired, this has not been used for the present investigation. An example of the model acquired with Structure Sensor is shown in Fig. 2.

Fig. 2.
figure 2

Example of a textured mesh as it is acquired by the Structure Sensor.

Full size image

Once the model is acquired, it is automatically pre-processed through a 3D processing software (Meshlab [21]), in order to remove noise, isolated vertices and faces. Mesh editing is followed by a manual definition, through cropping, of the Region of Interest (ROI). ROI extraction is a critical part of the proposed procedure. We adopt a simple approach that has been proved to be replicable and reasonable precise. We manually selected the ROI exploiting four anatomical reperees suggested by the breast surgeons (Figs. 2 and 3). In our acquisitions, we scanned both left and right breasts but all of them have been, when needed, vertically mirrored in order to make the dataset right-left side invariant, as shown in Fig. 3.

Fig. 3.
figure 3

Definition of ROI through 4 anatomical reperees suggested by the breast surgeons.

Full size image

Each model is saved with the standard OBJ format, which describes the information on vertices, faces and face normals. The average number of vertices is \(\sim \)1,500, while the average number of faces is \(\sim \)4,000. These models resolution is not extremely high but it is enough to capture information about breast shape, which is the point of this work.

2.1 3D Breast Dataset

After review of the study protocol and formal approval by the internal ethic committee of ASLT (Associazione Santantonese per la Lotta ai Tumori) we gathered a dataset with breasts acquired from different volunteers, aged between 25 and 65, with different shapes and volumes. The breast surgeons put a label on each model, describing size and ptosis of the breast. The severity of ptosis is characterized by evaluating the position of the nipple relative to the infra-mammary fold. Supervised by the doctors, we created a dataset in the following way:

  • Main Dataset: is made up of 31 breasts, 17 left and 14 right. To guarantee a proper dataset variability, we have included breasts of different size and ptosis.

Then, in order to test the strength of the proposed methodology, we selected a patient and acquired her breast several times in pre-operation and post-operation conditions. Hence, two more groups of meshes is distinguishable:

  • Pre-operation - Group 1: is made by 52 meshes, 26 left and 26 right. Notice that this set of meshes has been acquired by two operators, namely a junior and an expert one, so it can be used to investigate how the proficiency of the operator may change the parametrization

  • Post-operation - Group 2: is made by 16 breasts, 8 left and 8 right. These models come from the same patient of Group 1 after a surgery on both breasts.

2.2 Shape Parametrization

In this subsection we present the method employed to process the 3D models in order to parametrize the breast shape with a minimum number of parameters. Our idea is to describe each 3D model as histograms of normals. Since normal vectors define the orientation of each model vertex/face, the proposed algorithm starts with a registration step. Actually, although the acquisition device is calibrated, it doesn’t have a system to get the correct orientation into the real space (e.g., gravity sensors). Hence, the meshes have initially to be oriented along the same direction and its centroid moved on the origin of a 3D Cartesian coordinate system. As second step, the normal space is clustered and the occurrences for each cluster counted. This descriptor is finally reduced by Principal Component Analysis. The summarized pipeline of proposed method is shown in Fig. 4.

Breast Registration. As mentioned before, the acquired data have to be roto-translated since the built descriptor depends on normals orientation. This process is automatically performed as described in [9]. First of all the mesh centroid is moved into the origin of a cartesian coordinate system. Subsequently, the average normal is computed to find the rotation matrix, in order to align it along the Z axis. This means, we use the unit vector (0, 0, 1) as reference. Finally, to get the matrix, a closed form named Rodrigues’s rotation formula [22] is employed. Specifically, given two vectors \(u_1\) and \(u_2\), formula computes the rotation to align \(u_1\) to \(u_2\). In our case \(u_1=averageNormal\) and \(u_2=(0,0,1)\).

Fig. 4.
figure 4

Pipeline of the proposed method. Note that PCA is applied on n breast descriptors. Then, the “learnt” transformation matrix is used as model to extract parameters of all the 3D meshes. Additional details are reported in Sect. 3.

Full size image

Bag of Normals. After each mesh has been correctly oriented in our coordinate system, we can proceed to obtain a representation of the normals distribution over a suitably quantized grid. Firstly, all the normals are normalized. We divided each normal u for ||u||, in order to get a unit vector. By performing this process, the three components of normal vector \((u_x,u_y,u_z)\) fall in the range \([-1, 1]\). We linearly quantize the space of each component into 4 levels, in order to obtain \(4 \times 4 \times 4 = 64\) different cluster. Finally, each mesh is represented by counting the occurrences in each cluster. This histogram with 64 bins is then in turn normalized to get the final bag of normals descriptor.

Principal Component Analysis (PCA). PCA is a popular statistical method that is commonly used for finding patterns in data of high dimension or reducing such dimensionality. This reduction is more interesting when one wants to extract the main characteristics of complex data. PCA is applied on datasets which are described by several attributes. It is able to find a linear transformation which move the data into another space where the transformed attributes are uncorrelated. The aim is to identify the “Principal Components”, or rather a reduced set of attributes which represent the original data [11].

We applied PCA on the 64-d descriptors obtained at the previous step in order to describe each 3D breast with a very small set of parameters, namely 2. This procedure allows us to represent each 3D model as a point in 2D coordinate system where axes are the first two Principal Components. This kind of representation, allows to visually asses the results of a surgery intervention by observing the change of position of a breast in the 2D space. It the next section we report the results that show that 2 components are enough to represent breast shape.

3 Results

We computed PCA on the 31 models in the main dataset. Exploiting only the first 2 principal components we obtained a variance retain of \(48.04+29.35=77.39\%\) (Fig. 5(a)) and the models can be represented in a chart, as shown in Fig. 5(b). The breast surgeons confirmed us the evidence of Fig. 5(b): the first 2 principal components seem enough to distinguish characteristic traits of the labelled models, since models are clearly separated in the obtained result. However, since there are not official metrics to describe breast shape, we currently cannot associate each component to a specific geometrical property.

Fig. 5.
figure 5

PCA computed on the Main Dataset. (a) Variance Retain of the first 5 principal components. The sum of the first 2 principal components is \(77.39\%\). (b) Plot of the 31 models in the Main Dataset using the first 2 principal components.

Full size image
Fig. 6.
figure 6

Plots of the models in Group 1 and Group 2 using the first 2 principal components of PCA computed on the Main Dataset. (a) Visual comparison of the principal components of Group 1 between models acquired by the two groups of operators, properly juniors and experts. (b) Comparison of the principal components between Group 1 (pre-surgery) and 2 (post-surgery). Error in the parametrization has been highlighted through error ellipses added on each set of models. Starting from the ellipsis centroid (the mean value of the set), each concentric error ellipsis contains the \(68\%\) (\(\sigma \)), the \(95\%\) (\(2\sigma \)) and the \(99\%\) (\(3\sigma \)) of the elements, respectively.

Full size image

In order to further assess the soundness of the proposed method we plotted the models of Group 1, exploiting the PCA computed only on the main dataset (Fig. 6(a)). The left breast is clearly distinguishable from the right one, as expected. Once more, using the same principal components, we plotted also the models from Group 2 (Fig. 6(b)). We remark that 3D models in Groups 1 and 2 includes the right and left breast of the same patient, before and after a surgery, respectively. The mean and standard deviation of models in Groups 1 and 2 have been reported in Table 1. Error ellipses including the \(68\%\), \(95\%\) and \(99\%\) of the data are contextually shown in Fig. 6(b). The Euclidean distances between centroids of left and right breast clusters for Group 1 and Group 2 are 0.137 and 0.12, respectively. Although the Euclidean distances are similar, the distance related to the first principal component (the most meaningful, with \(48\%\) of variance retain) is way lower: from 0.136 to 0.014. The distance related to the second principal component (\(29\%\) of variance retain) is increased from 0.008 to 0.119. So, the results shown in Table 1 and Fig. 6(b) are a confirmation that the right breast and left breast after the surgery (meshes from Group 2) have now a first principal components that has pretty similar mean and variance values, while before the surgery (Group 1) they were different.

Table 1. Mean and Standard Deviation of models in Group 1 and 2. L stands for Left, R for Right. Each entry is a pair in which the values are related to the first and second principal component, respectively.
Full size table
Fig. 7.
figure 7

Comparison of the first 2 principal components (X and Y axis, respectively) between different datasets. PCA computed on the main dataset, comparison between the main dataset, Group 1 and Group 2.

Full size image

The comparative chart with the components of all the digitized breasts is shown in Fig. 7. Some significant cases from Main Dataset are shown in Figs. 8(a)–(c), while the patient scanned in Groups 1 and 2 is shown in Figs. 8(d) and (e). The breast surgeons confirmed us that the positions of models from these latter sets are coherent with respect to the one of the models from Main Dataset. These results show that the first principal component is strong enough to characterize the shape of a breast, and through the standard deviation computations on Group 1 and 2 we can also give a cue about the error in this estimation.

Fig. 8.
figure 8

Significant acquired models. (a–c) Models from Main Dataset with principal components \((-0.17;-0.04)\), \((0;-0.02)\) and \((0.11;-0.01)\), respectively. They are in the most left, central e right position of the plot of Fig. 5(b). A clear difference about the shape of the breasts can be noticed. (d–e) Patient of Group 1 (pre-surgery) and Group 2 (post-surgery), respectively; note that we considered right breast the one corresponding to the right arm of the patient.

Full size image

4 Conclusions

In this work we have focused on digital shape analysis of breast models to assist breast specialists for medical and surgical purposes. We fixed three key points for our proposed solution: minimally invasive for the patient approach, use of low cost devices, easy data visualization-&-understanding for people with a medical background. We proposed a clinical procedure in which the female patients hold the hands behind and above the head, while an operator can digitize her breast with a 3D scanner. After a manual ROI definition through cropping, the meshes are automatically processed. The breasts are represented exploiting bag of normals representation, resulting in a 64-d descriptor. A reference dataset has been used to compute PCA on a set of discriminative and different breasts, and the obtained first 2 principal components have been used to plot the breasts into a 2D space. We empirically proved that breasts subjected to a surgery change their representation in this space, and through the variance computations on Group 1 and 2 we also gave a cue about the error in this estimation. We believe that the proposed procedure, assessed by the surgeon, represents a valid solution to evaluate the results of surgeries, since one of the most important goal of the specialists is to symmetrically reconstruct breasts, but an objective tool to measure the result is currently missing. As future works, we planned to augment the ROI extraction phase, which is a critical part of the proposed procedure and requires professionals with a proper know-how of 3D object editing.