Preoperative Planning and Simulation Framework for Twin-to-Twin Transfusion Syndrome Fetal Surgery

Abstract

Twin-to-twin transfusion syndrome (TTTS) is a complication of monochorionic twin pregnancies in which arteriovenous vascular communications in the shared placenta lead to blood transfer between the fetuses. Selective fetoscopic laser photocoagulation of abnormal blood vessel connections has become the most effective treatment. Preoperative planning is thus an essential prerequisite to increase survival rates for severe TTTS. In this work, we present the very first TTTS fetal surgery planning and simulation framework. The placenta is segmented in both magnetic resonance imaging (MRI) and 3D ultrasound (US) via novel 3D convolutional neural networks. Likewise, the umbilical cord is extracted in MRI using 3D convolutional long short-term memory units. The detection of the placenta vascular tree is carried out through a curvature-based corner detector in MRI, and the Modified Spatial Kernelized Fuzzy C-Means with a Markov random field refinement in 3D US. The proposed TTTS planning software integrates all aforementioned algorithms to explore the intrauterine environment by simulating the fetoscope camera, determine the correct entry point, train doctors’ movements ahead of surgery, and consequently, improve the success rate and reduce the operation time. The promising results indicate potential of our TTTS planner and simulator for further assessment on clinical real surgeries.

Equally contributing authors: Jordina Torrents-Barrena implemented the novel research algorithms, and Rocío López-Velazco designed and developed the framework.

1 Introduction

Twin-to-twin transfusion syndrome (TTTS) is a rare and fatal condition that affects around 10–15% of monochorionic twin pregnancies between 16–26 weeks of gestation. This syndrome is caused by the presence of small anastomoses in the placenta vasculature that let the twins exchange an unbalanced blood flow [1]. The most effective treatment of severe TTTS is fetoscopic laser photo-coagulation and it consists in closing blood vessels connecting the twins. Nevertheless, in up to 33% of operated pregnancies, some inter-twin vascular connections remain open causing a recurrence of the TTTS [1].

TTTS surgery is very complex and risky because of several constraints to the fetal clinical setting. Firstly, the placenta position, size and shape greatly varies between pregnancies. Secondly, there is a tiny space to move the fetoscope and the field-of-view is limited. Thirdly, the exact localization of the vessels to coagulate is usually not known ahead of surgery [2]. If the surgeon is not able to reach and coagulate all the anastomoses, reentry in a different point is not allowed (i.e., one-shot procedure), as it is associated with high mortality [1]. Hence, the choice of the entry point is the most critical factor of the intervention as it directly affects the fetoscope maneuverability and the possibility to reach all the anastomoses.

Prenatal evaluation of placental abnormalities is mainly performed by Ultrasound (US) B-scan and Doppler images. US allows fast assessment of the fetus and appears to be safe when following clinical guidelines. TTTS diagnosis can also benefit from fetal magnetic resonance imaging (MRI), which offers outstanding visualization of both the fetal anatomy and its tissue characterization. However, it has limited availability and does not provide complete real-time imaging although dynamic sequences can be acquired [3]. The registration and fusion of MRI with real-time US can therefore be of interest in prenatal diagnosis of targeted anastomoses during the laser ablation therapy.

Related Work. The work proposed by [4] illustrated the feasibility of planning the TTTS preoperative phase with MRI and computerized volume rendering. Authors studied and rendered the anatomy of each amniotic cavity and fetus, the umbilical cord insertions and the location of the inter-twin membrane in relation to the port placement. The optimum port entry point was calculated, as well as the length and angle required to reach the target region. However, authors did not implement a user-specific application to provide real functionality and visualization. Also, they did not segment the placenta vasculature which plays a key role in TTTS fetal surgery. The presented study was not validated quantitatively as the 3D reconstructions were performed almost manually.

Contribution. In this paper we present the first TTTS planning software oriented to clinical use. Novel computer vision and deep learning algorithms are integrated to create a 3D model of the womb including the placenta and its vascular tree, and the umbilical cord of both twins, extracted from MRI and 3D US. The fetoscope entry point is estimated taking into account the cord insertions located on the placenta surface. Our framework is also capable to simulate the movements of the fetoscope and the camera visualization to explore the entire placenta volume. Therefore, our TTTS planning software can aid fetal surgeons to know more about the intrauterine environment (i.e., placental vascular tree, boundary between the vascular hemispheres), determine a suitable entry point, train the fetoscope movements before the intraoperative phase, and consequently, successfully improve the performance rate and reduce the surgery time.

2 Methods

The TTTS surgical outcome depends greatly on choosing the right entry point so that all the anastomoses can be individuated and properly coagulated. To plan the ideal insertion point, we developed an application to load MRI and 3D US images and create a personalized 3D model of the mother uterus. It also provides tools to explore the best entry point and simulate the surgeons’ fetoscope movements, calculating the probability that all the targets can be favorably reached. The following subsections describe our software (see Fig. 1).

Fig. 1.

Modules of our TTTS preoperative planning and simulation framework.

Module 1. Fetal MRI. The acquisition of 3D MRI is challenging in fetal setting because maternal respiratory motion and fetal movements cause motion artifacts between individual slices. To reduce this effect, we acquire several stacks for the whole womb in axial, sagittal and coronal views. Those stacks are resampled to 0.75 $\times$ 0.75 $\times$ 3.5 mm$^3$, normalized, denoised and fused together using a super-resolution reconstruction method [5].

Once the MRI is reconstructed a 3D Convolutional Neural Network (CNN) [6] to automatically segment the placenta is defined as follows:

$$\begin{aligned} f(\left[ \mathbf v , \{s^{(j)}\}_{j \in A}\right] ;\theta ) = f_{out} \circ f_L \circ ... \circ f_2 \circ f_1 \circ f(\left[ \mathbf v , \{s^{(j)}\}_{j \in A}\right] ;\theta _1), \end{aligned}$$

(1)

where $\circ$ denotes the composition operator, $\theta$ represents the CNN parameters, and the output is a binary value indicating whether the voxel belongs to the placenta or not. Each network layer in Eq. 1 contains a set of filters, with each filter being defined by:

$$\begin{aligned} \mathbf x (l + 1) = f_l(\mathbf x (l);\theta _l) = \sigma (\mathbf W _l^\top \mathbf x (l) + \beta _l), \end{aligned}$$

(2)

where $\sigma (.)$ represents a non-linearity, $\mathbf W _l$ and $\beta _l$ denote the weight and bias parameter, and $\mathbf x (1) = \left[ \mathbf v , \{s^{(j)}\}_{j \in A}\right]$. The last layer L of the model in Eq. 1 produces a response $\mathbf x (L + 1)$, which is the input for $f_{out}(.)$ that contains two output nodes, where layers L and out are fully-connected. The training of the model minimizes the binary cross entropy loss.

Small vessels are also quite difficult to recognize in MRI scans, because the size of their lumen is close to the spatial resolution limit of the images. We apply the corner definition (i.e., high intensity variations in all directions due to the amniotic fluid and the placenta texture) to efficiently recognize peripheral vessels, since they appear as dark and non-tubular areas attached to the placenta surface [6]. To detect corners, our method finds the extrema of the Gaussian curvature, which is the product of the minimum and maximum values of the local curvature estimated from the following partial derivatives [7]:

$$\begin{aligned} K = \frac{I_x^2 (I_{yy}I_{zz} - I_{yz}^2) + 2I_xI_{xz}(I_yI_{yz} - I_zI_{yy}) + cycl.(x, y, z)}{I_x^2 + I_y^2 + I_z^2} , \end{aligned}$$

(3)

where $I_u$ and $I_{uv}$ denote the first and second partial derivative w.r.t u and $u, v \in \{x, y, z\}$, respectively, of image I(x, y, z), and cycl. (x, y, z) stands for a cyclic permutation of the coordinates.

To extract the umbilical cord a 3D CNN similar to [8] and Eq. 1 is also implemented. The motivation behind the addition of a Long Short-Term Memory (LSTM) recurrent model is to explore the spatial dependences across adjacent MRI slices and learn image features that capture the global (and intricate) anatomical structure of the umbilical cord. Hence, three main phases define our recurrent network: a down-sampling step, a recurrent component and an up-sampling step. The former deploys four convolutional layers with 5$^3$ volumetric kernels followed by a rectified linear unit (ReLU) and 2$^3$ max-pooling operations. The recurrent mechanism extracts global features that capture the spatial changes of the MRI slices and compensates the max-pooling reduction. The last up-sampled stage is based on four convolutional layers followed by ReLU and several feature map concatenation modules (that combines both the up-sample layer output and the parallel feature extraction). The final segmentation is obtained by a soft-max function.

Module 2. Fetal 3D US. The US volumes are down-sampled to 128 $\times$ 128 $\times$ 64, in which the central part (more information) is preserved.

We also adopt a 3D CNN architecture [9] to segment the placenta. The down-sampling path operates at different resolutions via three convolutional layers with 5$^3$ kernels and PReLu non-linearities. The up-sampling path enlarges the spatial resolution of the feature maps to gather and assemble information. Residual functions are employed to gather fine details and improve the model convergence. The final output is converted to a binary segmentation using a soft-max function. A multiple refinement is subsequently applied to reduce the false positives. An automatic thresholding is computed as the mean of the voxel values in the original US weighted by its gradient image. Afterwards, a morphological closing operation eliminates isolated dark voxels to refine the placental boundary. A 2D slice-by-slice filter is employed to detect and remove unconnected structures associated to smallest non-placenta regions.

Regarding the placental vascular tree, three different classes are considered to fully describe the intrauterine environment. The darkest voxels stand for blood vessels, placenta cavities, amniotic fluid and acoustic shadows. The gray voxels apply to the various mother and fetus tissues. Finally, the brightest voxels correspond to bone and gases. The goal is to identify the structures linked to the darkest voxels to further extract the peripheral blood vessels only. The unsupervised classification is automatically done via the Modified Spatial Kernelized Fuzzy C-Means (MSKFCM) [10]. We initialize three MSKFCM centroids using a Kd-tree based K-means estimator. Each mean represents the statistical distribution of intensity values in the voxels associated to different uterus tissues. To refine the resulting fuzzy segmentation, a maximum a posteriori Markov random field (MRF) [11] is employed. The Mahalanobis distance is iteratively updated to measure the distance between each voxel in the original US to a set of known classes provided by the MSKFCM labeled image. The initial MSKFCM mask is provided as MRF input to guarantee an accurate segmentation of the vessels.

Module 3. Fetal MRI + 3D US. Registration of fetal MRI and 3D US is challenging due to several factors such as the choice of a suitable similarity measure (i.e., the relationship between MRI and US intensities is difficult to express), the artefacts existent in US acquisitions (i.e., acoustic shadows, attenuation and reverberations), among others.

To overcome some of these issues, we directly register the output segmentation mask of the placenta in MRI and 3D US. Once the placenta is aligned, we employ the same transformation to register the placenta vasculature. Because of its large field-of-view we select the MRI image modality as fixed volume and the 3D US as moving volume. More specifically, we employ a Danielsson mapping [12] to compute the signed distance field for the placenta output segmentation mask in MRI. Such filter returns: (1) a signed distance map with the approximation to the euclidean distance, (2) a voronoi partition, and (3) a vector map relating the current voxel with the closest point of the closest object to this voxel. Afterwards, the same mapping is performed on the 3D US mask of the placenta. A BSpline transformation is subsequently computed to register both image modalities (the mask and the original image for placenta). Finally, identical transformations are applied for the placental vasculature co-registration.

Module 4. TTTS Planning. Once the personalized model of the patient is obtained, we provide several functionalities to virtually place the fetoscope into the mother’s womb. The fetal MRI was clinically selected as the most relevant image modality to plan the surgery due to its large anatomical field-of-view. A fetoscope mesh was previously generated via 3DSlicer¹ using the plug-in IGT-CreateModels. The original direction of this mesh is named d$_{original} = (0, 0, -1) = (d_{ox}, d_{oy}, d_{oz})$, and its length meshLength.

A multi-planar widget initializes the axial, sagittal and coronal views and the 3D model. The clinician places two points in an axial slice: the entry ($p_{entry}$) and target ($p_{target}$) points. Both points create a new directional vector named d$_{planning}$ (see Eq. 4). Following 3D classical linear algebra, we apply a rotation transform to place the mesh following d$_{planning}$.

$$\begin{aligned} {\varvec{d}}_{planning} = p_{entry} - p_{target} = (d_{px}, d_{py}, d_{pz}) \end{aligned}$$

(4)

At this point, we need to solve a system of equations where the rotation axis and the angle $\alpha$ are both unknown variables. The rotation axis is calculated by the cross product of d$_{original}$ and d$_{planning}$ (see Eq. 5):

$$\begin{aligned} {\varvec{d}}_{original} \times {\varvec{d}}_{planning} \, = \, \mid {\varvec{d}}_{original}\mid \cdot \mid {\varvec{d}}_{planning}\mid \cdot \sin \alpha = \left[ {\begin{array}{ccc} \text {x}\quad \, \text {y}\quad \, \text {z} \\ {d_{ox}\, d_{oy}\, d_{oz}} \\ {d_{px} \, d_{py}\, d_{pz}} \end{array}} \right] \end{aligned}$$

(5)

where the rotation angle can be isolated as $\alpha = \frac{\arccos \left[ \frac{(d_{px} \cdot d_{ox} + d_{py} \cdot d_{oy} + d_{pz} \cdot d_{oz})}{\root \of {(d_{px}^2 + d_{py}^2 + d_{pz}^2)}}\right] }{\frac{\pi }{180}}$.

Afterwards, the mesh is translated to $p_{target} - {\varvec{d}}_{planning} \times meshLength$, so that the tip of the fetoscope reaches the destination point. At this point, it is essential to know the anatomical constraints (i.e., range of movement of the fetoscope) derived from the insertion point of the fetoscope into the uterus. The rotation point is finally computed automatically from the intersection between the vector d$_{planning}$ and the uterus mesh through an oriented bounding box (OBB) tree [13]. The rotation point will be used in the simulation module.

Furthermore, the interface of this module gives the clinician the opportunity to eliminate and edit both the $p_{entry}$ and $p_{target}$ points.

Module 5. TTTS Simulation. The fetoscope insertion is simulated using two different 3D render windows (see Fig. 4). One allows the clinician to explore the registered 3D model being the main reference space. The other offers an intrauterine visualization provided by the virtual camera located at the tip of the fetoscope mesh. The simulation reproduces a real clinical intervention.

The aforementioned virtual environment is implemented in VTK². A new vtkRenderWindow interactor is deployed in the second render window to enable the mouse navigation. By clicking the left button of the mouse, the interactor captures its position on the display, and returns a rotation transform that will move the fetoscope mesh to the desired direction.

More specifically, the proposed method takes into account three references: the center of the VTK render window, the clicked point and the rotation point (see Fig. 2). These points located on the display are converted to world coordinates into the 3D scene. These world coordinates allow us to compute iteratively the rotation needed to move the fetoscope in the scene. We take advantage of the previous Eqs. 4 and 5 to calculate this transform, where d$_{original}$ = d$_{center}$ and d$_{planning} = {\varvec{d}}_{mouse\_clicked}$.

Fig. 2.

Simulation scheme to show the computation of the fetoscope rotation.

In addition, the simulation interface offers the following functionalities: (1) show the range of movement of the fetoscope using a virtual cone, (2) highlight the area of the placenta that can be reached from the planned insertion point, (3) record the trajectory of the fetoscope, (4) show or hide the umbilical cord insertions on the placenta surface, and (5) save the TTTS surgical planning.

3 Experimental Results

Database. An in-house database provided by the Hospital Clínic de Barcelona was used according with its Ethical Research Committee and the current legislation. MRI and 3D US scanning of 12 monochorionic twin pregnancies between 25–37 gestational weeks were collected using a Siemens MAGNETOM Aera 1.5T (Fat Saturated (FS): 1.5, Echo Time (TE): 98 ms and Repetition Time (TR): 1200 ms). Each fetus had several MRI volumetric data in different views (axial, sagittal and coronal) with slice dimension 256 $\times$ 208, slice thickness 3.5 mm, and voxel spacing 1.4$^{2}$ mm$^{2}$. The 3D US scanning were collected using a GE Voluson E10 (GE Healthcare, Milwaukee, WI, USA) with a curved electronic matrix 4D probe transducer.

Programming Environment. The software uses proven open source technology such as VTK, ITK³, MITK⁴, and Qt⁵. Supplementary programming languages (i.e., C++, Python and Bash Shell scripting) and libraries (i.e., Tensorflow⁶) are also used to implement the aforementioned segmentation and registration algorithms. The experimentation was executed on an Intel Core i7 2.60 GHz, 16 GB of RAM with Fedora 24 and a NVIDIA GeForce GTX Titan X.

Experiments. Module 1 and 2 are quantitatively tested using the following metrics (see Table 1): Dice and Jaccard coefficients, Area under ROC (AUC), Sensitivity, and Specificity. A heterogeneous set of images from different placenta positions and twin pregnancies is used to cover a real clinical environment.

Results are accurate (see Table 1) although some regions inside the uterus (i.e., fetal tissues) possess a texture similar to that of the placenta and for this reason some non-placenta pixels are misclassified. There is room for further improvement in the case of blood vessels extraction. The color Doppler modality has been a great support to validate 3D US placental vasculature segmentation. So far, the MRI/3D US registration assessment has been done clinically by taking advantage of the surgeons’ anatomical knowledge. More work is required to accurately fuse the registered vessels.

Table 1. Performance measures (i.e., Dice, Jaccard, AUC, Sensitivity and Specificity) for placenta, vessels and umbilical cord segmentation methods from MRI and 3D US.

Fig. 3.

User interface of the proposed TTTS planning and simulation software (from left to right and up to down): (1) Module 1: Fetal MRI, (2) Module 2: Fetal 3D US, (3) Module 3: Fetal MRI + 3D US, and (4) Module 4: TTTS planning.

Fig. 4.

User interface of Module 5 (from left to right): (1) reference and fetoscope camera scenes with both umbilical cord insertions (purple), and (2) reference and fetoscope camera scenes with cone illumination and fetoscope trajectory (black). (Color figure online)

Figures 3 and 4 show the interface and tools of the proposed TTTS surgical planning and simulation framework. We performed a usability trial with a post-test questionnaire to gauge what doctors think about our application. Two different expert surgeons successfully tested the software and reported clinical feedback. Both agreed on the clear visualization of the 3D anatomical models, as well as the comfortable user-experience provided by the current functionalities. Nevertheless, they agreed on the importance of simulating the laser burn to know exactly the already coagulated vessels. A fixed orientation of the uterus in all tabs with respect to the patients’ position was also required to avoid a reference-free navigation.

4 Discussion and Conclusion

We present the first TTTS surgery planning and simulation software. State-of-the-art computer vision and deep learning algorithms are integrated together to generate a full 3D model of the womb from MRI and 3D US. The movements of the fetoscope and the camera visualization used to examine the mother uterus are also simulated through an intuitive and easy-to-use user interface. Hence, the main objective of the presented application is to enable doctors ahead of TTTS surgery by knowing the correct entry point of the fetoscope, the movements and the estimation of an approximate trajectory that will traverse the vascular hemisphere. In short, this software aims to increase the success rate and survival of both twins.

The proposed framework is validated on a set of 12 (monochorionic) twin pregnancies between 25–37 weeks of gestation. Dice coefficients of 0.75 ± 0.11, 0.85 ± 0.06 and 0.79 ± 0.03 are achieved for placenta, vessels and umbilical cord segmentation from MRI, respectively. Similarly, Dice coefficients of 0.88 ± 0.07 and 0.79 ± 0.05 are obtained for placenta and its vasculature from 3D US, respectively. Nevertheless, there is room for further improvement. Although MRI/3D US placenta registration is assessed under anatomy experience of doctors, the fidelity of the resulting 3D model needs to be improved. In near future, we will investigate how the current simulation module can mimic the laser ablation of placental vessels as well as the fetoscope motion constraints in order to be as realistic as possible. In addition, several 3D reference markers regarding the patients’ position and orientation are required to improve the user experience. We are also planning to move soon in the clinical evaluation phase.