Regularized multi-structural shape modeling of the knee complex based on deep functional maps

Highlights

• Enhancement of shape matching and model generalization ability through the incorporation of Deep Functional Maps

• Prevention of overfitting to invalid shapes and improvement of shape inference using a regularization term on the shape likelihood

• Prediction of missing structures by shape correlation analysis and multi-structure statistical shape models

Abstract

The incorporation of a-priori knowledge on the shape of anatomical structures and their variation through Statistical Shape Models (SSMs) has shown to be very effective in guiding highly uncertain image segmentation problems. In this paper, we construct multiple-structure SSMs of purely geometric nature, that describe the relationship between adjacent anatomical components through Canonical Correlation Analysis. Shape inference is then conducted based on a regularization term on the shape likelihood providing more reliable structure representations. A fundamental prerequisite for performing statistical shape analysis on a set of objects is the identification of corresponding points on their associated surfaces. We address the correspondence problem using the recently proposed Functional Maps framework, which is a generalization of point-to-point correspondence to manifolds. Additionally, we show that, by incorporating techniques from the deep learning theory into this framework, we can further enhance the ability of SSMs to better capture the shape variation in a given dataset. The efficiency of our approach is illustrated through the creation of 3D models of the human knee complex in two application scenarios: incomplete or noisy shape reconstruction and missing structure estimation.

Introduction

The reconstruction of geometric shapes plays an important role in many fields such as computer vision, augmented and virtual reality, personalized computer-aided intervention, robotic mapping, and other. If 3D scans are available, as in the case of medical imaging, automated image segmentation is usually performed to localize and extract the object of interest. However, in real case scenarios with noise and intrinsic artifacts, such as pathologies, most of the automated algorithms produce invalid geometric representations if they rely solely on the image content. Image segmentation methods are also especially sensitive to missing or incomplete data producing unrealistic shapes.

One way to reduce uncertainty in estimation is through the incorporation of prior information on the shape variability of anatomical structures, known as statistical shape analysis (Dryden and Mardia, 1998, Goodall, 1991). The use of Statistical Shape Models (SSMs) in image analysis has been well established more than a decade ago, with the most popular variants being the Active Shape Models and Active Appearance Models that, in addition to the expected shape, represent also the texture (complete appearance) of the volumetric object (Heimann and Meinzer, 2009). Despite their success, the main disadvantage of these methods is the excessive memory usage in case of high texture resolution that often requires to be scaled down radically. On the other hand, purely geometrical shape models, often referred to as Point Distribution Models (PDMs) (Cootes et al., 1995) are more intuitive, easy to implement, reasonably robust, and fast.

Lately, there has been a growing interest in the construction of models that consist of multiple anatomical structures (Cerrolaza et al., 2015, Cerrolaza et al., 2016, Cerrolaza et al., 2019, Saito et al., 2013). The key concept in such methods is to overcome the limitations of the small availability of training data, by considering the inter-relation between neighboring structures, i.e., they describe how the shape variation of one structure affects the shape of the other, and vice versa. This can potentially lead to more efficient and accurate shape representation, while also enabling us to conduct shape inference about adjacent structures.

In this work we employ such methods for building statistical shape models of purely geometric nature and demonstrate their application in volumetric data, such as medical images. In particular, we are interested in constructing SSMs that capture the shape variation of multiple anatomical components, while also encoding the correlation between neighboring structures. Moreover, by using more sophisticated methods like the manifold representation and deep learning techniques, we are able to better capture the variation of an object's class.

The fundamental requirement in constructing an accurate shape model, is to first establish correspondence among the elements of the training shapes. The correspondence problem, or shape matching, is the most crucial task in order to capture true geometric variation, as false identification of point pairs may lead to unnatural shapes.

Solving the problem of surface matching is an ambitious goal and is still not adequately addressed, although, there are many techniques that tackle effectively this problem (Biasotti et al., 2016, Van Kaick et al., 2011, Sahillioğlu, 2020). For the construction of SSMs, the most popular shape matching method is the Iterative Closest Point (ICP) (Rusinkiewicz and Levoy, 2001) which is based on repetitive rigid transformations and assignment of closest points, until convergence to a local minimum. However, proximity-based methods, are often insufficient to correctly identify corresponding points. Therefore, the focus of shape matching methods is on computing correct correspondences from an invariant and robust point of view.

One promising framework was proposed by Ovsjanikov et al. called Functional Maps (FM) (Ovsjanikov et al., 2012), which is a generalization of the notion of point-to-point shape correspondence. The framework describes how mappings act on real-valued functions defined on manifolds and allows the computation of point correspondences between two shapes in a non-rigid fashion and independently of their spatial orientation, assuming that the shapes undergo (near-) isometric deformations. This method has achieved state-of-the-art performance in shape correspondence benchmarks for both full and partial shapes (Rodolà et al., 2017; Kovnatsky et al., 2015, Kovnatsky et al., 2013; Rodolà et al., 2017; Litany et al., 2016, 2017; Ren et al., 2018), but it has rarely been employed for estimation and refinement of anatomical structures in challenging situations including image noise, artifacts, and partial information.

Recently, there has been a growing interest in techniques that attempt to generalize deep neural networks to non-Euclidean domains like manifolds, which are collectively referred to as geometric deep learning (Bronstein et al., 2017). For shape matching, it has been shown that the extracted information from shape data can be used in order to compute more accurate point-to-point correspondences (Monti et al., 2017, Masci et al., 2015, Boscaini et al., 2016, Litany et al., 2017a; Roufosse et al., 2019).

The aim of this work is to investigate whether image-guided shape reconstruction methods can be further improved if SSMs are incorporated for refinement of the reconstructed shape, where we propose a regularized multi-structure statistical shape modeling approach for estimation of partial or degenerate 3D data. Specifically, we address the issue of topological alterations (noise, artifacts, missing parts) in the segmentation outcome by formulating an optimization function that balances between the original subject-specific shape and a prior likelihood term. The advantage of this approach is two-fold:

• We address cases where the full multi-shape model is fitted to new shapes (obtained from automatic image segmentation methods) that do not represent anatomically correct structures due to inaccuracies in image segmentation, such as in regions with low intensity contrast, inhomogeneity or imaging artifacts, resulting in missing parts or protrusions in the segmented image. A regularization term on the solution space can potentially prevent overfitting of the SSMs to such defective shapes.

• We address cases where whole structures are missing (not detected) due to the low sensitivity of some imaging modalities to depict certain tissues. For example, the relationship of a tumor to adjacent normal structures, including joints and neurovascular structures, is better assessed with MRI, whereas CT is superior in visualizing calcific deposits and pathologic fractures (Zimmer et al., 1985). Multi-structure SSMs, once constructed, have the potential to infer unknown or missing structures (due to availability of a single modality) through shape correlation analysis, by exploiting the observed shape conformations of one structure to approximate the shape of another highly correlated neighboring structure.

We examine the application of our approach in data of the human knee complex. While there have been various approaches on the construction of statistical shape models of the knee joint (Rao et al., 2013, Fitzpatrick et al., 2011, Baldwin et al., 2010, Williams et al., 2010, Bredbenner et al., 2010), to our best knowledge, it has never been carried out in such way.

To summarize, this work makes the following contributions to the field of 3D shape modeling:

• We employ the recently proposed framework of Functional Maps for solving the isometric shape matching problem, and we also incorporate deep learning techniques to enhance the ability of SSMs to capture shape variation.

• We construct multiple-structure shape models that encode the relationship between neighboring structures, and we jointly optimize reconstruction performance and shape likelihood to improve the quality of automated segmentation of the knee complex.

• Finally, we exploit the knowledge captured by the model regarding the inter-dependence of related structures through Canonical Correlation Analysis, to conduct inference about unknown or missing structures.