Real-time 3D interactive segmentation of echocardiographic data through user-based deformation of B-spline explicit active surfaces

https://doi.org/10.1016/j.compmedimag.2013.10.002Get rights and content

Abstract

Image segmentation is an ubiquitous task in medical image analysis, which is required to estimate morphological or functional properties of given anatomical targets. While automatic processing is highly desirable, image segmentation remains to date a supervised process in daily clinical practice. Indeed, challenging data often requires user interaction to capture the required level of anatomical detail. To optimize the analysis of 3D images, the user should be able to efficiently interact with the result of any segmentation algorithm to correct any possible disagreement. Building on a previously developed real-time 3D segmentation algorithm, we propose in the present work an extension towards an interactive application where user information can be used online to steer the segmentation result. This enables a synergistic collaboration between the operator and the underlying segmentation algorithm, thus contributing to higher segmentation accuracy, while keeping total analysis time competitive. To this end, we formalize the user interaction paradigm using a geometrical approach, where the user input is mapped to a non-cartesian space while this information is used to drive the boundary towards the position provided by the user. Additionally, we propose a shape regularization term which improves the interaction with the segmented surface, thereby making the interactive segmentation process less cumbersome. The resulting algorithm offers competitive performance both in terms of segmentation accuracy, as well as in terms of total analysis time. This contributes to a more efficient use of the existing segmentation tools in daily clinical practice. Furthermore, it compares favorably to state-of-the-art interactive segmentation software based on a 3D livewire-based algorithm.

Introduction

Image segmentation is an ubiquitous task in medical imaging analysis. Indeed, the quantification of morphological or functional properties of a given organ or tissue always involves its identification in the image domain, either by identifying the boundaries of anatomical structures or by defining a region for subsequent analysis. Therefore, the role of image segmentation is central in any medical image analysis workflow. This task has been initially addressed through manual delineation by a user of the desired object in the image. However, with the ever increasing amount of 3D imaging modalities, a simple slice-by-slice 2D manual processing of medical image data becomes a very time consuming, tedious and clinically inefficient solution for routine practice. Furthermore, manual delineation of anatomical structures in 3D data is prone to significant intra and inter-observer variability. It should also be noted that 3D contouring through slice-by-slice 2D processing is not trivial, since boundary continuity in the out-of-plane direction is not explicitly enforced. Therefore, computational tools are crucial to aid the clinicians in this task and there is a vast literature on diverse methods suited to alleviate the need of fully manual analysis. Naturally, fully automatic image segmentation is highly desirable, and several powerful concepts such as active shape models, statistical atlases and other knowledge-based modeling have been proposed to automatically segment organs within 3D volumetric data. Nonetheless, due to inter-subject variability in anatomy and due to changes in imaging conditions, such automatic approaches are very challenging, and often need manual correction. Furthermore, medical image segmentation remains a supervised process in clinical practice and thus the segmentation result should reflect the clinicians’ vision of the anatomical structures. Thus, the user should be able to quickly and efficiently interact with the result of any segmentation algorithm to correct any possible disagreement.

User interaction in medical image segmentation has been reviewed by Olabarriaga and Smeulders [1]. The user input can be introduced at different levels of the processing pipeline, either by initializing and setting the parameters of the segmentation algorithm or by interacting with it, allowing online steering of the segmentation result. The segmentation algorithms that follow the first approach can be broadly classified as semi-automatic, where the segmentation process has a ‘fire and forget’ behavior [2]. On the other hand, the second approach more closely resembles a truly interactive process, where the algorithm reacts to the information introduced by the user preferably in real-time. While the first corresponds to a more automatized approach, with consequently less inter-observer variability, the control of the segmentation outcome by contour initialization and parameter setting might not be sufficient in more challenging areas of the image, where direct user input is required [3]. This can lead to a cumbersome tuning process, which requires user's knowledge of the underlying algorithm. On the other hand, purely interactive segmentation methods may require too much user input, which could affect the intra- and inter-observer variability, and require near real-time processing of the segmentation method for optimal user steering. An efficient image segmentation algorithm is characterized by having a computational part that is fast, accurate, highly autonomous and predictable and whose user interventions are few, quick and simple [1]. Therefore, a hybrid approach, using initial (semi-) automatic segmentation with subsequent interactive real-time correction could provide an interesting trade-off for optimal clinical application.

Several algorithms are able to include user clues in the segmentation process. The most simple approaches are geometrical modeling tools which can be used to aid the manual construction of a 3D model, such as in [4], [5]. More recently, Heckel et al. have proposed a variational interpolation framework to implement near real-time reconstruction of a 3D model from a set of contours [2]. On cardiac segmentation, the ellipsoidal shape of the LV can be used to improve the efficiency of the generation of the 3D model, as in the guide-point modeling algorithm from Young et al. [6]. Obviously, these methods are relying solely on heavy user input, therefore not directly exploiting image information.

The classical example of the interactive segmentation driven by the user paradigm is the livewire algorithm and its variants. In this method, the user sets a number of sequential seed points, while the optimal cost path between them is estimated via graph-based analysis, such as Dijkstra's algorithm [7]. Several extensions have been proposed for 3D segmentation [8], [9], [10] and algorithms for efficient 3D livewire interactive segmentation of complex topology objects also exist [11]. However, the extension of the livewire framework to 3D is not straightforward, as the interaction paradigm is intrinsically 2D, since the optimal path remains a 1D entity. Although the recent work of Grady on minimal weight surfaces provides a truly three-dimensional extension of shortest path-based methods [12], 3D interactive segmentation using livewire-based approaches still rely on a collection of 2D contours from where a 3D model needs to be constructed.

Graph-Cuts, originally proposed by Boykov et al. [13], [14], and its variants offer an alternative paradigm for interactive segmentation. In this framework, the user introduces seed points that will correspond to different regions (object/background) and the segmentation result is calculated as the optimal partition graph between the object and the background, which can be efficiently computed using the max-flow/min-cut algorithm [15]. One important advantage offered by this framework is its computational efficiency, as well as the guarantee that the segmentation result corresponds to the global energy minimum of the segmentation criterion. By interactively placing additional seeds, the user can efficiently interact with the segmentation results. However, these seed points correspond to hard constraints on the segmented regions and not to boundary positions, which can make precise boundary manipulation cumbersome to the user.

Deformable models are a very popular family of segmentation methods, arising from the seminal work of Kass et al. [16]. In the original formulation of snakes, user input points could be used as attractor/repulsor sinks for the parametric curve evolution. ITK-SNAP is a particularly popular software tool that offers several options for 3D active contour segmentation, where the user can control the parameters of the segmentation algorithm and visually follow the evolution of the segmentation process, enabling online re-tuning of the parameters [17]. Therefore, ITK-SNAP is a perfect example for the type-I interaction as defined by Olabarriaga and Smeulders. More recently, Delgado-Gonzalo et al. have proposed a 3D parametric snake where user interaction plays an important role, since the compact parametrization of the surface allows shape deformation through the manipulation of a limited number of control points [18]. Interestingly, in the work of Liang et al. a unifying framework between snakes and livewire is proposed, allowing to take the complimentary advantages of these up until then competing techniques [19]. It should be noted that interactive free-form surface editing operators are also available for level-set based segmentation algorithms [20]. However, 3D segmentation using level-set approaches is computationally intensive, often requiring advanced GPU-based implementations to achieve acceptable running times for near real-time applications [21], [22].

We propose in this paper to take advantage of the existing framework for real-time segmentation of challenging inhomogeneous 3D data, introduced very recently by the authors [23], in order to design a hybrid process smoothly integrating automatic segmentation and real-time interaction. The segmentation method uses B-spline explicit active surfaces (BEAS) to recover objects from volumetric data, allowing to use both global or local region-based segmentation energies to evolve the contour. In spite of obtaining promising results, the initial semi-automated algorithm relied on manual input (6 clicks per 3D volume) to fit an ellipsoid used to initialize the segmentation algorithm [24]. Since manual initialization accounted for the vast majority (∼95%) of the total analysis time, a fast and automatic initialization method was posteriorly developed, allowing fully automatic segmentation [25].

Although the existing BEAS framework offers very competitive performance, the supervised nature of the segmentation process introduces the need of pursuing strategies to efficiently allow the user to correct the segmentation results if needed. The manual correction of the segmentation results provided by BEAS can be efficiently steered by the user by evolving the segmented surface towards the additional points introduced by the operating physician. This is mathematically straightforward thanks to BEAS explicit parametric formulation. Furthermore, the computational speed of the method allows a truly interactive experience, similar to the livewire segmentation paradigm, with the advantage of explicitly deforming the surface rather than a 1D path.

The originality of the proposed method is twofold: first, a new energy term is introduced in the variational formulation of the method, in order to include the contour information introduced by the user. The proposed energy term can be easily extended to the case of multiple points being annotated by the operating physician. Secondly, we introduce a regularization term which yields a geometrically optimal interaction with the surface. This allows to propagate the input information in a geometrically smooth manner and to avoid a too localized deformation of the surface in the neighborhood of the points introduced by the user. Thanks to the computational efficiency of the underlying BEAS segmentation algorithm, the resulting tool offers a synergistic solution between a fully automatic segmentation tool and a real-time interactive segmentation method, where the user can introduce additional points to steer the segmentation process. In this paper we focus on the application of the proposed framework to real-time 3D echocardiographic (RT3DE) data. Note however that this framework is generic and can thus be easily used for a wide class of 3D segmentation tasks.

The paper is structured as follows. In Section 2, we focus on the general formulation of image segmentation problems using BEAS and we introduce the proposed variational formulation user interaction term. Also the geometric regularizer enabling a more efficient surface manipulation is presented here. In Section  3, we discuss some implementation issues of our method. In Section  4, we evaluate the performance of the method using RT3DE data. Furthermore, a benchmark comparison against a state-of-the-art 3D interactive segmentation software is equally presented. The key findings are discussed in depth in Section 5, while the study limitations are highlighted in Section 6. Lastly, we give the main conclusions and perspectives of this work in Section 7.

Section snippets

B-spline explicit active surfaces

The segmentation framework (BEAS) used in the present work has been recently proposed by the authors in order to allow real-time segmentation of challenging inhomogeneous 3D data [23]. The fundamental concept of this method is to regard the boundary of an object as an explicit function in a generic coordinate system, where one of the coordinates of the points within the surface is given explicitly as a function of the remaining coordinates [26]. Such explicit relation can be mathematically

Segmentation parameters

The following settings were applied for all experiments:

  • We used a cubic B-spline function as basis for the BEAS representation. This function provides a good trade-off between smoothing properties and computational cost. The scale parameter was set to h = 2; The size of the neighborhood used to estimate the local means was set to 16.

  • As previously mentioned, the ψ function was defined in spherical coordinates and the LV surface was discretized in 24 × 16 points along the zenithal and azimuthal

Data acquisition and analysis

RT3DE data was acquired using a GE E9 scanner (GE Vingmed, Horten, Norway) equipped with a 4V transducer. The data used in the present study has been acquired at the enrollment of patients at UZ Leuven in a large ongoing multi-center clinical study (DOPPLER-CIP). The patients enrolled in this clinical study have a clinical profile corresponding to suspected chronic ischemic heart disease. The data used in the present work was taken randomly from the DOPPLER-CIP study database where the only

Discussion

Despite the efforts towards fully automatic segmentation in the medical image processing community, at current image segmentation remains a supervised process in clinical routine. It therefore remains crucial to allow user input in any segmentation tool to be clinically used in order to modify incorrect segmentation in an easy manner. This was the main motivation behind the present work in which an existing segmentation framework was modified in order to allow convenient user interaction.

By

Study limitations

The most important limitation of the present work is the fact that only two clinical experts were included in the study. Indeed, significant inter-observer variability in manual delineation of LV contours in RT3DE is widely acknowledged and thus several experts are needed to properly assess the statistical relevance of differences found between methods. Thus the presented results should be carefully interpreted, while keeping in mind that additional validation is still required in a larger

Conclusion

The proposed interactive framework allows to explore a synergistic balance between fully automatic segmentation and user-based interactive segmentation. By only allowing local user-based steering of the automatically segmented surface, the proposed method is able to significantly increase its accuracy while keeping its robustness to the subjectivity introduced by the user. To improve the fluency of the interaction, we introduced a geometrical regularization term allowing to reduce the number of

Acknowledgement

This work was supported by FCT, Portuguese Ministry of Science, Technology and Higher Education, through the grant SFRH/BD/62230/2009, and by the Rhône-Alpes region, through a Explora’Doc and a Accueil’Doc scholarship.

References (42)

  • Q. Duan et al.

    Real-time segmentation by active geometric functions

    Computer Methods and Programs in Biomedicine

    (2010)
  • A. Yezzi et al.

    A fully global approach to image segmentation via coupled curve evolution equations

    Journal of Visual Communication and Image Representation

    (2002)
  • H. Delingette et al.

    Shape and topology constraints on parametric active contours

    Computer Vision and Image Understanding

    (2001)
  • G. Xu et al.

    G2 surface modeling using minimal mean-curvature-variation flow

    Computer-Aided Design

    (2007)
  • E. Angelini et al.

    Segmentation of real-time three-dimensional ultrasound for quantification of ventricular function: a clinical study on right and left ventricles

    Ultrasound in Medicine and Biology

    (2005)
  • K. Rajpoot et al.

    The evaluation of single-view and multi-view fusion 3D echocardiography using image-driven segmentation and tracking

    Medical Image Analysis

    (2011)
  • P. Karasev et al.

    Interactive MRI segmentation with controlled active vision

  • A.A. Young et al.

    Left ventricular mass and volume: fast calculation with guide-point modeling on MR images1

    Radiology

    (2000)
  • A. Falc ao et al.

    An ultra-fast user-steered image segmentation paradigm: live wire on the fly

    IEEE Transactions on Medical Imaging

    (2000)
  • G. Hamarneh et al.

    3D live-wire-based semi-automatic segmentation of medical images

  • K. Lu et al.

    Interactive segmentation based on the live wire for 3D CT chest image analysis

    International Journal of Computer Assisted Radiology and Surgery

    (2007)
  • Cited by (17)

    • Deep Atlas Network for Efficient 3D Left Ventricle Segmentation on Echocardiography

      2020, Medical Image Analysis
      Citation Excerpt :

      Statistical models are based on statistical information, which are mainly contour borders and image texture information, and are modeled using some specific parameters (Zhang et al., 2012). The following two statistical models: active appearance model (Mitchell et al., 2002) and active shape model (Van Assen et al., 2006) have also been widely applied for 3D LV segmentation on echocardiography and have achieved meaningful LV segmentation results (Hansegard et al., 2007; Ma et al., 2010; Barbosa et al., 2013; 2014; Orderud and Rabben, 2008). Graph-cuts were also used for 3D LV segmentation (Juang et al., 2011; Bernier et al., 2017; Mukherjee et al., 2013), which adopted graphs and graph-flow optimization techniques to solve segmentation problems (Bernard et al., 2016).

    • 3D aortic morphology and stiffness in MRI using semi-automated cylindrical active surface provides optimized description of the vascular effects of aging and hypertension

      2018, Computers in Biology and Medicine
      Citation Excerpt :

      In regions with a high centerline curvature (such as the aortic arch), 2 radii might cross, leading to surface self-intersections. To avoid this behavior, a regularization term Er was added, minimizing the surface mean curvature [22] to the data attachment term. A manual estimation of the aortic arch length, commonly used for PWVMRI estimation, was performed using a previously established procedure [2,9,23] implemented in the ARTFUN software (Laboratoire d’Imagerie Biomédicale, Sorbonne Université, INSERM, CNRS).

    • Validation of a Novel Software Tool for Automatic Aortic Annular Sizing in Three-Dimensional Transesophageal Echocardiographic Images

      2018, Journal of the American Society of Echocardiography
      Citation Excerpt :

      Within Speqle3D, after loading the patient's volume sequence and identifying the midsystolic frame of interest, the user can sequentially apply each of the aforementioned steps to ultimately extract the annular area. In addition, the Speqle3D environment allows the user to interact, if needed, with the result of each step (Figure 3), being thus able to (1) correct the valve's axis or the location of the two points placed along it; (2) refine the delineated AV wall surface by manually adding control points in suboptimal regions (the surface is then nearly instantaneously corrected by automatically fusing this user input with the automatic border detection algorithm, similar to Barbosa et al.23; note that such fusion guarantees that a valve-like shape is still extracted, instead of simply creating a deformed version of the surface that passes through the user points); and (3) rectify the location and/or orientation of the identified short-axis planes. In the present study, because of their clinical value, only annular measurements were assessed.

    • Interactive-cut: Real-time feedback segmentation for translational research

      2014, Computerized Medical Imaging and Graphics
      Citation Excerpt :

      As a consequence, the state of the art or rather clinical practice in medical departments is still manual slice-by-slice segmentations which are very time consuming. Thus, interactive segmentation approaches like [8–12] get more and more popular, because they allow the user to support the algorithm with more information, especially in difficult segmentation tasks. However, in this contribution we introduce an interactive graph-based approach with a specific design of the graph which requires only one user-defined seed point inside an object for the segmentation process.

    • DEEPBEAS3D: Deep Learning and B-Spline EXPLICIT Active Surfaces

      2023, IEEE International Ultrasonics Symposium, IUS
    • Interactive Segmentation via Deep Learning and B-Spline Explicit Active Surfaces

      2021, Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
    View all citing articles on Scopus
    View full text