Elsevier

Medical Image Analysis

Volume 15, Issue 4, August 2011, Pages 477-488
Medical Image Analysis

Evaluation framework for carotid bifurcation lumen segmentation and stenosis grading

https://doi.org/10.1016/j.media.2011.02.004Get rights and content

Abstract

This paper describes an evaluation framework that allows a standardized and objective quantitative comparison of carotid artery lumen segmentation and stenosis grading algorithms. We describe the data repository comprising 56 multi-center, multi-vendor CTA datasets, their acquisition, the creation of the reference standard and the evaluation measures. This framework has been introduced at the MICCAI 2009 workshop 3D Segmentation in the Clinic: A Grand Challenge III, and we compare the results of eight teams that participated. These results show that automated segmentation of the vessel lumen is possible with a precision that is comparable to manual annotation. The framework is open for new submissions through the website http://cls2009.bigr.nl.

Graphical abstract

This paper describes an evaluation framework that allows a standardized and objective quantitative comparison of carotid artery lumen segmentation and stenosis grading algorithms. We describe the data repository comprising 56 multi-center, multi-vendor CTA datasets, their acquisition, the creation of the reference standard and the evaluation measures. This framework has been introduced at the MICCAI 2009 workshop 3D Segmentation in the Clinic: A Grand Challenge III, and we compare the results of eight teams that participated. These results show that automated segmentation of the vessel lumen is possible with a precision that is comparable to manual annotation. The framework is open for new submissions through the website http://cls2009.bigr.nl.

  1. Download : Download high-res image (55KB)
  2. Download : Download full-size image

Research highlights

► Evaluation framework for carotid artery lumen segmentation and stenosis grading. ► Description of the datasets and creation of reference standard is given. ► Standardized evaluation measures are defined. ► Results on current eight submissions are presented. ► Framework is open for new submissions.

Introduction

Cardiovascular diseases account for 30% of all deaths worldwide (Abegunde et al., 2005). Atherosclerosis, a disease of the vessel wall, is the major cause of cardiovascular diseases such as stroke (Frostegård, 2005). Atherosclerosis may lead to stenosis (luminal narrowing), but it is also possible for atherosclerotic plaque to build up without narrowing the lumen (Glagov et al., 1987). Cardiovascular imaging is an important means to monitor and quantify the state of the vessel wall and lumen.

Manual lumen segmentation and stenosis quantification is laborious and suffers from inter and intra rater variabilities (Scherl et al., 2007). Consequently much work has been performed on (semi)-automated cardiovascular image processing, of which the majority focuses on lumen quantification and assessment of the severity of luminal stenosis in various imaging modalities.

In 2004, Kirbas et al. (Kirbas and Quek, 2004) published a review paper on vessel lumen segmentation algorithms. More recently, an extensive review paper on the same subject was published by Lesage et al. (Lesage et al., 2009). Both papers provide an overview of the currently published vessel segmentation methods, categorized according to the technology (Kirbas and Quek, 2004) or the combination of method characteristics such as the used vessel models, image features and extraction schemes (Lesage et al., 2009). However, as explicitly indicated by Lesage et al., direct performance comparisons of different approaches are lacking because of the wide range of applications, and more generally, because of the lack of standard image databases and validation criteria for most vascular segmentation applications. Moreover, many of the reviewed algorithms are not publicly available which hampers an objective and fair comparison by third parties.

The goal of the framework described in this paper (CLS2009 framework) is to provide a standardized evaluation framework for carotid artery lumen segmentation and stenosis grading in Computed Tomography Angiography (CTA). We focus on the carotid bifurcation (Fig. 1), where the Common Carotid Artery (CCA) splits into the External Carotid Artery (ECA) and Internal Carotid Artery (ICA). The latter is one of the major blood supplying arteries to the brain. In 25% of all stroke patients, the stroke is caused by atherosclerotic disease in the carotid artery bifurcation. As shown by the North American Symptomatic Carotid Endarterectomy Trial (NASCET Collaborators, 1991) and the European Carotid Surgery Trial (ECST Collaborators, 1998) the stenosis grade is an important clinical measure in deciding whether or not to perform carotid endarterectomy (surgical procedure to remove the atherosclerotic plaque from the vessel). Carotid endarterectomy is indicated in case of a stenosis of 50–99% (Rothwell et al., 2003). So the accuracy of stenosis analysis influences clinical decision making. Traditionally stenoses around the carotid bifurcation have been assessed with intra-arterial catheter angiography, i.e. Digital Subtraction Angiography (DSA), which is still considered the gold standard. DSA has a 0.3–1% risk of neurological deficits such as stroke (Waugh et al., 1992, Hankey et al., 1990) which makes the use of less invasive diagnostic imaging preferable. CTA is a good, less invasive alternative to DSA for the assessment of stenosis in the carotid bifurcation (Binaghi et al., 2001, Koelemay et al., 2004).

Besides the assessment of the stenosis caused by atherosclerotic plaque, imaging also allows for the evaluation of the atherosclerotic plaque itself. Much research is currently focused on the question whether specific plaque features like composition or morphology are associated with an increased risk for clinical events. However, in current practice the degree of stenosis is still the only biomarker which influences clinical decision making.

Besides CTA, MRI and US are able to visualize the plaque. Currently it is not clear which technique is most suitable in providing clinical useful information on both stenosis and the atherosclerotic plaque.

Our choice for lumen segmentation of the carotid bifurcation is motivated by the clinical relevance of this structure. Additionally, the carotid is a medium sized vessel which makes it suitable as a test case for a wide range of vessel segmentation algorithms, while still having the typical challenges in segmenting diseased vessels, such as calcifications and severe stenosis. Also the presence of a bifurcation and nearby bone and veins imposes additional challenges on the segmentation. Moreover, lumen segmentation is a first step in the assessment of stenosis, which is the clinical relevant parameter that is assessed in the CLS2009 framework.

The remainder of this paper is organized as follows: in Section 2 previous work is discussed, followed in Section 3 by a description of the CLS2009 framework for the segmentation of the carotid artery lumen and its stenosis grading. Section 4 gives a description of the first use of the framework during the MICCAI workshop, including a short description of the methods that were tested and the results of these methods as produced by the framework. This is followed by some concluding remarks in Section 5. This paper is an extended version of the editorial of the carotid challenge workshop proceedings (Hameeteman et al., 2009a).

Section snippets

Previous work

There is a growing number of initiatives that set up a publicly available data repository and standardized evaluation framework. This demonstrates an increasing interest in standardized evaluation and the possibility to compare methods to each other. In this section we give a few examples and we briefly discuss some published lumen segmentations and stenosis grading methods.

In the field of computer vision the following frameworks can be used: the Range Image Segmentation Comparison (Hoover et

Evaluation framework

The CLS2009 framework comprises a publicly accessible data repository, a set of standardized evaluation measures and an online evaluation system. This section starts with a description of the segmentation tasks, followed by a description of the datasets and their acquisition protocols. Next we describe the procedure that was used to generate manual segmentations and stenosis gradings, followed by a description of the used procedure to merge the different manual segmentations into a reference

MICCAI workshop

The evaluation framework was used in a competition setup at the MICCAI 2009 workshop 3D Segmentation in the Clinic: A Grand Challenge III. Around 100 groups from academia and industry were invited by e-mail to participate in the workshop. Thirty-one teams registered at the website, 21 of which sent in the data confidentiality form, which was required to download the data, 13 teams downloaded the data and 9 teams submitted results. For the lumen segmentation, there was one submission from a

General discussion and conclusion

We presented an evaluation framework for carotid lumen segmentation and stenosis grading and made this framework publicly available via the website http://cls2009.bigr.nl.

The evaluation framework focuses on the effectiveness of lumen segmentation and stenosis gradings. Another important aspect for getting a novel technology accepted in daily clinical routine is efficiency. We decided not to take the computational costs into account in our framework. The main reasons are the lack of a standard

Acknowledgments

Reinhard Hameeteman, Wiro Niessen and Theo van Walsum are supported by the Stichting voor de Technische Wetenschappen (STW) of The Netherlands Organization for Scientific Research (NWO). Aad van der Lugt is supported by Netherlands Organization for Health Research and Development (NWO-KF Grant 907-00-122) and The Dutch Heart Foundation (Grant 2007B161).

References (47)

  • Berg, M., Zhang, Z., Ikonen, A., Manninen, H., Vanninen, R., 2005. Carotid stenosis assessment with ct angiography...
  • S. Binaghi et al.

    Three-dimensional computed tomography angiography and magnetic resonance angiography of carotid bifurcation stenosis

    Eur. Neurol.

    (2001)
  • Catmull, E., Rom, R., 1974. A class of local interpolating splines. In: Barnhill, R.E., Riesenfeld, R.F. (Eds.),...
  • T. Chan et al.

    Active contours without edges

    IEEE Trans. Image Process.

    (2001)
  • Cuisenaire, O., 2009. Fully automated segmentation of carotid and vertebral arteries from CTA. In: The Midas Journal....
  • O. Cuisenaire et al.

    Fully automated segmentation of carotid and vertebral arteries from contrast enhanced cta

  • L.R. Dice

    Measures of the amount of ecologic association between species

    Ecology

    (1945)
  • V. Doré et al.

    Robust NL-means filter with optimal pixel-wise smoothing parameter for statistical image denoising

    IEEE Trans. Signal Process.

    (2009)
  • ECST Collaborators, 1998. Randomised trial of endarterectomy for recently symptomatic carotid stenosis: final results...
  • L. Flórez Valencia et al.

    Carotid arteries segmentation in CT images with use of a right generalized cylinder model

    Midas J.

    (2009)
  • Flórez Valencia, L., Azencot, J., Vincent, F., Orkisz, M., Magnin, I., 2006. Segmentation and quantification of blood...
  • Freiman, M., Frank, J., Weizman, L., Nammer, E., Shilon, O., Joskowicz, L., Sosna, J., 2009. Nearly automatic vessels...
  • J. Frostegård

    Sle, atherosclerosis and cardiovascular disease

    J. Intern. Med.

    (2005)
  • Cited by (0)

    View full text