Evaluation framework for carotid bifurcation lumen segmentation and stenosis grading
Graphical abstract
This paper describes an evaluation framework that allows a standardized and objective quantitative comparison of carotid artery lumen segmentation and stenosis grading algorithms. We describe the data repository comprising 56 multi-center, multi-vendor CTA datasets, their acquisition, the creation of the reference standard and the evaluation measures. This framework has been introduced at the MICCAI 2009 workshop 3D Segmentation in the Clinic: A Grand Challenge III, and we compare the results of eight teams that participated. These results show that automated segmentation of the vessel lumen is possible with a precision that is comparable to manual annotation. The framework is open for new submissions through the website http://cls2009.bigr.nl.
Research highlights
► Evaluation framework for carotid artery lumen segmentation and stenosis grading. ► Description of the datasets and creation of reference standard is given. ► Standardized evaluation measures are defined. ► Results on current eight submissions are presented. ► Framework is open for new submissions.
Introduction
Cardiovascular diseases account for 30% of all deaths worldwide (Abegunde et al., 2005). Atherosclerosis, a disease of the vessel wall, is the major cause of cardiovascular diseases such as stroke (Frostegård, 2005). Atherosclerosis may lead to stenosis (luminal narrowing), but it is also possible for atherosclerotic plaque to build up without narrowing the lumen (Glagov et al., 1987). Cardiovascular imaging is an important means to monitor and quantify the state of the vessel wall and lumen.
Manual lumen segmentation and stenosis quantification is laborious and suffers from inter and intra rater variabilities (Scherl et al., 2007). Consequently much work has been performed on (semi)-automated cardiovascular image processing, of which the majority focuses on lumen quantification and assessment of the severity of luminal stenosis in various imaging modalities.
In 2004, Kirbas et al. (Kirbas and Quek, 2004) published a review paper on vessel lumen segmentation algorithms. More recently, an extensive review paper on the same subject was published by Lesage et al. (Lesage et al., 2009). Both papers provide an overview of the currently published vessel segmentation methods, categorized according to the technology (Kirbas and Quek, 2004) or the combination of method characteristics such as the used vessel models, image features and extraction schemes (Lesage et al., 2009). However, as explicitly indicated by Lesage et al., direct performance comparisons of different approaches are lacking because of the wide range of applications, and more generally, because of the lack of standard image databases and validation criteria for most vascular segmentation applications. Moreover, many of the reviewed algorithms are not publicly available which hampers an objective and fair comparison by third parties.
The goal of the framework described in this paper (CLS2009 framework) is to provide a standardized evaluation framework for carotid artery lumen segmentation and stenosis grading in Computed Tomography Angiography (CTA). We focus on the carotid bifurcation (Fig. 1), where the Common Carotid Artery (CCA) splits into the External Carotid Artery (ECA) and Internal Carotid Artery (ICA). The latter is one of the major blood supplying arteries to the brain. In 25% of all stroke patients, the stroke is caused by atherosclerotic disease in the carotid artery bifurcation. As shown by the North American Symptomatic Carotid Endarterectomy Trial (NASCET Collaborators, 1991) and the European Carotid Surgery Trial (ECST Collaborators, 1998) the stenosis grade is an important clinical measure in deciding whether or not to perform carotid endarterectomy (surgical procedure to remove the atherosclerotic plaque from the vessel). Carotid endarterectomy is indicated in case of a stenosis of 50–99% (Rothwell et al., 2003). So the accuracy of stenosis analysis influences clinical decision making. Traditionally stenoses around the carotid bifurcation have been assessed with intra-arterial catheter angiography, i.e. Digital Subtraction Angiography (DSA), which is still considered the gold standard. DSA has a 0.3–1% risk of neurological deficits such as stroke (Waugh et al., 1992, Hankey et al., 1990) which makes the use of less invasive diagnostic imaging preferable. CTA is a good, less invasive alternative to DSA for the assessment of stenosis in the carotid bifurcation (Binaghi et al., 2001, Koelemay et al., 2004).
Besides the assessment of the stenosis caused by atherosclerotic plaque, imaging also allows for the evaluation of the atherosclerotic plaque itself. Much research is currently focused on the question whether specific plaque features like composition or morphology are associated with an increased risk for clinical events. However, in current practice the degree of stenosis is still the only biomarker which influences clinical decision making.
Besides CTA, MRI and US are able to visualize the plaque. Currently it is not clear which technique is most suitable in providing clinical useful information on both stenosis and the atherosclerotic plaque.
Our choice for lumen segmentation of the carotid bifurcation is motivated by the clinical relevance of this structure. Additionally, the carotid is a medium sized vessel which makes it suitable as a test case for a wide range of vessel segmentation algorithms, while still having the typical challenges in segmenting diseased vessels, such as calcifications and severe stenosis. Also the presence of a bifurcation and nearby bone and veins imposes additional challenges on the segmentation. Moreover, lumen segmentation is a first step in the assessment of stenosis, which is the clinical relevant parameter that is assessed in the CLS2009 framework.
The remainder of this paper is organized as follows: in Section 2 previous work is discussed, followed in Section 3 by a description of the CLS2009 framework for the segmentation of the carotid artery lumen and its stenosis grading. Section 4 gives a description of the first use of the framework during the MICCAI workshop, including a short description of the methods that were tested and the results of these methods as produced by the framework. This is followed by some concluding remarks in Section 5. This paper is an extended version of the editorial of the carotid challenge workshop proceedings (Hameeteman et al., 2009a).
Section snippets
Previous work
There is a growing number of initiatives that set up a publicly available data repository and standardized evaluation framework. This demonstrates an increasing interest in standardized evaluation and the possibility to compare methods to each other. In this section we give a few examples and we briefly discuss some published lumen segmentations and stenosis grading methods.
In the field of computer vision the following frameworks can be used: the Range Image Segmentation Comparison (Hoover et
Evaluation framework
The CLS2009 framework comprises a publicly accessible data repository, a set of standardized evaluation measures and an online evaluation system. This section starts with a description of the segmentation tasks, followed by a description of the datasets and their acquisition protocols. Next we describe the procedure that was used to generate manual segmentations and stenosis gradings, followed by a description of the used procedure to merge the different manual segmentations into a reference
MICCAI workshop
The evaluation framework was used in a competition setup at the MICCAI 2009 workshop 3D Segmentation in the Clinic: A Grand Challenge III. Around 100 groups from academia and industry were invited by e-mail to participate in the workshop. Thirty-one teams registered at the website, 21 of which sent in the data confidentiality form, which was required to download the data, 13 teams downloaded the data and 9 teams submitted results. For the lumen segmentation, there was one submission from a
General discussion and conclusion
We presented an evaluation framework for carotid lumen segmentation and stenosis grading and made this framework publicly available via the website http://cls2009.bigr.nl.
The evaluation framework focuses on the effectiveness of lumen segmentation and stenosis gradings. Another important aspect for getting a novel technology accepted in daily clinical routine is efficiency. We decided not to take the computational costs into account in our framework. The main reasons are the lack of a standard
Acknowledgments
Reinhard Hameeteman, Wiro Niessen and Theo van Walsum are supported by the Stichting voor de Technische Wetenschappen (STW) of The Netherlands Organization for Scientific Research (NWO). Aad van der Lugt is supported by Netherlands Organization for Health Research and Development (NWO-KF Grant 907-00-122) and The Dutch Heart Foundation (Grant 2007B161).
References (47)
- et al.
Deterministic and stochastic state model of right generalized cylinder (RGC-sm): application in computer phantoms synthesis
Graph. Models
(2003) - et al.
A review of 3d vessel lumen segmentation techniques: models, features and extraction schemes
Med. Image Anal.
(2009) - et al.
Analysis of pooled data from the randomised controlled trials of endarterectomy for symptomatic carotid stenosis
Lancet
(2003) - et al.
Standardized evaluation methodology and reference database for evaluating coronary artery centerline extraction algorithms
Med. Image Anal.
(2009) - et al.
Semi-automatic level-set based segmentation and stenosis quantification of the internal carotid artery in 3d cta data sets
Med. Image Anal.
(2007) - et al.
Comparing and combining algorithms for computer-aided detection of pulmonary nodules in computed tomography scans: the ANODE09 study
Med. Image Anal.
(2010) - et al.
Semi-automated computer assessment of the degree of carotid artery stenosis compares favorably to visual evaluation
J. Neurol. Sci.
(2008) - et al.
Preventing chronic disease: a vital investment
World Health Organization
(2005) - et al.
Robust and objective decomposition and mapping of bifurcating vessels
IEEE Trans. Med. Imag.
(2004) - Baltaxe, M., Flórez Valencia, L., Hernández Hoyos, M., Magnin, I.E., Orkisz, M., 2007. Fast-marching contours for the...