Challenge ReportA deep learning framework for pancreas segmentation with multi-atlas registration and 3D level-set
Graphical abstract
Introduction
Pancreatic cancer is a deterious disease that has been significantly affecting human health. In the United States alone, it is estimated that annually there are about 40,000 people dying of pancreatic cancer and 50,000 new cases (Siegel et al., 2017). It is the 4th-5th leading cause of cancer-related mortality (Rahib et al., 2014). The prognosis for patients with pancreatic cancer is dismal, with a 5-year survival rate being less than 5% (Schima, Ba-Ssalamah, Kölblinger, Kulinna-Cosentini, Puespoek, Götzinger, 2007, Ryan, Hong, Bardeesy, 2014).
Computed tomography (CT) provides detailed imaging of the pancreas and is one of the most common examinations for diagnosing pancreatic cancer. A direct sign of pancreatic adenocarcinoma is the presence of hypoattenuating/hyperintense mass on contrast-enhanced CT images (Yang et al., 2013). Pancreas segmentation from CT images is also useful for detecting unusual volume changes and monitoring abnormal growths, playing important roles in diagnosis, prognosis, surgical planning as well as intra-operative guidance (Shimizu et al., 2010).
Usually, the recognized gold standard pancreas segmentation is obtained from experienced radiologists via manual delineation (Dou et al., 2017). However, manually tracing volumetric CT images in a slice-by-slice manner is tedious, time-consuming and labor expensive. In addition, manual labeling is highly prone to inter- and intra-variability. As such, a fully-automated pancreas segmentation pipeline is desired.
The pancreas is one of the most challenging organs in terms of automated segmentation because it occupies a relatively small region in the input CT image, e.g., less than 1.5% in a 2D image and less than 0.5% in a 3D image (Roth, Lu, Farag, Shin, Liu, Turkbey, Summers, 2015, Zhou, Xie, Shen, Wang, Fishman, Yuille, 2017). Another fundamental challenge is that pancreas’s appearance and shape often vary significantly from subject image to subject image (Farag et al., 2017). Furthermore, the boundary between the pancreas and its neighboring tissues is usually ambiguous with limited contrast, because of their similar imaging-related physical properties, e.g., the attenuation coefficient in CT imaging (Kronman and Joskowicz, 2016).
To solve these challenges, various segmentation algorithms have been designed in the past decades, mainly employing statistical shape modeling, level-set, multi-atlas and graphical models with hand-crafted features (Shimizu, Kimoto, Kobatake, Nawano, Shinozaki, 2010, Saito, Nawano, Shimizu, 2016, Tsai, Yezzi, Wells, Tempany, Tucker, Fan, Grimson, Willsky, 2003, Wolz, Chu, Misawa, Mori, Rueckert, 2012). The hand-crafted features may nevertheless have limited representation capabilities in dealing with the large variations of pancreas’s appearance and shape.
Recently, great progress has been made in natural image processing using convolutional neural network (CNN), given that multi-layer convolutions can hierachically learn highly representative features (Krizhevsky, Sutskever, Hinton, 2012, He, Zhang, Ren, Sun, 2016, Huang, Liu, Van Der Maaten, Weinberger, 2017). It has also inspired some successful applications in the medical image analysis domain (Ronneberger, Fischer, Brox, 2015, Chen, Dou, Yu, Qin, Heng, 2018, Setio, Ciompi, Litjens, Gerke, Jacobs, Van Riel, Wille, Naqibullah, Sánchez, van Ginneken, 2016, Li, Jiang, Zhang, Wang, Wang, Zheng, Menze, 2018, Milletari, Navab, Ahmadi, 2016, Yu, Cheng, Dou, Yang, Chen, Qin, Heng, 2017, Shen, Wu, Suk, 2017, Roth, Lu, Lay, Harrison, Farag, Sohn, Summers, 2018). CNN based volumetric medical image segmentation methods can be roughly categorized into two groups, i.e., 2D based and 3D based. In 2D CNN segmentation methods, the volumetric data was typically sliced along one of the three image directions (axial, sagittal, and coronal), and then those 2D slices were used as the input images of specifically designed networks (Li et al., 2018). A representative work is the U-net (Ronneberger et al., 2015). To make further use of the information encoded in the volumetric data, 2.5D image patches composed of 2D slices along all three directions were extracted by sliding window through all voxels (Roth, Lu, Liu, Yao, Seff, Cherry, Kim, Summers, 2016, Ciompi, de Hoop, van Riel, Chung, Scholten, Oudkerk, de Jong, Prokop, van Ginneken, 2015). Another type is slice-based 2.5D methods, wherein three 2D CNNs were constructed to separately segment the input image at three directions and then the segmentation results were fused via voting. These kinds of methods have also been applied to pancreas segmentation (Zhou, Xie, Shen, Wang, Fishman, Yuille, 2017, Yu, Xie, Wang, Zhou, Fishman, Yuille, 2018). However, 2D CNN based segmentation results usually suffer from topological errors (missing or extra parts). Later on, 3D CNN based methods have been proposed and adapted to pancreas segmentation (Dou, Yu, Chen, Jin, Yang, Qin, Heng, 2017, Zhu, Xia, Shen, Fishman, Yuille, 2018). For example, a 3D fully convolutional network (FCN) equipped with a 3D deep supervision mechanism was proposed (Dou et al., 2017). The input images of a 3D CNN are mainly local patches sampled from the whole image, considering the limited dataset with ground truth and computation memory. Patch-based models nevertheless could not sufficiently learn the global features of the volumetric data.
To alleviate this issue, we present a novel and efficient coarse-fine-refine deep learning-based fully-automated segmentation method for the pancreas. The fact that a smaller input region may lead to a more accurate segmentation motivates researchers to employ coarse-to-fine methods (Zhou, Xie, Shen, Wang, Fishman, Yuille, 2017, Zhu, Xia, Shen, Fishman, Yuille, 2018), wherein the coarse stage provides a rough localization and the fine stage polishes the segmentation to make it more accurate. Specifically, we first employ multi-atlas based registration to determine a relatively small input region (coarse step), then jointly use a patch-based 3D CNN and three slice-based 2D CNNs to fully utilize the volumetric information embedded in the CT image (fine step), and finally fine-tune the segmentation result based on a 3D level-set method (refine step).
Overall, our contributions in this work are three-fold:
- (1)
We come up with a coarse approach for pancreas segmentation in the framework of multi-atlas based fast diffeomorphic image registration (Wu and Tang, 2019). This step aims to eliminate redundant computation in 3D patch-based CNN and help subsequent neural networks achieve more accurate segmentation by decreasing the false positive probability.
- (2)
We propose a fine approach by jointly using a 3D patch-based CNN and a 2.5D slice-based CNN to fully extract local connected image features and global shape features. The probability maps obtained from the 3D CNN model are used to determine pancreas’s bounding box which is then concatenated with the original CT image to form two-channel inputs to three subsequent 2D U-nets. Such a mechanism can simultaneously speed up the optimization process and improve segmentation accuracy.
- (3)
We propose a 3D level-set method to refine the probability maps predicted from the previous 3D + 2.5D CNNs, with the segmentation results from the fine step serving as a constraint in the level-set method. We validate the proposed segmentation pipeline on three pancreas CT datasets.
The remainder of this paper is organized as follows. Section 2 provides a detailed description of our proposed coarse-fine-refine framework. The experimental setup and results are presented in Section 3, followed by a discussion of the results in Section 4.
Section snippets
Method
The flow chart of the entire testing procedure is illustrated in Fig. 1. At the coarse segmentation stage, we roughly locate pancreas via multi-atlas based image registration, from which a bounding box of the pancreas is determined and then 3D image patches are extracted and fed into a 3D CNN. At the fine segmentation stage, we segment pancreas by jointly considering the local connection feature (3D patch-based CNN) and global shape feature (2D slice-based CNN). At the refine segmentation
Dataset and evaluation measures
The accuracy of the pancreas segmentation results produced by our methodology is assessed using three datasets. The first dataset is obtained from a pancreas segmentation challenge on the 2nd International Symposium on Image Computing and Digital Medicine (ISICDM 2018)2. During the ISICDM 2018 Pancreas Segmentation Challenge, the organizers release 20 training and 16 testing images. The training set has manual annotations, but the ground truth segmentations
Discussion
The importance of each stage in our method has been rigorously identified based on the experimental results. They assembled effectively in the proposed pancreas segmentation framework. Firstly, the coarse segmentation is an important prior step for the subsequent CNN based segmentation. Multi-atlas based label fusion can effectively identify the rough location of the pancreas and narrow the potential segmenting region of interest in the subsequent CNNs, especially when combined with the largest
Conclusion
In this study, we proposed and validated a coarse-to-fine-to-refine framework for automated pancreas segmentation. The proposed coarse segmentation can largely reduce the input size to a subsequent 3D CNN. The output of the 3D CNN provides a rough location for the subsequent 2-channel 2D CNN. The output of the 2.5D CNN provides an initial contour and constraint for level-set based segmentation used at the refine stage. These components constitute a cascaded framework for segmenting pancreas. In
CRediT authorship contribution statement
Yue Zhang: Conceptualization, Methodology, Software, Writing - original draft. Jiong Wu: Conceptualization, Methodology, Software. Yilong Liu: Visualization, Investigation. Yifan Chen: Supervision, Funding acquisition. Wei Chen: Data curation. Ed. X. Wu: Supervision. Chunming Li: Software, Data curation. Xiaoying Tang: Supervision, Funding acquisition, Writing - review & editing.
Declaration of Competing Interest
The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.
Acknowledgments
The authors would like to thank Benxiang Jiang and Junyan Lyu from Southern University of Science and Technology as well as Mengye Lyu from the University of Hong Kong for their help on this work. This study was supported by the National Natural Science Foundation of China (62071210), the Shenzhen Basic Research Program (JCYJ20190809120205578), the National Key R&D Program of China (2017YFC0112404), the High-level University Fund (G02236002), and the National Natural Science Foundation of China
References (45)
- et al.
VoxResNet: deep voxelwise residual networks for brain segmentation from 3D MR images
NeuroImage
(2018) - et al.
Automatic classification of pulmonary peri-fissural nodules in computed tomography using an ensemble of 2D views and a convolutional neural network out-of-the-box
Med. Image Anal.
(2015) - et al.
3D fully convolutional networks for subcortical segmentation in MRI: a large-scale study
NeuroImage
(2018) - et al.
3D deeply supervised network for automated segmentation of volumetric medical images
Med. Image Anal.
(2017) - et al.
Efficient multi-scale 3D CNN with fully connected CRF for accurate brain lesion segmentation
Med. Image Anal.
(2017) - et al.
Fully convolutional network ensembles for white matter hyperintensities segmentation in MR images
NeuroImage
(2018) - et al.
Combining deep learning and level set for the automated segmentation of the left ventricle of the heart from cardiac cine magnetic resonance
Med. Image Anal.
(2017) - et al.
Spatial aggregation of holistically-nested convolutional neural networks for automated pancreas localization and segmentation
Med. Image Anal.
(2018) - et al.
Joint optimization of segmentation and shape prior from level-set-based statistical shape model, and its application to the automated segmentation of abdominal organs
Med. Image Anal.
(2016) - et al.
Bayesian parameter estimation and segmentation in the multi-atlas random orbit model
PLoS One
(2013)
Quicksilver: fast predictive image registration–a deep learning approach
NeuroImage
User-guided 3D active contour segmentation of anatomical structures: significantly improved efficiency and reliability
Neuroimage
Computing large deformation metric mappings via geodesic flows of diffeomorphisms
Int. J. Comput. Vis.
Pancreas segmentation in MRI using graph-based decision fusion on convolutional neural networks
International Conference on Medical Image Computing and Computer-Assisted Intervention
Computational analysis of LDDMM for brain mapping
Front. Neurosci.
Globally guided progressive fusion network for 3D pancreas segmentation
International Conference on Medical Image Computing and Computer-Assisted Intervention
A bottom-up approach for pancreas segmentation using cascaded superpixels and (deep) image patch labeling
IEEE Trans. Image Process.
A method for modeling noise in medical images
IEEE Trans. Med. Imaging
Deep residual learning for image recognition
Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition
Densely connected convolutional networks
Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition
Imagenet classification with deep convolutional neural networks
Advances in Neural Information Processing Systems
A geometric method for the detection and correction of segmentation leaks of anatomical structures in volumetric medical images
Int. J. Comput. Assist. Radiol. Surg.
Cited by (0)
- 1
Equal contribution.