Elsevier

Neurocomputing

Volume 422, 21 January 2021, Pages 345-353
Neurocomputing

Probability-based Mask R-CNN for pulmonary embolism detection

https://doi.org/10.1016/j.neucom.2020.10.022Get rights and content

Abstract

Pulmonary embolism (PE), a blockage of the lung artery, is common and sometimes fatal. Early diagnosis and treatment of PE can reduce the risk of associated morbidity and mortality. However, it is a huge challenge to accurately detect PE, particularly for the case of small segmental and subsegmental emboli. In this paper, a flexible probability-based Mask R-CNN model, namely P-Mask RCNN, is proposed for PE detection. Specifically, the feature map is firstly upsampled to enrich the local details of the small objects and to extract anchors at a higher density. Then, a candidate area is constructed based on the probability of the appearance of PE. Finally, we extract the anchors in the candidate area of the enlarged feature map for subsequent detection. Extracting anchors in the candidate area instead of the entire image can not only reduce both time and space consumption caused by the enlarging feature maps but also improve the detection performance by eliminating most invalid anchors. Compared with Mask R-CNN, the anchors extracted by the proposed P-Mask RCNN is closer to the ground truth. Extensive experimental results demonstrate the effectiveness and efficiency of the proposed approach. The source code of our method is available athttps://github.com/longkun-uestc/P_Mask_RCNN.

Introduction

Pulmonary embolism (PE) refers to the situation that a portion of a blood clot remains in the pulmonary artery (Fig. 1), which ranks as the third most common cardiovascular disease. The small blood clots in the pulmonary arteries will block blood flow to lung, which can lead to insufficient oxygen supply to vital organs [1]. Early detection and treatment of PE could effectively decrease the mortality rate. Computed tomography pulmonary angiography (CTPA) has become one of the main methods for diagnosing PE today [2] since manual diagnosis of PE is a time-consuming and laborious task where the current best radiologists have a range of 6–23% misdiagnosis rate [1]. Computer-aided detection methods have been developed for aiding the radiologists in PE detection automatically. However, the traditional automatic PE detection methods [3], [4], [5], [6] could not perform successfully since the area of many PE lesions is generally with small size and irregular shape.

In recent years, Mask R-CNN [7], a kind of deep neural network framework, has been developed for object detection and instance segmentation. [8] applied Mask R-CNN for the detection and segmentation of oral diseases. [9] demonstrated that Mask R-CNN can efficiently and automatically segment microscope images of various nuclei. Mask R-CNN performs well in both large and medium objects detection and segmentation, while it underperforms with small objects [10].

Small object detection can usually be addressed by enlarging feature maps of original images. [11] proposed an upsampling-based anchor generation scheme using the high-resolution feature map of small objects. [12] designed a codec framework with symmetric convolutional deconvolution layer, which could improve the resolution of the feature map to make it better perform on small object detection. These approaches improve detection accuracy by changing the network structure to retain the information of small objects as much as possible. However, the anchor extraction strategy has a significant influence on the detection performance of small objects. Many anchor boxes with multiple scales and aspect ratios are generally designed manually on the entire image to match objects with various aspect ratios and spatial layout [13]. Due to the limited size of the feature map, few suitable anchors match small objects. Magnifying feature maps can alleviate the issue, but it will increase both time and space consumption. In most cases, anchors are extracted on the entire image with equal strides. Thus, a large number of invalid anchors are probably generated, which will decrease the final detection performance. Most anchor extraction methods imply a hypothesis that the probability of objects appearing on an image is subject to a uniform distribution, as investigated in [14].

Clinical diagnosis based on the medical image has shown the area prone to lesions is usually a small part of the entire image. For example, in clinical coronary artery calcification quantification, calcified areas generally distribute a narrow area around the coronary artery [15]. In breast cancer examination, the diseased tissue always appears in the internal area of the breast in the image [16]. On detection of PE, most plaques in CTPAs appeared on the left or right pulmonary artery, as well as its branches. Based on the clinical medical prior knowledge, it is reasonable to take into consideration the probability of lesion’s appearance at various positions in the medical image. Specifically, we construct a candidate area for the original image based on probability and then extract anchors in the corresponding candidate area of the enlarged feature map for subsequent detection and segmentation. The proposed model, namely probability-based Mask R-CNN (P-Mask RCNN), can improve small PE object detection precision without increasing time and space overhead.

The major contributions of our work are threefold:

We propose that the probability distribution characteristics of lesions is a kind of significant factor to narrow the scope of detection and thus improve detection accuracy for medical image processing.

A new framework for small object detection in the medical image is proposed. The performance of small object detection can be improved significantly by combining the probability-based anchor extraction strategy with appropriate feature map upsampling.

Our approach upgrades the AP, AP50 and AP75 from 37.55%, 77.39% and 33.21% to 41.87%, 81.55% and 41.43%, respectively, on small PE object detection task, compared with that of Mask R-CNN.

Section snippets

PE detection

Computer-aided detection (CAD) plays an important role in detecting PE. [6] proposed to compute circularity of the bright lumen and isophote curvature as region-level features for false-positive removal. [17] detected PE regions by extracting handcrafted features based on CT values, local contrast, and the second derivatives of voxels. However, limited by the representation ability of handcrafted features, these traditional methods were apt to suffer from a high false detection rate. To address

Data description

Our research includes 35 patients with pulmonary embolism between the age of 24 and 82. Each patient underwent a chest CT examination with the slice-thickness 1 mm and slice-interval1.5 mm. A total of 8792 CTPA images with a size of 512 × 512 pixels were obtained, 2304 of which contained lesion areas with 3781 PE regions of interest (PE-ROIs) altogether. More than 85% of these PE-ROIs are small objects with the square root of the area 32 pixels which only occupy on average 0.3% of the image

Our approach

Every pixel of the feature map in Mask R-CNN would generate 3 anchors with different aspect ratios, which is equivalent to anchor extraction performed on every wimage/wfeature pixels (defined as sampling stride) within the original image, where wimage is the width of the input image and wfeature is the width of the feature map.

However, the anchor extraction strategy has two shortcomings: First, the sampling stride is limited by the size of the feature map so that most ground truths cannot have

Experiments and results

In our experiments, we use the public dataset [26] (please refer to Section 3 for more details) to validate the performance of P-Mask RCNN in small PE detection. In this dataset, 2304 CTPA images with PE are collected from 35 patients. The images are sorted according to patient ID and slice ID. This dataset is divided into 3 parts, i.e., the training (the first 2000 images from 29 patients), validation (the following 100 images from 2 patients), and test (the last 204 images from 4 patients)

Conclusion

In this paper, we analyzed the shortcomings of Mask R-CNN in detecting small objects in medical images such as PE. Most anchors extracted by Mask R-CNN could not match the ground truths resulting in poor detection performance, due to the excessive extraction stride of the anchor. Increasing the density of the sample by simply magnifying the feature map will bring extra overhead of both memory and time. Therefore, we propose the regularity of lesions in medical images: most PEs only appear in

CRediT authorship contribution statement

Kun Long: Conceptualization, Methodology, Formal analysis, Software, Resources, Writing - original draft, Writing - review & editing. Lei Tang: Software, Validation, Investigation, Writing - original draft, Visualization. Xiaorong Pu: Methodology, Formal analysis, Investigation, Resources, Writing - review & editing, Funding acquisition, Project administration, Supervision. Yazhou Ren: Methodology, Resources, Writing - review & editing, Funding acquisition, Project administration, Supervision.

Declaration of Competing Interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Acknowledgments

This work was supported in part by the National Natural Science Foundation of China (No. 61806043 and 61832001), the China Postdoctoral Science Foundation (No. 2016M602674), the Guangdong Basic and Applied Basic Research Foundation (No. 2020A1515011002) and the Key Research and Development Project of Sichuan Province (No. 2020YFS0119).

Kun Long is currently a master student in the School of Computer Science and Engineering, University of Electronic Science and Technology of China, China. His research interests include machine learning and its applications in small object detection in medical imaging.

References (34)

  • N. Tajbakhsh et al.

    Computer-aided detection and visualization of pulmonary embolism using a novel, compact, and discriminative image representation

    Med. Image Anal.

    (2019)
  • G. Gonzlez, Cad-pe challenge, http://www.cad-pe.org/, 2019. Accessed September 15,...
  • X. Yang et al.

    A two-stage convolutional neural network for pulmonary embolism detection from ctpa images

    IEEE Access

    (2019)
  • M. Das et al.

    Computer-aided detection of pulmonary embolism: influence on radiologists’ detection performance with respect to vessel segments

    Eur. Radiol.

    (2008)
  • R. Wittenberg et al.

    Acute pulmonary embolism: effect of a computer-assisted detection prototype on diagnosis an observer study

    Radiology

    (2012)
  • C. Zhang et al.

    Automatic segmentation of arterial tree from 3d computed tomographic pulmonary angiography (ctpa) scans

    Comput. Assist. Surg.

    (2019)
  • H. Bouma et al.

    Automatic detection of pulmonary embolism in cta images

    IEEE Trans. Med. Imag.

    (2009)
  • K. He et al.

    Mask r-cnn

  • R. Anantharaman, M. Velazquez, Y. Lee, Utilizing mask r-cnn for detection and segmentation of oral diseases, in: 2018...
  • J.W. Johnson, Automatic nucleus segmentation with mask-rcnn, in: Science and Information Conference, Springer, 2019,...
  • M. Kisantal, Z. Wojna, J. Murawski, J. Naruniec, K. Cho, Augmentation for small object detection, arXiv preprint...
  • C. Eggert, D. Zecha, S. Brehm, R. Lienhart, Improving small object proposals for company logo detection, in:...
  • X.-J. Mao, C. Shen, Y.-B. Yang, Image restoration using convolutional auto-encoders with symmetric skip connections,...
  • X. Zhang, F. Wan, C. Liu, R. Ji, Q. Ye, Freeanchor: Learning to match anchors for visual object detection, arXiv...
  • C. Zhu et al.

    Seeing small faces from robust anchor’s perspective

  • L. Lu, Y. Zheng, G. Carneiro, L. Yang, Deep learning and convolutional neural networks for medical image computing,...
  • M.B. Nagarajan et al.

    Classification of small lesions in dynamic breast mri: eliminating the need for precise lesion segmentation through spatio-temporal analysis of contrast enhancement

    Mach. Vision Appl.

    (2013)
  • Cited by (0)

    Kun Long is currently a master student in the School of Computer Science and Engineering, University of Electronic Science and Technology of China, China. His research interests include machine learning and its applications in small object detection in medical imaging.

    Lei Tang is currently a master student in the School of Computer Science and Engineering, University of Electronic Science and Technology of China, China. His research interests include machine learning and e-Health.

    Xiaorong Pu is currently a professor with the School of Computer Science and Engineering, University of Electronic Science and Technology of China, Chengdu, China. She received the Ph.D. degree in computer application from the University of Electronic Science and Technology of China, Chengdu, China, in 2007. Her current research interests include neural networks, computer vision, computer aided diagnosis (CAD), e-Health.

    Yazhou Ren is currently an associate professor with the School of Computer Science and Engineering, University of Electronic Science and Technology of China, Chengdu, China. He received the B.Sc. degree in information and computation science and the Ph.D. degree in computer science from the South China University of Technology, Guangzhou, China, in 2009 and 2014, respectively. He visited the Data Mining Laboratory, George Mason University, USA, from 2012 to 2014. He has published more than 30 peer-reviewed research articles. His current research interests include clustering, self-paced learning, and transfer learning.

    Mingxiu Zheng is an associate professor with School of Computer Science and Engineering of Southwest University for Nationalities of China, Chengdu, China. She received the MA degree in Southwest University, Chongqing, China, in 2002. Her current research interests include data mining, big data, software engineering.

    Li Gao is currently the director and chief physician of neurology department of the 3rd People’s Hospital of Chengdu, the executive director of Chengdu Institute of Neurological Diseases, professor of neurology and tutor of postgraduate students in Southwest Medical University. His current research interests include consciousness and consciousness disorders, nerve injury and repair, aging neurology, remote diagnosis and management of chronic neurological diseases, intelligent diagnosis and treatment of neurological diseases.

    Chunjiang Song is currently a chief physician of the department of Neurology, Chengdu Sixth People’s Hospital, Chengdu, China. She received the bachelor’s degree from the Chongqing Medical University, Chongqing, China, in 1985. Her current research interests include several national, provincial and municipal scientific projects.

    Su Han is currently an attending physician in West China Fourth Hospital of Sichuan University, and also the director of Department of Information Technology of the hospital. She received the Master Degree in Medicine from Sichuan University, Chengdu, China, in 2005. Her current research interest is healthcare information technology.

    Min Zhou is currently a deputy chief physician in West China Fourth Hospital of Sichuan University. She received the Ph.D. degree in West China School of Medicine from Sichuan University, Chengdu, China, in 2005. Her current research interests include interstitial lung disease, pulmonary infectious, heavy metal poisoning.

    Fengbin Deng is currently pursuing a master’s degree at Nanchong North Sichuan Medical College. He graduated from Dalian Medical University with a bachelor’s degree in 2018. His current research interests include carotid atherosclerosis and plaque, stroke, and machine learning algorithms.

    View full text