Skip to main content

Advertisement

Log in

MaskMitosis: a deep learning framework for fully supervised, weakly supervised, and unsupervised mitosis detection in histopathology images

  • Original Article
  • Published:
Medical & Biological Engineering & Computing Aims and scope Submit manuscript

Abstract

Counting the mitotic cells in histopathological cancerous tissue areas is the most relevant indicator of tumor grade in aggressive breast cancer diagnosis. In this paper, we propose a robust and accurate technique for the automatic detection of mitoses from histological breast cancer slides using the multi-task deep learning framework for object detection and instance segmentation Mask RCNN. Our mitosis detection and instance segmentation framework is deployed for two main tasks: it is used as a detection network to perform mitosis localization and classification in the fully annotated mitosis datasets (i.e., the pixel-level annotated datasets), and it is used as a segmentation network to estimate the mitosis mask labels for the weakly annotated mitosis datasets (i.e., the datasets with centroid-pixel labels only). We evaluate our approach on the fully annotated 2012 ICPR grand challenge dataset and the weakly annotated 2014 ICPR MITOS-ATYPIA challenge dataset. Our evaluation experiments show that we can obtain the highest F-score of 0.863 on the 2012 ICPR dataset by applying the mitosis detection and instance segmentation model trained on the pixel-level labels provided by this dataset. For the weakly annotated 2014 ICPR dataset, we first employ the mitosis detection and instance segmentation model trained on the fully annotated 2012 ICPR dataset to segment the centroid-pixel annotated mitosis ground truths, and produce the mitosis mask and bounding box labels. These estimated labels are then used to train another mitosis detection and instance segmentation model for mitosis detection on the 2014 ICPR dataset. By adopting this two-stage framework, our method outperforms all state-of-the-art mitosis detection approaches on the 2014 ICPR dataset by achieving an F-score of 0.475. Moreover, we show that the proposed framework can also perform unsupervised mitosis detection through the estimation of pseudo labels for an unlabeled dataset and it can achieve promising detection results. Code has been made available at: https://github.com/MeriemSebai/MaskMitosis.

Overview of MaskMitosis framework.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12

Similar content being viewed by others

References

  1. Abadi M, Agarwal A, Barham P, Brevdo E, Chen Z, Citro C, Corrado GS, Davis A, Dean J, Devin M, Ghemawat S, Goodfellow I, Harp A, Irving G, Isard M, Jia Y, Jozefowicz R, Kaiser L, Kudlur M, Levenberg J, Mané D, Monga R, Moore S, Murray D, Olah C, Schuster M, Shlens J, Steiner B, Sutskever I, Talwar K, Tucker P, Vanhoucke V, Vasudevan V, Viégas F, Vinyals O, Warden P, Wattenberg M, Wicke M, Yu Y, Zheng X (2015) TensorFlow: large-scale machine learning on heterogeneous systems. https://www.tensorflow.org/. (Software available from tensorflow.org)

  2. Abdulla W (2017) Mask R-CNN for object detection and instance segmentation on Keras and TensorFlow. https://github.com/matterport/Mask_RCNN

  3. Ben-Cohen A, Diamant I, Klang E, Amitai M, Greenspan H (2016) Fully convolutional network for liver segmentation and lesions detection. In: Deep learning and data labeling for medical applications, pp 77–85. Springer

  4. Biswas M, Kuppili V, Saba L, Edla D R, Suri H S, Sharma A, Cuadrado-Godia E, Laird J R, Nicolaides A, Suri J S (2019) Deep learning fully convolution network for lumen characterization in diabetic patients using carotid ultrasound: a tool for stroke risk. Med Biol Eng Comput 57(2):543–564

    PubMed  Google Scholar 

  5. Chen H, Dou Q, Wang X, Qin J, Heng P A, et al. (2016) Mitosis detection in breast cancer histology images via deep cascaded networks. In: AAAI, pp 1160–1166

  6. Chen H, Wang X, Heng P A (2016) Automated mitosis detection with deep regression networks. In: 2016 IEEE 13th International Symposium on Biomedical Imaging (ISBI), pp 1204–1207. IEEE

  7. Chollet F et al (2015) Keras . https://github.com/fchollet/keras

  8. Cireṡan DC, Giusti A, Gambardella L M, Schmidhuber J (2013) Mitosis detection in breast cancer histology images with deep neural networks. In: International Conference on Medical Image Computing and Computer-assisted Intervention, pp 411–418. Springer

  9. Cui S, Mao L, Jiang J, Liu C, Xiong S (2018) Automatic semantic segmentation of brain gliomas from MRI images using a deep cascaded neural network. J Healthc Eng 2018:4940593

    PubMed  PubMed Central  Google Scholar 

  10. Dong H, Yang G, Liu F, Mo Y, Guo Y (2017) Automatic brain tumor detection and segmentation using U-Net based fully convolutional networks. In: Annual Conference on Medical Image Understanding and Analysis, pp 506–517. Springer

  11. Elston C W, Ellis I O (1991) Pathological prognostic factors in breast cancer. I. the value of histological grade in breast cancer: experience from a large study with long-term follow-up. Histopathology 19(5):403–410

    CAS  PubMed  Google Scholar 

  12. Fu H, Cheng J, Xu Y, Zhang C, Wong D W K, Liu J, Cao X (2018) Disc-aware ensemble network for glaucoma screening from fundus image. IEEE Trans Med Imaging 37(11):2493–2501

    PubMed  Google Scholar 

  13. Ganin Y, Ustinova E, Ajakan H, Germain P, Larochelle H, Laviolette F, Marchand M, Lempitsky V (2016) Domain-adversarial training of neural networks. J Mach Learn Res 17(1):2096–2030

    Google Scholar 

  14. García-Zapirain B, Elmogy M, El-Baz A, Elmaghraby AS (2018) Classification of pressure ulcer tissues with 3D convolutional neural network. Med Biol Eng Comput 56(12):2245–2258

    PubMed  Google Scholar 

  15. Girshick R, Donahue J, Darrell T, Malik J (2014) Rich feature hierarchies for accurate object detection and semantic segmentation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp 580–587

  16. Giusti A, Caccia C, Cireṡari DC, Schmidhuber J, Gambardella L M (2014) A comparison of algorithms and humans for mitosis detection. In: 2014 IEEE 11th International Symposium on Biomedical Imaging (ISBI), pp 1360–1363. IEEE

  17. Gu Z, Cheng J, Fu H, Zhou K, Hao H, Zhao Y, Zhang T, Gao S, Liu J (2019) CE-Net: context encoder network for 2D medical image segmentation. arXiv:1903.02740

  18. He K, Gkioxari G, Dollár P, Girshick R (2017) Mask R-CNN. In: Computer Vision (ICCV), 2017 IEEE International Conference on, pp 2980–2988. IEEE

  19. He K, Zhang X, Ren S, Sun J (2016) Deep residual learning for image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp 770–778

  20. Huang C H, Lee H K (2012) Automated mitosis detection based on exclusive independent component analysis. In: Proceedings of the 21st International Conference on Pattern Recognition (ICPR2012), pp 1856–1859. IEEE

  21. Huang Z, Wang X, Wang J, Liu W, Wang J (2018) Weakly-supervised semantic segmentation network with deep seeded region growing. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp 7014–7023

  22. Irshad H (2013) Automated mitosis detection in histopathology using morphological and multi-channel statistics features. J Pathol Inform 4:10

    PubMed  PubMed Central  Google Scholar 

  23. Jader G, Fontineli J, Ruiz M, Abdalla K, Pithon M, Oliveira L (2018) Deep instance segmentation of teeth in panoramic x-ray images. In: 2018 31st SIBGRAPI Conference on Graphics, Patterns and Images (SIBGRAPI), pp 400–407. IEEE

  24. Johnson J W (2018) Adapting Mask-RCNN for automatic nucleus segmentation. arXiv:1805.00500

  25. Khan A M, El-Daly H, Rajpoot N M (2012) A gamma-Gaussian mixture model for detection of mitotic cells in breast cancer histopathology images. In: 2012 21st International Conference on Pattern Recognition (ICPR), pp 149–152. IEEE

  26. Krizhevsky A, Sutskever I, Hinton G E (2012) Imagenet classification with deep convolutional neural networks. In: Advances in neural information processing systems, pp 1097–1105

  27. Lafarge M, Pluim J, Eppenhof K, Veta M (2019) Learning domain-invariant representations of histological images. Front Med 6:162

    Google Scholar 

  28. Lafarge M W, Pluim J P, Eppenhof K A, Moeskops P, Veta M (2017) Domain-adversarial neural networks to address the appearance variability of histopathology images. In: Deep learning in medical image analysis and multimodal learning for clinical decision support, pp 83–91. Springer

  29. Li C, Wang X, Liu W (2017) Neural features for pedestrian detection. Neurocomputing 238:420–432

    Google Scholar 

  30. Li C, Wang X, Liu W, Latecki L J (2018) Deepmitosis: mitosis detection via deep detection, verification and segmentation networks. Med Image Anal 45:121–133

    PubMed  Google Scholar 

  31. Lin T Y, Dollár P, Girshick R, He K, Hariharan B, Belongie S (2017) Feature pyramid networks for object detection. In: CVPR, vol 1, p. 4

  32. Lin T Y, Maire M, Belongie S, Hays J, Perona P, Ramanan D, Dollár P, Zitnick C L (2014) Microsoft COCO: common objects in context. In: European conference on computer vision, pp 740–755. Springer

  33. Liu J, Li P (2018) A mask R-CNN model with improved region proposal network for medical ultrasound image. In: International Conference on Intelligent Computing, pp 26–33. Springer

  34. Liu M, Dong J, Dong X, Yu H, Qi L (2018) Segmentation of lung nodule in CT images based on mask R-CNN. In: 2018 9th International Conference on Awareness Science and Technology (iCAST), pp 1–6. IEEE

  35. Liu X, Jiang D, Wang M, Song Z (2018) Image synthesis-based multi-modal image registration framework by using deep fully convolutional networks. Med Biol Eng Comput 57:1–12

    CAS  Google Scholar 

  36. Liu Y, Zhang P, Song Q, Li A, Zhang P, Gui Z (2018) Automatic segmentation of cervical nuclei based on deep learning and a conditional random field. IEEE Access 6:53709–53721

    Google Scholar 

  37. Long J, Shelhamer E, Darrell T (2015) Fully convolutional networks for semantic segmentation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp 3431–3440

  38. Macenko M, Niethammer M, Marron J S, Borland D, Woosley J T, Guan X, Schmitt C, Thomas N E (2009) A method for normalizing histology slides for quantitative analysis. In: IEEE International Symposium on Biomedical Imaging: From Nano to Macro, 2009. ISBI’09, pp 1107–1110. IEEE

  39. Malon C D, Cosatto E (2013) Classification of mitotic figures with convolutional neural networks and seeded blob features. J Pathol Inf 4:9

    Google Scholar 

  40. MITOS-ATYPIA-14: Mitos-atypia-14-dataset https://mitos-atypia-14.grand-challenge.org/dataset/ https://mitos-atypia-14.grand-challenge.org/dataset/ (2014). (Online; accessed 19.02.04)

  41. Otálora S, Atzori M, Andrearczyk V, Khan A, Müller H. (2019) Staining invariant features for improving generalization of deep convolutional neural networks in computational pathology. Front Bioeng Biotech 7:198

    Google Scholar 

  42. Pang S, Du A, Orgun M A, Yu Z (2019) A novel fused convolutional neural network for biomedical image classification. Med Biol Eng Comput 57(1):107–121

    PubMed  Google Scholar 

  43. Paul A, Dey A, Mukherjee D P, Sivaswamy J, Tourani V (2015) Regenerative random forest with automatic feature selection to detect mitosis in histopathological breast cancer images. In: International Conference on Medical Image Computing and Computer-assisted Intervention, pp 94–102. Springer

  44. Ren S, He K, Girshick R, Sun J (2015) Faster R-CNN: towards real-time object detection with region proposal networks. In: Advances in neural information processing systems, pp 91–99

  45. Ronneberger O, Fischer P, Brox T (2015) U-net: Convolutional networks for biomedical image segmentation. In: International Conference on Medical Image Computing and Computer-assisted Intervention, pp 234–241. Springer

  46. Roux L, Racoceanu D, Loménie N, Kulikova M, Irshad H, Klossa J, Capron F, Genestie C, Naour G L, Gurcan M N (2013) Mitosis detection in breast cancer histological images an ICPR 2012 contest. J Pathol Inform 4:8

    PubMed  PubMed Central  Google Scholar 

  47. Russakovsky O, Deng J, Su H, Krause J, Satheesh S, Ma S, Huang Z, Karpathy A, Khosla A, Bernstein M (2015) Imagenet large scale visual recognition challenge. Int J Comput Vis 115(3):211–252

    Google Scholar 

  48. Russakovsky O, Deng J, Su H, Krause J, Satheesh S, Ma S, Huang Z, Karpathy A, Khosla A, Bernstein M, Berg A C, Fei-Fei L (2015) Imagenet large scale visual recognition challenge. Int J Comput Vis 115(3):211–252. 10.1007/s11263-015-0816-y

    Google Scholar 

  49. Sommer C, Fiaschi L, Hamprecht F A, Gerlich D W (2012) Learning-based mitotic cell detection in histopathological images. In: 2012 21st International Conference on Pattern Recognition (ICPR), pp 2306–2309. IEEE

  50. Tang P, Wang X, Huang Z, Bai X, Liu W (2017) Deep patch learning for weakly supervised object classification and discovery. Pattern Recogn 71:446–459

    Google Scholar 

  51. Tashk A, Helfroush M S, Danyali H, Akbarzadeh M (2013) An automatic mitosis detection method for breast cancer histopathology slide images based on objective and pixel-wise textural features classification. In: 2013 5th Conference on Information and Knowledge Technology (IKT), pp 406–410. IEEE

  52. Tek F B (2013) Mitosis detection using generic features and an ensemble of cascade AdaBoosts. J Pathol Inform 4:12

    PubMed  PubMed Central  Google Scholar 

  53. TUPAC16: Tumor-proliferation-assessment-challenge. http://tupac.tue-image.nl/ http://tupac.tue-image.nl/ (2016). (Online; accessed 19.02.04)

  54. Uijlings JR, Van De Sande KE, Gevers T, Smeulders AW (2013) Selective search for object recognition. Int J Comput Vis 104(2):154–171

    Google Scholar 

  55. Veta M, van Diest P J, Pluim J P (2013) Detecting mitotic figures in breast cancer histopathology images. In: Medical imaging 2013: digital pathology, vol 8676, p. 867607. International Society for Optics and Photonics

  56. Veta M, Van Diest P J, Willems S M, Wang H, Madabhushi A, Cruz-Roa A, Gonzalez F, Larsen A B, Vestergaard J S, Dahl A B et al (2015) Assessment of algorithms for mitosis detection in breast cancer histopathology images. Med Image Anal 20(1):237–248

    PubMed  Google Scholar 

  57. Wang H, Cruz-Roa A, Basavanhally A, Gilmore H, Shih N, Feldman M, Tomaszewski J, Gonzalez F, Madabhushi A (2014) Cascaded ensemble of convolutional neural networks and handcrafted features for mitosis detection. In: Medical imaging 2014: digital pathology, vol 9041, p. 90410B. International society for optics and photonics

  58. Wang J, Wang X, Liu W (2018) Weakly-and semi-supervised faster R-CNN with curriculum learning. In: 2018 24th International Conference on Pattern Recognition (ICPR), pp 2416–2421. IEEE

  59. Wang X, Yan Y, Tang P, Bai X, Liu W (2018) Revisiting multiple instance neural networks. Pattern Recogn 74:15–24

    Google Scholar 

  60. Zhao H, Qi X, Shen X, Shi J, Jia J (2018) ICNet for real-time semantic segmentation on high-resolution images. In: Proceedings of the European Conference on Computer Vision (ECCV), pp 405–420

  61. Zhao H, Shi J, Qi X, Wang X, Jia J (2017) Pyramid scene parsing network. In: CVPR

Download references

Author information

Authors and Affiliations

Authors

Corresponding authors

Correspondence to Meriem Sebai or Tianjiang Wang.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Sebai, M., Wang, X. & Wang, T. MaskMitosis: a deep learning framework for fully supervised, weakly supervised, and unsupervised mitosis detection in histopathology images. Med Biol Eng Comput 58, 1603–1623 (2020). https://doi.org/10.1007/s11517-020-02175-z

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11517-020-02175-z

Keywords

Navigation