Skip to main content

Advertisement

SpringerLink
Log in

An improved Hover-net for nuclear segmentation and classification in histopathology images

  • Original Article
  • Published:
Neural Computing and Applications Aims and scope Submit manuscript

Abstract

Concurrent nuclear segmentation and classification in Hematoxylin & Eosin-stained histopathology images are a crucial task in disease diagnosis and prognosis. Albeit recent advancement of deep learning models, this task remains challenging as each nucleus occupies a limited number of pixels, and nuclei have large intra-class variability and high inter-class similarities in morphology. In this work, we proposed a tissue region-guided dilated Hover-net (TRG-Dilated Hover-net) that consists of a tissue region segmentation model and a dilated Hover-net model. The latter incorporated the dilated convolution and the atrous spatial pyramid pooling feature pyramids to expand the receptive field; therefore, more information about nuclei and their spacial locations can be captured. Our method achieved the state-of-the-art performance on four benchmark datasets of various cancer types and the in-house curated Breast Cancer dataset.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9

Similar content being viewed by others

Data availability

The datasets generated during and/or analyzed during the current study are available from the corresponding author on reasonable request. The source code of the model is available at: https://github.com/LuluQin766/TRG-Dilated_Hover_net.

References

  1. Ferlay J, Soerjomataram I, Dikshit R, Eser S, Mathers C, Rebelo M, Parkin DM, Forman D, Bray F (2015) Cancer incidence and mortality worldwide: sources, methods and major patterns in GLOBOCAN 2012. Int J Cancer 136(5):359–386

    Article  Google Scholar 

  2. Elmore JG, Longton GM, Carney PA, Geller BM, Onega T, Tosteson AN, Nelson HD, Pepe MS, Allison KH, Schnitt SJ (2015) Diagnostic concordance among pathologists interpreting breast biopsy specimens. JAMA 313(11):1122–1132

    Article  Google Scholar 

  3. Hollandi R, Moshkov N, Paavolainen L, Tasnadi E, Piccinini F, Horvath P (2022) Nucleus segmentation: towards automated solutions. Trends Cell Biol 32:295–310

    Article  Google Scholar 

  4. Labani-Motlagh A, Ashja-Mahdavi M, Loskog A (2020) The tumor microenvironment: a milieu hindering and obstructing antitumor immune responses. Front Immunol 11:940

    Article  Google Scholar 

  5. Rastogi P, Khanna K, Singh V (2022) Gland segmentation in colorectal cancer histopathological images using u-net inspired convolutional network. Neural Comput Appl 34(7):5383–5395

    Article  Google Scholar 

  6. Orhan A, Vogelsang RP, Andersen MB, Madsen MT, Hölmich ER, Raskov H, Gögenur I (2020) The prognostic value of tumour-infiltrating lymphocytes in pancreatic cancer: a systematic review and meta-analysis. Eur J Cancer 132:71–84

    Article  Google Scholar 

  7. Agustin RI, Arif A, Sukorini U (2021) Classification of immature white blood cells in acute lymphoblastic leukemia l1 using neural networks particle swarm optimization. Neural Comput Appl 33(17):10869–10880

    Article  Google Scholar 

  8. Zafar MM, Rauf Z, Sohail A, Khan AR, Obaidullah M, Khan SH, Lee YS, Khan A (2022) Detection of tumour infiltrating lymphocytes in cd3 and cd8 stained histopathological images using a two-phase deep cnn. Photodiagn Photodyn Ther 37:102676

    Article  Google Scholar 

  9. Mahmood T, Owais M, Noh KJ, Yoon HS, Koo JH, Haider A, Sultan H, Park KR (2021) Accurate segmentation of nuclear regions with multi-organ histopathology images using artificial intelligence for cancer diagnosis in personalized medicine. J Pers Med 11(6):515

    Article  Google Scholar 

  10. Brindha V, Jayashree P, Karthik P, Manikandan P (2022) Tumor grading model employing geometric analysis of histopathological images with characteristic nuclei dictionary. Comput Biol Med 149:106008

    Article  Google Scholar 

  11. Çayır S, Solmaz G, Kusetogullari H, Tokat F, Bozaba E, Karakaya S, Iheme LO, Tekin E, Özsoy G, Ayaltı S et al (2022) Mitnet: a novel dataset and a two-stage deep learning approach for mitosis recognition in whole slide images of breast cancer tissue. Neural Comput Appl 34:1–15

    Article  Google Scholar 

  12. Hashimoto N, Fukushima D, Koga R, Takagi Y, Ko K, Kohno K, Nakaguro M, Nakamura S, Hontani H, Takeuchi I (2020) Multi-scale domain-adversarial multiple-instance CNN for cancer subtype classification with unannotated histopathological images. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 3852–3861

  13. Dodington DW, Lagree A, Tabbarah S, Mohebpour M, Sadeghi-Naini A, Tran WT, Lu F-I (2021) Analysis of tumor nuclear features using artificial intelligence to predict response to neoadjuvant chemotherapy in high-risk breast cancer patients. Breast Cancer Res Treat 186(2):379–389

    Article  Google Scholar 

  14. Lu C, Romo-Bucheli D, Wang X, Janowczyk A, Ganesan S, Gilmore H, Rimm D, Madabhushi A (2018) Nuclear shape and orientation features from H &E images predict survival in early-stage estrogen receptor-positive breast cancers. Lab Investig 98(11):1438–1448

    Article  Google Scholar 

  15. Hayakawa T, Prasath V, Kawanaka H, Aronow BJ, Tsuruoka S (2021) Computational nuclei segmentation methods in digital pathology: a survey. Arch Comput Methods Eng 28(1):1–13

    Article  Google Scholar 

  16. Kumar N, Verma R, Anand D, Zhou Y, Onder OF, Tsougenis E, Chen H, Heng P-A, Li J, Hu Z (2019) A multi-organ nucleus segmentation challenge. IEEE Trans Med Imaging 39(5):1380–1391

    Article  Google Scholar 

  17. Komura D, Ishikawa S (2018) Machine learning methods for histopathological image analysis. Comput Struct Biotechnol J 16:34–42

    Article  Google Scholar 

  18. Wang S, Yang DM, Rong R, Zhan X, Xiao G (2019) Pathology image analysis using segmentation deep learning algorithms. Am J Pathol 189(9):1686–1698

    Article  Google Scholar 

  19. He H, Huang Z, Ding Y, Song G, Wang L, Ren Q, Wei P, Gao Z, Chen J (2021) Cdnet: centripetal direction network for nuclear instance segmentation. In: Proceedings of the IEEE/CVF international conference on computer vision (ICCV), pp 4026–4035

  20. Phansalkar N, More S, Sabale A, Joshi M (2011) Adaptive local thresholding for detection of nuclei in diversity stained cytology images. In: 2011 international conference on communications and signal processing, pp 218–220

  21. Veta M, Van Diest PJ, Kornegoor R, Huisman A, Viergever MA, Pluim JP (2013) Automatic nuclei segmentation in H &E stained breast cancer histopathology images. PLoS ONE 8(7):70221

    Article  Google Scholar 

  22. Wang P, Hu X, Li Y, Liu Q, Zhu X (2016) Automatic cell nuclei segmentation and classification of breast cancer histopathology images. Signal Process 122:1–13

    Article  Google Scholar 

  23. Ananthi V, Balasubramaniam P (2016) A new thresholding technique based on fuzzy set as an application to leukocyte nucleus segmentation. Comput Methods Programs Biomed 134:165–177

    Article  Google Scholar 

  24. Ronneberger O, Fischer P, Brox T (2015) U-net: convolutional networks for biomedical image segmentation. In: International conference on medical image computing and computer-assisted intervention, pp 234–241

  25. Liu X, Guo Z, Cao J, Tang J (2021) Mdc-net: a new convolutional neural network for nucleus segmentation in histopathology images with distance maps and contour information. Comput Biol Med 135:104543

    Article  Google Scholar 

  26. He H, Zhang C, Chen J, Geng R, Chen L, Liang Y, Lu Y, Wu J, Xu Y (2021) A hybrid-attention nested unet for nuclear segmentation in histopathological images. Front Mol Biosci 8:614174

    Article  Google Scholar 

  27. Kaur A, Kaur L, Singh A (2021) Ga-unet: Unet-based framework for segmentation of 2d and 3d medical images applicable on heterogeneous datasets. Neural Comput Appl 33(21):14991–15025

    Article  Google Scholar 

  28. Peng D, Yu X, Peng W, Lu J (2021) Dgfau-net: global feature attention upsampling network for medical image segmentation. Neural Comput Appl 33(18):12023–12037

    Article  Google Scholar 

  29. Chen B, Liu Y, Zhang Z, Lu G, Zhang D (2021) Transattunet: multi-level attention-guided u-net with transformer for medical image segmentation. arXiv preprint https://arxiv.org/abs/2107.05274

  30. Qin J, He Y, Zhou Y, Zhao J, Ding B (2022) Reu-net: region-enhanced nuclei segmentation network. Comput Biol Med 146:105546

    Article  Google Scholar 

  31. Wu Y, Liao K, Chen J, Wang J, Chen DZ, Gao H, Wu J (2022) D-former: a u-shaped dilated transformer for 3d medical image segmentation. Neural Comput Appl 35:1–14

    Google Scholar 

  32. Graham S, Vu QD, Raza SEA, Azam A, Tsang YW, Kwak JT, Rajpoot N (2019) Hover-net: simultaneous segmentation and classification of nuclei in multi-tissue histology images. Med Image Anal 58:101563

    Article  Google Scholar 

  33. Bancher B, Mahbod A, Ellinger I, Ecker R, Dorffner G (2021) Improving mask r-CNN for nuclei instance segmentation in Hematoxylin & Eosin-stained histological images. In: MICCAI workshop on computational pathology, pp 20–35

  34. Huang H, Feng X, Jiang J, Chen P, Zhou S (2022) Mask RCNN algorithm for nuclei detection on breast cancer histopathological images. Int J Imaging Syst Technol 32(1):209–217

    Article  Google Scholar 

  35. Ilyas T, Mannan ZI, Khan A, Azam S, Kim H, De Boer F (2022) Tsfd-net: tissue specific feature distillation network for nuclei segmentation and classification. Neural Netw 151:1–15

    Article  Google Scholar 

  36. Shephard AJ, Graham S, Bashir S, Jahanifar M, Mahmood H, Khurram A, Rajpoot NM (2021) Simultaneous nuclear instance and layer segmentation in oral epithelial dysplasia. In: Proceedings of the IEEE/CVF international conference on computer vision, pp 552–561

  37. Azzuni H, Ridzuan M, Xu M, Yaqub M (2022) Color space-based hover-net for nuclei instance segmentation and classification. arXiv preprint https://arxiv.org/abs/2203.01940

  38. Zhang W, Zhang J (2022) Aughover-net: augmenting hover-net for nucleus segmentation and classification. arXiv preprint https://arxiv.org/abs/2203.03415

  39. Doan TN, Song B, Vuong TT, Kim K, Kwak JT (2022) Sonnet: a self-guided ordinal regression neural network for segmentation and classification of nuclei in large-scale multi-tissue histology images. IEEE J Biomed Health Inform 26(7):3218–3228

    Article  Google Scholar 

  40. Chen L-C, Zhu Y, Papandreou G, Schroff F, Adam H (2018) Encoder–decoder with atrous separable convolution for semantic image segmentation. In: Proceedings of the European conference on computer vision (ECCV), pp 801–818

  41. Czajkowska J, Badura P, Korzekwa S, Płatkowska-Szczerek A (2022) Automated segmentation of epidermis in high-frequency ultrasound of pathological skin using a cascade of deeplab v3+ networks and fuzzy connectedness. Comput Med Imaging Graph 95:102023

    Article  Google Scholar 

  42. Chen L-C, Papandreou G, Schroff F, Adam H (2017) Rethinking atrous convolution for semantic image segmentation. arXiv preprint https://arxiv.org/abs/1706.05587

  43. Chen L-C, Papandreou G, Kokkinos I, Murphy K, Yuille AL (2017) Deeplab: semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs. IEEE Trans Pattern Anal Mach Intell 40(4):834–848

    Article  Google Scholar 

  44. Gridach M (2021) Pydinet: pyramid dilated network for medical image segmentation. Neural Netw 140:274–281

    Article  Google Scholar 

  45. Huang G, Liu Z, Van Der Maaten L, Weinberger KQ (2017) Densely connected convolutional networks. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 4700–4708

  46. Amgad M, Elfandy H, Hussein H, Atteya LA, Elsebaie MA, Abo Elnasr LS, Sakr RA, Salem HS, Ismail AF, Saad AM (2019) Structured crowdsourcing enables convolutional segmentation of histology images. Bioinformatics 35(18):3461–3467

    Article  Google Scholar 

  47. Verma R, Kumar N, Patil A, Kurian NC, Rane S, Graham S, Vu QD, Zwager M, Raza SEA, Rajpoot N (2021) Monusac 2020: a multi-organ nuclei segmentation and classification challenge. IEEE Trans Med Imaging 40(12):3413–3423

    Article  Google Scholar 

  48. Gamper J, Koohbanani NA, Graham S, Jahanifar M, Khurram SA, Azam A, Hewitt K, Rajpoot N (2020) Pannuke dataset extension, insights and baselines. arXiv preprint https://arxiv.org/abs/2003.10778

  49. Amgad M, Atteya LA, Hussein H, Mohammed KH, Hafiz E, Elsebaie MA, Alhusseiny AM, AlMoslemany MA, Elmatboly AM, Pappalardo PA, et al (2022) Nucls: a scalable crowdsourcing approach and dataset for nucleus classification and segmentation in breast cancer. GigaScience 11(giac037)

  50. Zhou Z, Rahman Siddiquee MM, Tajbakhsh N, Liang J (2018) Unet++: a nested u-net architecture for medical image segmentation. In: Deep learning in medical image analysis and multimodal learning for clinical decision support, pp 3–11

  51. Chaurasia A, Culurciello E (2017) Linknet: exploiting encoder representations for efficient semantic segmentation. In: 2017 IEEE visual communications and image processing (VCIP), pp 1–4

  52. Fan T, Wang G, Li Y, Wang H (2020) Ma-net: a multi-scale attention network for liver and tumor segmentation. IEEE Access 8:179656–179665

    Article  Google Scholar 

  53. Li H, Xiong P, An J, Wang L (2018) Pyramid attention network for semantic segmentation. arXiv preprint https://arxiv.org/abs/1805.10180

  54. Zhao H, Shi J, Qi X, Wang X, Jia J (2017) Pyramid scene parsing network. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 2881–2890

  55. He K, Zhang X, Ren S, Sun J (2016) Identity mappings in deep residual networks. In: European conference on computer vision, pp 630–645

  56. Deng J, Dong W, Socher R, Li L-J, Li K, Fei-Fei L (2009) Imagenet: a large-scale hierarchical image database. In: 2009 IEEE conference on computer vision and pattern recognition, pp 248–255

Download references

Acknowledgements

This work was partially supported by the National Key Research and Development Program of China, under Grant 2022YFF1202104, the National Natural Science Foundation of China (Grant No. 61871272) and the Shenzhen Fundamental Research Program (Grant No. JCYJ20190808173617147).

Author information

Author notes

  1. Ji Wang and Lulu Qin have contributed equally to this work.

Authors and Affiliations

  1. Department of Breast Surgery, Yantai Yuhuangding Hospital, Yantai, 264008, Shandong, China

    Ji Wang & Guangdong Qiao

  2. College of Computer Science and Software Engineering, Shenzhen University, Nanhai Avenue 3688, Shenzhen, 518060, China

    Lulu Qin & Zexuan Zhu

  3. Guangdong Jiyin Biotech Co. Ltd, Shenzhen, 518055, China

    Dan Chen, Juan Wang & Bo-Wei Han

Authors
  1. Ji Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Lulu Qin
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Dan Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Juan Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Bo-Wei Han
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Zexuan Zhu
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Guangdong Qiao
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding authors

Correspondence to Zexuan Zhu or Guangdong Qiao.

Ethics declarations

Conflict of interest

The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Wang, J., Qin, L., Chen, D. et al. An improved Hover-net for nuclear segmentation and classification in histopathology images. Neural Comput & Applic 35, 14403–14417 (2023). https://doi.org/10.1007/s00521-023-08394-3

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00521-023-08394-3

Keywords

Search

Navigation