Skip to main content

Advertisement

Springer Nature Link
Log in

Cell detection in pathology and microscopy images with multi-scale fully convolutional neural networks

  • Published:
World Wide Web Aims and scope Submit manuscript

Abstract

Automated nucleus/cell detection is usually considered as the basis and a critical prerequisite step of computer assisted pathology and microscopy image analysis. However, due to the enormous variability (cell types, stains and different microscopes) and data complexity (cell overlapping, inhomogeneous intensities, background clutters and image artifacts), robust and accurate nucleus/cell detection is usually a difficult problem. To address this issue, we propose a novel multi-scale fully convolutional neural networks approach for regression of a density map to robustly detect the nuclei of pathology and microscopy images. The procedure can be divided into three main stages. Initially, instead of working on the simple dot label space, regression on the proposed structured proximity space for patches is performed so that centers of image patches are explicitly forced to produce larger values than their adjacent areas. Then, several multi-scale fully convolutional regression networks are developed for this task; this will enlarge the receptive field and not only can detect the single, small size cells, but also benefit to detecting cells with big size and overlapping states. In this stage, we copy the full feature maps from the contracting path and merge with the feature maps of the expansive path. This operation will make full use of shallow and deep semantic information of the networks. The networks do not have any fully connected layers; this strategy allows the seamless probability map prediction of arbitrarily large images. At the same time, data augmentations (e.g., small range shift, zoom and randomly flip) are carefully used to enhance the robustness of detection. Finally, morphological operations and suitable filters are employed and some prior information is introduced to find the centers of the cells more robustly. Our method achieves about 99.25% detection precision and the F1-measure is 0.9924 on fluorescence microscopy cell images; about 85.90% detection precision and the F1-measure is 0.9020 on Lymphocyte cell images and about 78.41% detection precision and the F1-measure is 0.8440 on breast histopathological images. This result leads to a promising detection performance that equates and sometimes exceeds the recently published leading detection approaches with the same benchmark datasets.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Figure 1
Figure 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15

Similar content being viewed by others

References

  1. Achanta, R., Shaji, A., Smith, K., Lucchi, A., Fua, P., SüSstrunk, S.: Slic superpixels compared to state-of-the-art superpixel methods. IEEE Trans Pattern Anal Mach Intell. 34(11), 2274 (2012)

    Article  Google Scholar 

  2. Andrew, J., Anant, M.: Deep learning for digital pathology image analysis: a comprehensive tutorial with selected use cases. Journal of Pathology Informatics. 7(1), 29 (2016)

    Article  Google Scholar 

  3. Arteta, C., Lempitsky, V., Noble, J. A., Zisserman, A.: Learning to detect cells using non-overlapping extremal regions. In: Medical Image Computing & Computer-assisted Intervention: Miccai International Conference on Medical Image Computing & Computer-assisted Intervention, 15, 348–356 (2012)

  4. Arteta, C., Lempitsky, V., Noble, J.A., Zisserman, A.: Learning to detect cells using non-overlapping extremal regions. In: International Conference on Medical Image Computing and Computer Assisted Intervention (Lecture Notes in Computer Science). MICCAI, 15, pp. 348–356 (2012)

    Google Scholar 

  5. Arteta, C., Lempitsky, V., Noble, J. A., & Zisserman, A.: Interactive object counting. In: Proceedings of the European Conference on Computer Vision (ECCV), 8691, 504–518 (2014)

  6. Boykov, Y., Kolmogorov, V.: An experimental comparison of min-cut/max-flow algorithms for energy minimization in vision. IEEE Trans Pattern Anal Mach Intell. 26(9), 1124 (2004)

    Article  Google Scholar 

  7. Cai, Z., Fan, Q., Feris, R. S., Vasconcelos, N.: A unified multi-scale deep convolutional neural network for fast object detection. In: Leibe B., Matas J., Sebe N., Welling M. (eds.) European Conference on Computer Vision. pp.354–370. Springer, Cham (2016)

    Chapter  Google Scholar 

  8. Cireşan, D. C., Giusti, A., Gambardella, L. M., Schmidhuber, J.: Mitosis detection in breast cancer histology images with deep neural networks. In: International Conference on Medical Image Computing & Computer-assisted Intervention, 16, 411–418 (2013)

  9. Cruz-Roa A.A., Arevalo Ovalle J.E., Madabhushi A., González Osorio F.A.: A deep learning architecture for image representation, visual interpretability and automated basal-cell carcinoma cancer detection. In: Mori K., Sakuma I., Sato Y., Barillot C., Navab N. (eds.) International Conference on Medical Image Computing and Computer-assisted Intervention (MICCAI), pp. 403–410. Springer, Berlin (2013)

    Chapter  Google Scholar 

  10. Dan, C.C., Giusti, A., Gambardella, L.M., Schmidhuber, J.: Deep neural networks segment neuronal membranes in electron microscopy images. Adv. Neural Inf. Proces. Syst. 25, 2852–2860 (2012)

    Google Scholar 

  11. Dong, B., Shao, L., Costa, M. D., Bandmann, O., Frangi, A. F.: Deep learning for automatic cell detection in wide-field microscopy zebrafish images. In: Proceedings of IEEE 12th International Symposium on Biomedical Imaging (ISBI), pp. 772–776. IEEE New York (2015)

  12. Dundar, M.M., Badve, S., Bilgin, G., Raykar, V., Jain, R., Sertel, O., et al.: Computerized classification of intraductal breast lesions using histopathological images. IEEE Trans. Biomed. Eng. 58(7), 1977–1984 (2011)

    Article  Google Scholar 

  13. Fatakdawala, H., Xu, J., Basavanhally, A., Bhanot, G., Ganesan, S., Feldman, M., et al.: Expectation-maximization-driven geodesic active contour with overlap resolution (emagacor): application to lymphocyte segmentation on breast cancer histopathology. IEEE Trans. Biomed. Eng. 57(7), 1676–1689 (2010)

    Article  Google Scholar 

  14. Fiaschi, L., Koethe, U., Nair, R., Hamprecht, F. A.: Learning to count with regression forest and structured labels. In: Proceedings of 21st International Conference on Pattern Recognition (ICPR), pp. 2685–2688. IEEE (2012)

  15. Foran, D.J., Yang, L., Chen, W., Hu, J., Goodell, L.A., Reiss, M., et al.: Imageminer: a software system for comparative analysis of tissue microarrays using content-based image retrieval, high-performance computing, and grid technology. J. Am. Med. Inform. Assoc. 18(4), 403–415 (2011)

    Article  Google Scholar 

  16. García-Gojo, M.: State of the art and trends for digital pathology. Stud Health Technol Inform. 179, 15–28 (2012)

    Google Scholar 

  17. Giusti, A., Dan, C.C., Masci, J., Gambardella, L.M., Schmidhuber, J.: Fast image scanning with deep max-pooling convolutional. Neural Netw. 4034–4038 (2013)

  18. Guan, B.: Cell segmentation: 50 years down the road. IEEE Signal Process. Mag. 29(5), 140–145 (2012)

    Article  Google Scholar 

  19. Hu, R., Zhu, X., Cheng, D., et al.: Graph self-representation method for unsupervised feature selection. Neurocomputing. 220, 130–137 (2017)

    Article  Google Scholar 

  20. Ioffe, S., Szegedy, C.: Batch normalization: accelerating deep network training by reducing internal covariate shift. International Conference on International Conference on Machine Learning. 448–456 (2015)

  21. Khoshdeli, M., Cong R., Parvin, B.: Detection of nuclei in H&E stained sections using convolutional neural networks. In: 2017 IEEE EMBS International Conference on Biomedical & Health Informatics (BHI), pp. 105–108. IEEE (2017). https://doi.org/10.1109/BHI.2017.7897216

  22. Kuse, M., Wang, Y.F., Kalasannavar, V., Khan, M., Rajpoot, N.: Local isotropic phase symmetry measure for detection of beta cells and lymphocytes. Journal of Pathology Informatics. 2(2), S2 (2011)

    Article  Google Scholar 

  23. Lehmussola, A., Ruusuvuori, P., Selinummi, J., Huttunen, H., Yli-Harja, O.: Computational framework for simulating fluorescence microscope images with cell populations. IEEE Trans. Med. Imaging. 26(7), 1010–1016 (2007)

    Article  Google Scholar 

  24. Lempitsky, V., Zisserman, A.: Learning to count objects in images. In: Advances in Neural Information Processing Systems (NIPS). 43, 1324–1332 (2010)

    Google Scholar 

  25. Li, X., & Plataniotis, K. N.: Color model comparative analysis for breast cancer diagnosis using h and e stained images. In: SPIE Medical Imaging International Society for Optics and Photonics, 9420, 94200L–94200L-6 (2015)

  26. Liu, F., Yang, L.: A novel cell detection method using deep convolutional neural network and maximum-weight independent set. In: Lu L., Zheng Y., Carneiro G., Yang L. (eds.) Deep Learning and Convolutional Neural Networks for Medical Image Computing, Advances in Computer Vision and Pattern Recognition, pp. 349-357. Springer, Cham (2015)

  27. Long, J., Shelhamer, E., Darrell, T.: Fully convolutional networks for semantic segmentation. IEEE Transactions on Pattern Analysis & Machine Intelligence. 39(4), 640–651 (2014)

    Google Scholar 

  28. López, C., Lejeune, M., Bosch, R., Korzynska, A., Garcíarojo, M., Salvadó, M.T., et al.: Digital image analysis in breast cancer: an example of an automated methodology and the effects of image compression. Stud Health Technol Inform. 179, 155–171 (2012)

    Google Scholar 

  29. Pan, X., Li, L., Yang, H., et al.: Accurate segmentation of nuclei in pathological images via sparse reconstruction and deep convolutional networks. Neurocomputing. 229, 88–99 (2017)

    Article  Google Scholar 

  30. Parvin, B., Yang, Q., Han, J., Chang, H., Rydberg, B., Barcelloshoff, M.H.: Iterative voting for inference of structural saliency and characterization of subcellular events. IEEE transactions on image processing a publication of the IEEE signal processing. Society. 16(3), 615–623 (2007)

    Google Scholar 

  31. Ren, W., Liu, S., Zhang, H., Pan, J., Cao, X., Yang, M. H.: Single Image Dehazing via Multi-scale Convolutional Neural Networks. In: European Conference on Computer Vision. pp. 154–169. Springer (2016)

  32. Ronneberger, O., Fischer, P., & Brox, T.: U-net: convolutional networks for biomedical image segmentation. In: International Conference on Medical Image Computing and Computer-Assisted Intervention, 9351, 234–241 (2015)

  33. Saxe, A. M., Mcclelland, J. L., Ganguli, S.: Exact solutions to the nonlinear dynamics of learning in deep linear neural networks. arXiv preprint arXiv:1312.6120. (2014)

  34. Sirinukunwattana, K., Shan, E. A. R., Tsang, Y. W., Snead, D., Cree, I., Rajpoot, N.: A Spatially Constrained Deep Learning Framework for Detection of Epithelial Tumor Nuclei in Cancer Histology Images. In: Wu G., Coupé P., Zhan Y., Munsell B., Rueckert D. (eds.) Patch-Based Techniques in Medical Imaging. Lecture Notes in Computer Science, vol. 9467, pp. 154–162. Springer, Cham (2015)

    Chapter  Google Scholar 

  35. Sirinukunwattana, K., Raza, S., Tsang, Y.W., Snead, D., Cree, I., Rajpoot, N.: Locality sensitive deep learning for detection and classification of nuclei in routine colon cancer histology images. IEEE Trans. Med. Imaging. 35(5), 1196–1206 (2016)

    Article  Google Scholar 

  36. Sommer, C., Hoefler, R., Samwer, M., et al.: A deep learning and novelty detection framework for rapid phenotyping in high-content screening. Mol. Biol. Cell. (2017). https://doi.org/10.1101/134627

  37. Song, Y., Zhang, L., Chen, S., Ni, D., Lei, B., Wang, T.: Accurate segmentation of cervical cytoplasm and nuclei based on multiscale convolutional network and graph partitioning. IEEE Trans. Biomed. Eng. 62(10), 2421–2433 (2015)

    Article  Google Scholar 

  38. Song TH., Sanchez V., EIDaly H., Rajpoot N.: Simultaneous cell detection and classification with an asymmetric deep autoencoder in bone marrow histology images. In: Valdés Hernández M. and González-Castro V. (eds.) Medical Image Understanding and Analysis. MIUA 2017. Communications in Computer and Information Science, vol 723. pp. 829–838. Springer, Cham (2017)

    Google Scholar 

  39. Song, T., Sanchez, V., Eidaly, H., Rajpoot, N.: Hybrid deep autoencoder with Curvature Gaussian for detection of various types of cells in bone marrow trephine biopsy images. In: 2017 IEEE International Symposium on Biomedical Imaging (ISBI), pp. 1040–1043. IEEE (2017)

  40. Su, H., Yin, Z., Kanade, T., & Huh, S.: Phase contrast image restoration via dictionary representation of diffraction patterns. In: International Conference on Medical Image Computing and Computer-assisted Intervention, Springer, 615–622 (2012)

  41. Su, H., Xing, F., Kong, X., Xie, Y., Zhang, S., Yang, L.: Robust cell detection and segmentation in histopathological images using sparse reconstruction and stacked denoising autoencoders. In: Navab N., Hornegger J., Wells W., Frangi A. (eds.) Medical Image Computing and Computer-Assisted Intervention (MICCAI), Lecture Notes in Computer Science, vol. 9351, pp. 383–390. Springer, Cham (2015)

    Google Scholar 

  42. Wang, H., Cruz-Roa, A., Basavanhally, A., et al.: Mitosis detection in breast cancer pathology images by combining handcrafted and convolutional neural network features. Journal of Medical Imaging, 1(3), 034003–1-8 (2014)

    Article  Google Scholar 

  43. Wei, S., Zhou, M., Yang, F., Yang, C., Tian, J.: Multi-scale Convolutional Neural Networks for Lung Nodule Classification. In: Ourselin S., Alexander D., Westin CF., Cardoso M. (eds.) Information Processing in Medical Imaging (IPMI), vol. 9123, pp. 588–599. Springer, Cham (2015)

  44. Xie, Y., Kong, X., Xing, F., Liu, F., Su, H., Yang, L.: Deep voting: a robust approach toward nucleus localization in microscopy images. In: Navab N., Hornegger J., Wells W., Frangi A. (eds.) Medical Image Computing and Computer-Assisted Intervention (MICCAI). Lecture Notes in Computer Science, vol 9351. pp. 374–382. Springer, Cham (2015)

    Google Scholar 

  45. Xie, Y., Xing, F., Kong, X., Su, H., Yang, L.: Beyond Classification: Structured Regression for Robust Cell Detection Using Convolutional Neural Network. In: International Conference on Medical Image Computing and Computer-Assisted Intervention, vol 9351. pp. 358–365. Springer, (2015)

  46. Xie, W., Noble, J. A., Zisserman, A.: Microscopy cell counting and detection with fully convolutional regression networks. Computer Methods in Biomechanics and Biomedical Engineering: Imaging & Visualization, pp. 1–10. Taylor & Francis, Oxfordshire (2016)

    Google Scholar 

  47. Xing, F., Yang, L.: Robust nucleus/cell detection and segmentation in digital pathology and microscopy images: a comprehensive review. IEEE Rev. Biomed. Eng. 9, 234 (2016)

    Article  Google Scholar 

  48. Xing, F., Xie, Y., Yang, L.: An automatic learning-based framework for robust nucleus segmentation. IEEE Trans. Med. Imaging. 35(2), 550–566 (2015)

    Article  Google Scholar 

  49. Xu, J., Xiang, L., Liu, Q., Gilmore, H., Wu, J., Tang, J., et al.: Stacked sparse autoencoder (ssae) for nuclei detection on breast cancer histopathology images. IEEE Trans. Med. Imaging. 35(1), 119 (2016)

    Article  Google Scholar 

  50. Yellin, F., Haeffele, B. D., Vidal, R.: Blood cell detection and counting in holographic lens-free imaging by convolutional sparse dictionary learning and coding. IEEE International Symposium on Biomedical Imaging, 650–653 (2017)

  51. Zhang, S., Metaxas, D.: Large-scale medical image analytics: recent methodologies, applications and future directions. Med. Image Anal. 33, 98–101 (2016)

    Article  Google Scholar 

  52. Zhang, X., Liu, W., Dundar, M., Badve, S., Zhang, S.: Towards large-scale histopathological image analysis: hashing-based image retrieval. IEEE Trans. Med. Imaging. 34(2), 496–506 (2015)

    Article  Google Scholar 

  53. Zhang, X., Xing, F., Su, H., Yang, L., Zhang, S.: High-throughput histopathological image analysis via robust cell segmentation and hashing. Med. Image Anal. 26(1), 306–315 (2015)

    Article  Google Scholar 

  54. Zhu, X., Zhang, L., Huang, Z.: A sparse embedding and least variance encoding approach to hashing. IEEE Trans. Image Process. 23(9), 3737–3750 (2014)

    Article  MathSciNet  Google Scholar 

  55. Zhu, X., Suk, H.I., Wang, L., et al.: A novel relational regularization feature selection method for joint regression and classification in AD diagnosis. Med. Image Anal. 75(6), 570–577 (2015)

    Google Scholar 

  56. Zhu, X., Li, X., Zhang, S.: Block-row sparse multiview multilabel learning for image classification. IEEE Transactions on Cybernetics. 46(2), 450–461 (2016)

    Article  Google Scholar 

  57. Zhu, X., Suk, H.I., Lee, S.W., et al.: Subspace regularized sparse multitask learning for multiclass neurodegenerative disease identification. IEEE Trans. Biomed. Eng. 63(3), 607–618 (2016)

    Article  Google Scholar 

  58. Zhu, X., Li, X., Zhang, S., et al.: Robust joint graph sparse coding for unsupervised spectral feature selection. IEEE Transactions on Neural Networks & Learning Systems. 28(6), 1263–1275 (2017)

    Article  MathSciNet  Google Scholar 

Download references

Acknowledgments

The authors would like to thank Lehmussola et al. [23], Dr. Andrew Janowczyk et al. [2], and Dr. Zhang et al. [52] for publishing the datasets. We are grateful for helpful comments from the anonymous reviewers and the associate editor. This research was supported in part by the National Natural Science Foundation of China (Grant Nos. 21365008 and 61562013), and Natural Science Foundation of Guangxi Province (No. 2017GXNSFDA198025).

Author information

Authors and Affiliations

  1. School of Automation, Beijing University of Posts and Telecommunications, Beijing, China

    Xipeng Pan, Lingqiao Li, Huihua Yang, Zhiwei Cao, Zhen Ma & Yiyi Chen

  2. College of Arts and Sciences, University of Washington – Seattle, Seattle, Washington, USA

    Dengxian Yang

  3. School of Computer Science and Information Security, Guilin University of Electronic Technology, Guilin, China

    Lingqiao Li, Zhenbing Liu & Huihua Yang

  4. School of Computing and Information Systems, University of Melbourne, Melbourne, Australia

    Yubei He

Authors
  1. Xipeng Pan
    View author publications

    You can also search for this author inPubMed Google Scholar

  2. Dengxian Yang
    View author publications

    You can also search for this author inPubMed Google Scholar

  3. Lingqiao Li
    View author publications

    You can also search for this author inPubMed Google Scholar

  4. Zhenbing Liu
    View author publications

    You can also search for this author inPubMed Google Scholar

  5. Huihua Yang
    View author publications

    You can also search for this author inPubMed Google Scholar

  6. Zhiwei Cao
    View author publications

    You can also search for this author inPubMed Google Scholar

  7. Yubei He
    View author publications

    You can also search for this author inPubMed Google Scholar

  8. Zhen Ma
    View author publications

    You can also search for this author inPubMed Google Scholar

  9. Yiyi Chen
    View author publications

    You can also search for this author inPubMed Google Scholar

Corresponding author

Correspondence to Huihua Yang.

Additional information

Guest Editors: Xiaofeng Zhu, Gerard Sanroma, Jilian Zhang, and Brent C. Munsell

This article belongs to the Topical Collection: Special Issue on Deep Mining Big Social Data

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Pan, X., Yang, D., Li, L. et al. Cell detection in pathology and microscopy images with multi-scale fully convolutional neural networks. World Wide Web 21, 1721–1743 (2018). https://doi.org/10.1007/s11280-017-0520-7

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11280-017-0520-7

Keywords