Abstract
The assessment of HER2 expression is crucial in diagnosing breast cancer. Staining pathological tissues with immunohistochemistry (IHC) is a critically pivotal step in the assessment procedure, while it is expensive and time-consuming. Recently, generative models have emerged as a novel paradigm for virtual staining from hematoxylin-eosin (H&E) to IHC. Unlike traditional image translation tasks, virtual staining in IHC for HER2 scoring requires greater attention to regions like nuclei and stained membranes, informed by task-specific domain knowledge. Unfortunately, most existing virtual staining methods overlook this point. In this paper, we propose a novel generative adversarial network (GAN) based solution that incorporates specific knowledge of HER2 scoring, i.e., nuclei distribution and membrane staining intensity. We introduce a nuclei density estimator to learn the nuclei distribution and thus facilitate the cell alignment between the real and generated images by an auxiliary regularization branch. Moreover, another branch is tailored to focus on the stained membranes, ensuring a more consistent membrane staining intensity. We collect RegH2I, a dataset comprising 2592 pairs of registered H&E-IHC images and conduct extensive experiments to evaluate our approach, including H&E-to-IHC virtual staining on internal and external datasets, nuclei distribution and membrane staining intensity analysis, as well as downstream tasks for generated images. The results demonstrate that our method achieves superior performance than existing methods. Code and dataset are released at https://github.com/balball/TDKstain.
Q. Peng and W. Lin—Contributed equally.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Ahn, S., Woo, J.W., Lee, K., Park, S.Y.: Her2 status in breast cancer: changes in guidelines and complicating factors for interpretation. Journal of pathology and translational medicine 54(1), 34–44 (2020)
Bai, B., Yang, X., Li, Y., Zhang, Y., Pillar, N., Ozcan, A.: Deep learning-enabled virtual histological staining of biological samples. Light: Science & Applications 12(1), 57 (2023)
D’Alfonso, T.M., Liu, Y.F., Chen, Z., Chen, Y.B., Cimino-Mathews, A., Shin, S.J.: Sp3, a reliable alternative to herceptest in determining her-2/neu status in breast cancer patients. Journal of clinical pathology 66(5), 409–414 (2013)
De Cuyper, A., Van Den Eynde, M., Machiels, J.P.: Her2 as a predictive biomarker and treatment target in colorectal cancer. Clinical colorectal cancer 19(2), 65–72 (2020)
Ding, K., Ma, K., Wang, S., Simoncelli, E.P.: Image quality assessment: Unifying structure and texture similarity. IEEE Transactions on Pattern Analysis and Machine Intelligence 44, 2567–2581 (2020)
DoanNgan, B., Angus, D., Sung, L., et al.: Label-free virtual her2 immunohistochemical staining of breast tissue using deep learning. BME frontiers (2022)
He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 770–778 (2016)
Iqbal, N., Iqbal, N.S.: Human epidermal growth factor receptor 2 (her2) in cancers: Overexpression and therapeutic implications. Molecular Biology International 2014 (2014)
Isola, P., Zhu, J.Y., Zhou, T., Efros, A.A.: Image-to-image translation with conditional adversarial networks. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 1125–1134 (2017)
Karakas, C., Tyburski, H., Turner, B.M., Wang, X., Schiffhauer, L.M., Katerji, H., Hicks, D.G., Zhang, H.: Interobserver and interantibody reproducibility of her2 immunohistochemical scoring in an enriched her2-low–expressing breast cancer cohort. American Journal of Clinical Pathology 159(5), 484–491 (2023)
Li, F., Hu, Z., Chen, W., Kak, A.C.: Adaptive supervised patchnce loss for learning h &e-to-ihc stain translation with inconsistent groundtruth image pairs. In: International Conference on Medical Image Computing and Computer-Assisted Intervention (2023)
Li, J., Luo, H., Zhu, X., Zhao, J., Chen, T.: Designing dna cage-based immuno-fluorescence strategy for rapid diagnosis of clinical cervical cancer tissues. Chinese Chemical Letters 33(2), 788–792 (2022)
Liu, S., Zhu, C., Xu, F., Jia, X., Shi, Z., Jin, M.: Bci: Breast cancer immunohistochemical image generation through pyramid pix2pix. 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) pp. 1814–1823 (2022)
Liu, S., Zhang, B., Liu, Y., Han, A., Shi, H., Guan, T., He, Y.: Unpaired stain transfer using pathology-consistent constrained generative adversarial networks. IEEE Transactions on Medical Imaging 40, 1977–1989 (2021)
Mao, X., Li, Q., Xie, H., Lau, R.Y.K., Wang, Z., Smolley, S.P.: Least squares generative adversarial networks. 2017 IEEE International Conference on Computer Vision (ICCV) pp. 2813–2821 (2016)
Otsu, N.: A threshold selection method from gray-level histograms. IEEE Trans. Syst. Man Cybern. 9, 62–66 (1979)
Purdie, C.A., Jordan, L.B., McCullough, J.B., Edwards, S.L., Cunningham, J., Walsh, M., Grant, A., Pratt, N., Thompson, A.M.: Her2 assessment on core biopsy specimens using monoclonal antibody cb11 accurately determines her2 status in breast carcinoma. Histopathology 56(6), 702–707 (2010)
Rivenson, Y., de Haan, K., Wallace, W.D., Ozcan, A.: Emerging advances to transform histopathology using virtual staining. BME frontiers 2020 (2020)
Ruifrok, A.C., Johnston, D.A.: Quantification of histochemical staining by color deconvolution. Analytical and quantitative cytology and histology 23 4, 291–9 (2001)
Stringer, C., Wang, T., Michaelos, M., Pachitariu, M.: Cellpose: a generalist algorithm for cellular segmentation. Nature Methods 18, 100 – 106 (2020)
Swain, S.M., Shastry, M., Hamilton, E.: Targeting her2-positive breast cancer: Advances and future directions. Nature Reviews Drug Discovery 22(2), 101–126 (2023)
Wang, T.C., Liu, M.Y., Zhu, J.Y., Tao, A., Kautz, J., Catanzaro, B.: High-resolution image synthesis and semantic manipulation with conditional gans. 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition pp. 8798–8807 (2017)
Zeng, B., Lin, Y., Wang, Y., Chen, Y., Dong, J., Li, X., Zhang, Y.: Semi-supervised pr virtual staining for breast histopathological images. In: International Conference on Medical Image Computing and Computer-Assisted Intervention (2022)
Zhu, J.Y., Park, T., Isola, P., Efros, A.A.: Unpaired image-to-image translation using cycle-consistent adversarial networks. 2017 IEEE International Conference on Computer Vision (ICCV) pp. 2242–2251 (2017)
Acknowledgments
This work was supported by National Natural Science Foundation of China (Grant No. 62371409), the Research Grants Council of Hong Kong (T45-401/22-N and 27206123) and Hong Kong Innovation and Technology Fund (ITS/274/22).
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Ethics declarations
Disclosure of Interests
The authors have no competing interests to declare that are relevant to the content of this article.
Rights and permissions
Copyright information
© 2024 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Peng, Q. et al. (2024). Advancing H&E-to-IHC Virtual Staining with Task-Specific Domain Knowledge for HER2 Scoring. In: Linguraru, M.G., et al. Medical Image Computing and Computer Assisted Intervention – MICCAI 2024. MICCAI 2024. Lecture Notes in Computer Science, vol 15004. Springer, Cham. https://doi.org/10.1007/978-3-031-72083-3_1
Download citation
DOI: https://doi.org/10.1007/978-3-031-72083-3_1
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-72082-6
Online ISBN: 978-3-031-72083-3
eBook Packages: Computer ScienceComputer Science (R0)
