Abstract
This work presents PathAttFormer, a deep learning model that predicts the visual attention of pathologists viewing whole slide images (WSIs) while evaluating cancer. This model has two main components: (1) a patch-wise attention prediction module using a Swin transformer backbone and (2) a self-attention based attention refinement module to compute pairwise-similarity between patches to predict spatially consistent attention heatmaps. We observed a high level of agreement between model predictions and actual viewing behavior, collected by capturing panning and zooming movements using a digital microscope interface. Visual attention was analyzed in the evaluation of prostate cancer and gastrointestinal neuroendocrine tumors (GI-NETs), which differ greatly in terms of diagnostic paradigms and the demands on attention. Prostate cancer involves examining WSIs stained with Hematoxylin and Eosin (H &E) to identify distinct growth patterns for Gleason grading. In contrast, GI-NETs require a multi-step approach of identifying tumor regions in H &E WSIs and grading by quantifying the number of Ki-67 positive tumor cells highlighted with immunohistochemistry (IHC) in a separate image. We collected attention data from pathologists viewing prostate cancer H &E WSIs from The Cancer Genome Atlas (TCGA) and 21 H &E WSIs of GI-NETs with corresponding Ki-67 IHC WSIs. This is the first work that utilizes the Swin transformer architecture to predict visual attention in histopathology images of GI-NETs, which is generalizable to predicting attention in the evaluation of multiple sequential images in real world diagnostic pathology and IHC applications.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Govind, D., et al.: Improving the accuracy of gastrointestinal neuroendocrine tumor grading with deep learning. Sci. Rep. 10(1), 1–12 (2020)
Matsukuma, K., Olson, K.A., Gui, D., Gandour-Edwards, R., Li, Y., Beckett, L.: Synaptophysin-Ki-67 double stain: a novel technique that improves interobserver agreement in the grading of well-differentiated gastrointestinal neuroendocrine tumors. Mod. Pathol. 30(4), 620–629 (2017)
Hashimoto, N., et al.: Multi-scale domain-adversarial multiple-instance CNN for cancer subtype classification with unannotated histopathological images. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3852–3861 (2020)
Chakraborty, S., et al.: Visual attention analysis of pathologists examining whole slide images of Prostate cancer. In: 2022 IEEE 19th International Symposium on Biomedical Imaging (ISBI), pp. 1–5. IEEE, March 2022
Brunyé, T.T., Drew, T., Kerr, K.F., Shucard, H., Weaver, D.L., Elmore, J.G.: Eye tracking reveals expertise-related differences in the time-course of medical image inspection and diagnosis. J. Med. Imaging 7(5), 051203 (2020)
Sudin, E., et al.: Eye tracking in digital pathology: identifying expert and novice patterns in visual search behaviour. In: Medical Imaging 2021: Digital Pathology, vol. 11603, p. 116030Z. International Society for Optics and Photonics, February 2021
Brunyé, T.T., Mercan, E., Weaver, D.L., Elmore, J.G.: Accuracy is in the eyes of the pathologist: the visual interpretive process and diagnostic accuracy with digital whole slide images. J. Biomed. Inform. 66, 171–179 (2017)
Bombari, D., Mora, B., Schaefer, S.C., Mast, F.W., Lehr, H.A.: What was I thinking? Eye-tracking experiments underscore the bias that architecture exerts on nuclear grading in prostate cancer. PLoS ONE 7(5), e38023 (2012)
Raghunath, V., et al.: Mouse cursor movement and eye tracking data as an indicator of pathologists’ attention when viewing digital whole slide images. J. Pathol. Inform. 3, 43 (2012)
Mercan, E., Shapiro, L.G., Brunyé, T.T., Weaver, D.L., Elmore, J.G.: Characterizing diagnostic search patterns in digital breast pathology: scanners and drillers. J. Digit. Imaging 31(1), 32–41 (2018)
Saltz, J., et al.: A containerized software system for generation, management, and exploration of features from whole slide tissue images. Can. Res. 77(21), e79–e82 (2017)
Govind, D., et al.: Improving the accuracy of gastrointestinal neuroendocrine tumor grading with deep learning. Sci. Rep. 10(1), 1–12 (2020)
Liu, Z., et al.: Swin transformer: hierarchical vision transformer using shifted windows. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 10012–10022 (2021)
He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 770–778 (2016)
Loshchilov, I., Hutter, F.: Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101 (2017)
Tellez, D., et al.: Whole-slide mitosis detection in H &E breast histology using PHH3 as a reference to train distilled stain-invariant convolutional networks. IEEE Trans. Med. Imaging 37(9), 2126–2136 (2018)
Krähenbühl, P., Koltun, V.: Efficient inference in fully connected CRFS with gaussian edge potentials. In: Advances in Neural Information Processing Systems, vol. 24 (2011)
Bylinskii, Z., Judd, T., Oliva, A., Torralba, A., Durand, F.: What do different evaluation metrics tell us about saliency models? IEEE Trans. Pattern Anal. Mach. Intell. 41(3), 740–757 (2018)
Kümmerer, M., Theis, L., Bethge, M.: Deep gaze I: boosting saliency prediction with feature maps trained on imagenet. arXiv preprint arXiv:1411.1045 (2014)
Gonzales, R.C., Fittes, B.A.: Gray-level transformations for interactive image enhancement. Mech. Mach. Theory 12(1), 111–122 (1977)
Peacock, C.E., Hayes, T.R., Henderson, J.M.: Center bias does not account for the advantage of meaning over salience in attentional guidance during scene viewing. Front. Psychol. 11, 1877 (2020)
Dosovitskiy, A., et al.: An image is worth 16x16 words: transformers for image recognition at scale. arXiv preprint arXiv:2010.11929 (2020)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
1 Electronic supplementary material
Below is the link to the electronic supplementary material.
Rights and permissions
Copyright information
© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Chakraborty, S. et al. (2022). Predicting the Visual Attention of Pathologists Evaluating Whole Slide Images of Cancer. In: Huo, Y., Millis, B.A., Zhou, Y., Wang, X., Harrison, A.P., Xu, Z. (eds) Medical Optical Imaging and Virtual Microscopy Image Analysis. MOVI 2022. Lecture Notes in Computer Science, vol 13578. Springer, Cham. https://doi.org/10.1007/978-3-031-16961-8_2
Download citation
DOI: https://doi.org/10.1007/978-3-031-16961-8_2
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-16960-1
Online ISBN: 978-3-031-16961-8
eBook Packages: Computer ScienceComputer Science (R0)
