Skip to main content
Springer Nature Link
Log in

TAHIR: Transformer-Based Affine Histological Image Registration

  • Conference paper
  • First Online:
Pattern Recognition, Computer Vision, and Image Processing. ICPR 2022 International Workshops and Challenges (ICPR 2022)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 13644))

Included in the following conference series:

Abstract

In medical practice it is often necessary to jointly analyze differently stained histological sections. However, when slides are being prepared tissues are subjected to deformations and registration is highly required. Although the transformation between images is generally non-rigid, one of the most challenging subproblems is to calculate the initial affine transformation. Existing learning-based approaches adopt convolutional architectures that are usually limited in receptive field, while global context should be considered. Coupled with small datasets and unsupervised learning paradigm, this results in overfitting and adjusting the structure only locally. We introduce transformer-based affine histological image registration (TAHIR) approach. It successfully aggregates global information, requires no histological data to learn and is based on knowledge transfer from nature domain. The experiments show that TAHIR outperforms existing methods by a large margin on most-commonly used histological image registration benchmark in terms of target registration error, being more robust at the same time. The code is available at https://github.com/VladPyatov/ImgRegWithTransformers.

The work was supported by Russian Science Foundation grant 22-41-02002.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Abdelsamea, M.M., Zidan, U., Senousy, Z., Gaber, M.M., Rakha, E., Ilyas, M.: A survey on artificial intelligence in histopathology image analysis. Wiley Interdisc. Rev. Data Min. Knowl. Discovery, e1474 (2022)

    Google Scholar 

  2. Anoshina, N.A., Krylov, A.S., Sorokin, D.V.: Correlation-based 2D registration method for single particle cryo-EM images. In: Proceedings of the IEEE International Conference on Image Processing Theory, Tools and Applications (IPTA), pp. 1–6 (2017)

    Google Scholar 

  3. Arganda-Carreras, I., Fernandez-Gonzalez, R., Ortiz-de Solorzano, C.: Automatic registration of serial mammary gland sections. In: The 26th Annual International Conference of the IEEE Engineering in Medicine and Biology Society, vol. 1, pp. 1691–1694 (2004)

    Google Scholar 

  4. Balakrishnan, G., Zhao, A., Sabuncu, M.R., Guttag, J., Dalca, A.V.: VoxelMorph: a learning framework for deformable medical image registration. IEEE Trans. Med. Imaging 38(8), 1788–1800 (2019)

    Article  Google Scholar 

  5. Borovec, J., et al.: ANHIR: automatic non-rigid histological image registration challenge. IEEE Trans. Med. Imaging 39(10), 3042–3052 (2020)

    Article  Google Scholar 

  6. Carion, N., Massa, F., Synnaeve, G., Usunier, N., Kirillov, A., Zagoruyko, S.: End-to-end object detection with transformers. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, J.-M. (eds.) ECCV 2020. LNCS, vol. 12346, pp. 213–229. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-58452-8_13

    Chapter  Google Scholar 

  7. Ceritoglu, C., Wang, L., Selemon, L.D., Csernansky, J.G., Miller, M.I., Ratnanather, J.T.: Large deformation diffeomorphic metric mapping registration of reconstructed 3D histological section images and in vivo MR images. Frontiers Hum. Neurosci., 43 (2010)

    Google Scholar 

  8. Chee, E., Wu, Z.: AIRNet: self-supervised affine registration for 3D medical images using neural networks. arXiv preprint arXiv:1810.02583 (2018)

  9. Cooper, L., Sertel, O., Kong, J., Lozanski, G., Huang, K., Gurcan, M.: Feature-based registration of histopathology images with different stains: an application for computerized follicular lymphoma prognosis. Comput. Methods Programs Biomed. 96(3), 182–192 (2009)

    Article  Google Scholar 

  10. Dai, A., Chang, A.X., Savva, M., Halber, M., Funkhouser, T., Nießner, M.: ScanNet: richly-annotated 3D reconstructions of indoor scenes. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 5828–5839 (2017)

    Google Scholar 

  11. De Vos, B.D., Berendsen, F.F., Viergever, M.A., Sokooti, H., Staring, M., Išgum, I.: A deep learning framework for unsupervised affine and deformable image registration. Med. Image Anal. 52, 128–143 (2019)

    Article  Google Scholar 

  12. Fischler, M.A., Bolles, R.C.: Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography. Commun. ACM 24(6), 381–395 (1981)

    Article  MathSciNet  Google Scholar 

  13. Gupta, L., Klinkhammer, B.M., Boor, P., Merhof, D., Gadermayr, M.: Stain independent segmentation of whole slide images: a case study in renal histology. In: IEEE 15th International Symposium on Biomedical Imaging (ISBI 2018), pp. 1360–1364 (2018)

    Google Scholar 

  14. Khvostikov, A., Krylov, A., Mikhailov, I., Malkov, P., Danilova, N.: Tissue type recognition in whole slide histological images. CEUR Workshop Proc. 3027, 50 (2021). https://doi.org/10.20948/graphicon-2021-3027-496-507

  15. Li, X., Han, K., Li, S., Prisacariu, V.: Dual-resolution correspondence networks. Adv. Neural. Inf. Process. Syst. 33, 17346–17357 (2020)

    Google Scholar 

  16. Li, Z., Snavely, N.: MegaDepth: learning single-view depth prediction from internet photos. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2041–2050 (2018)

    Google Scholar 

  17. Liu, Z., et al.: Swin transformer: hierarchical vision transformer using shifted windows. In: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 10012–10022 (2021)

    Google Scholar 

  18. Lowe, D.G.: Distinctive image features from scale-invariant keypoints. Int. J. Comput. Vision 60(2), 91–110 (2004)

    Article  Google Scholar 

  19. Mikhailov, I., Khvostikov, A., Krylov, A.S., Malkov, P., Danilova, N., Oleynikova, N.: Development of CNN-based algorithm for automatic recognition of the layers of the wall of the stomach and colon. Virchows Arch. 479(Suppl 1), OFP-15-004 (2021). https://doi.org/10.1007/s00428-021-03157-8

  20. Obando, D.F.G., Frafjord, A., Øynebråten, I., Corthay, A., Olivo-Marin, J.C., Meas-Yedid, V.: Multi-staining registration of large histology images. In: 2017 IEEE 14th International Symposium on Biomedical Imaging (ISBI 2017), pp. 345–348 (2017)

    Google Scholar 

  21. Pichat, J., Iglesias, J.E., Yousry, T., Ourselin, S., Modat, M.: A survey of methods for 3D histology reconstruction. Med. Image Anal. 46, 73–105 (2018)

    Article  Google Scholar 

  22. Pyatov, V., Sorokin, D.: Affine registration of histological images using transformer-based feature matching. Pattern Recogn. Image Anal. 32(3), 626–630 (2022). https://doi.org/10.1134/S1054661822030324

    Article  Google Scholar 

  23. Rocco, I., Arandjelović, R., Sivic, J.: Efficient neighbourhood consensus networks via submanifold sparse convolutions. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, J.-M. (eds.) ECCV 2020. LNCS, vol. 12354, pp. 605–621. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-58545-7_35

    Chapter  Google Scholar 

  24. Sarlin, P.E., DeTone, D., Malisiewicz, T., Rabinovich, A.: SuperGlue: learning feature matching with graph neural networks. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 4938–4947 (2020)

    Google Scholar 

  25. Sorokin, D.V., Tektonidis, M., Rohr, K., Matula, P.: Non-rigid contour-based temporal registration of 2D cell nuclei images using the Navier equation. In: IEEE 11th International Symposium on Biomedical Imaging (ISBI 2014), pp. 746–749 (2014)

    Google Scholar 

  26. Sun, J., Shen, Z., Wang, Y., Bao, H., Zhou, X.: LoFTR: detector-free local feature matching with transformers. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 8922–8931 (2021)

    Google Scholar 

  27. Tang, S., Zhang, J., Zhu, S., Tan, P.: Quadtree attention for vision transformers. arXiv preprint arXiv:2201.02767 (2022)

  28. Vaswani, A., et al.: Attention is all you need. In: Advances in Neural Information Processing Systems, vol. 30 (2017)

    Google Scholar 

  29. Wodzinski, M., Müller, H.: Learning-based affine registration of histological images. In: International Workshop on Biomedical Image Registration, pp. 12–22 (2020)

    Google Scholar 

  30. Wodzinski, M., Müller, H.: DeepHistReg: unsupervised deep learning registration framework for differently stained histology samples. Comput. Methods Programs Biomed. 198, 105799 (2021)

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Laboratory of Mathematical Methods of Image Processing, Faculty of Computational Mathematics and Cybernetics, Lomonosov Moscow State University, Moscow, Russia

    Vladislav A. Pyatov & Dmitry V. Sorokin

Authors
  1. Vladislav A. Pyatov
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Dmitry V. Sorokin
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Dmitry V. Sorokin .

Editor information

Editors and Affiliations

  1. York University, Toronto, ON, Canada

    Jean-Jacques Rousseau

  2. Ontario Tech University, Oshawa, ON, Canada

    Bill Kapralos

Rights and permissions

Reprints and permissions

Copyright information

© 2023 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Pyatov, V.A., Sorokin, D.V. (2023). TAHIR: Transformer-Based Affine Histological Image Registration. In: Rousseau, JJ., Kapralos, B. (eds) Pattern Recognition, Computer Vision, and Image Processing. ICPR 2022 International Workshops and Challenges. ICPR 2022. Lecture Notes in Computer Science, vol 13644. Springer, Cham. https://doi.org/10.1007/978-3-031-37742-6_42

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-37742-6_42

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-37741-9

  • Online ISBN: 978-3-031-37742-6

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Societies and partnerships

Search

Navigation