Skip to main content

Multimodal CT and MR Segmentation of Head and Neck Organs-at-Risk

  • Conference paper
  • First Online:
Medical Image Computing and Computer Assisted Intervention – MICCAI 2023 (MICCAI 2023)

Abstract

Radiotherapy (RT) is a standard treatment modality for head and neck (HaN) cancer that requires accurate segmentation of target volumes and nearby healthy organs-at-risk (OARs) to optimize radiation dose distribution. However, computed tomography (CT) imaging has low image contrast for soft tissues, making accurate segmentation of soft tissue OARs challenging. Therefore, magnetic resonance (MR) imaging has been recommended to enhance the segmentation of soft tissue OARs in the HaN region. Based on our two empirical observations that deformable registration of CT and MR images of the same patient is inherently imperfect and that concatenating such images at the input layer of a deep learning network cannot optimally exploit the information provided by the MR modality, we propose a novel modality fusion module (MFM) that learns to spatially align MR-based feature maps before fusing them with CT-based feature maps. The proposed MFM can be easily implemented into any existing multimodal backbone network. Our implementation within the nnU-Net framework shows promising results on a dataset of CT and MR image pairs from the same patients. Furthermore, the evaluation on a clinically realistic scenario with the missing MR modality shows that MFM outperforms other state-of-the-art multimodal approaches.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

Notes

  1. 1.

    https://doi.org/10.5281/zenodo.7442914.

  2. 2.

    https://hanseg2023.grand-challenge.org.

References

  1. Brouwer, C.L., Steenbakkers, R.J., Bourhis, J., et al.: CT-based delineation of organs at risk in the head and neck region: DAHANCA, EORTC, GORTEC, HKNPCSG, NCIC CTG, NCRI, NRG oncology and TROG consensus guidelines. Radiother. Oncol. 117, 83–90 (2015). https://doi.org/10.1016/j.radonc.2015.07.041

    Article  Google Scholar 

  2. Chen, J., Zhan, Y., Xu, Y., Pan, X.: FAFNet: fully aligned fusion network for RGBD semantic segmentation based on hierarchical semantic flows. IET Image Process 17, 32–41 (2023). https://doi.org/10.1049/ipr2.12614

    Article  Google Scholar 

  3. Chow, L.Q.M.: Head and neck cancer. N. Engl. J. Med. 382, 60–72 (2020). https://doi.org/10.1056/NEJMRA1715715

    Article  Google Scholar 

  4. Dolz, J., Gopinath, K., Yuan, J., Lombaert, H., Desrosiers, C., Ben Ayed, I.: HyperDense-Net: a hyper-densely connected CNN for multi-modal image segmentation. IEEE Trans. Med. Imaging 38, 1116–1126 (2019). https://doi.org/10.1109/TMI.2018.2878669

    Article  Google Scholar 

  5. Dou, Q., Liu, Q., Heng, P.A., Glocker, B.: Unpaired multi-modal segmentation via knowledge distillation. IEEE Trans. Med. Imaging 39, 2415–2425 (2020). https://doi.org/10.1109/TMI.2019.2963882

    Article  Google Scholar 

  6. Finnveden, L., Jansson, Y., Lindeberg, T.: Understanding when spatial transformer networks do not support invariance, and what to do about it. In: 25th International Conference on Pattern Recognition - ICPR 2020, Milan, Italy, pp. 3427–3434. IEEE (2020). https://doi.org/10.1109/ICPR48806.2021.9412997

  7. Hu, X., Yang, K., Fei, L., Wang, K.: ACNet: attention based network to exploit complementary features for RGBD semantic segmentation. In: 26th International Conference on Image Processing - ICIP 2019, Taipei, Taiwan, pp. 1440–1444. IEEE (2019). https://doi.org/10.1109/ICIP.2019.8803025

  8. Isensee, F., Jaeger, P.F., Kohl, S.A., Petersen, J., Maier-Hein, K.H.: nnU-Net: a self-configuring method for deep learning-based biomedical image segmentation. Nat. Methods 18, 203–211 (2021). https://doi.org/10.1038/s41592-020-01008-z

    Article  Google Scholar 

  9. Jaderberg, M., Simonyan, K., Zisserman, A., Kavukcuoglu, K.: Spatial transformer networks. In: Advances in Neural Information Processing Systems - NIPS 2015, vol. 28. Curran Associates, Montréal, QC, Canada (2015). https://doi.org/10.48550/arxiv.1506.02025

  10. Jiang, J., Rimner, A., Deasy, J.O., Veeraraghavan, H.: Unpaired cross-modality educed distillation (CMEDL) for medical image segmentation. IEEE Trans. Med. Imaging 41, 1057–1068 (2022). https://doi.org/10.1109/TMI.2021.3132291

    Article  Google Scholar 

  11. Maier-Hein, L., Reinke, A., Kozubek, M., et al.: BIAS: transparent reporting of biomedical image analysis challenges. Med. Image Anal. 66, 101796 (2020). https://doi.org/10.1016/j.media.2020.101796

    Article  Google Scholar 

  12. Ngiam, J., Khosla, A., Kim, M., Nam, J., Lee, H., Ng, A.Y.: Multimodal deep learning. In: 28th International Conference on International Conference on Machine Learning - ICML 2011, Bellevue, WA, USA, pp. 689–696. Omnipress (2011)

    Google Scholar 

  13. Nikolov, S., Blackwell, S., Zverovitch, A., et al.: Clinically applicable segmentation of head and neck anatomy for radiotherapy: deep learning algorithm development and validation study. J. Med. Internet Res. 23, e26151 (2021). https://doi.org/10.2196/26151

    Article  Google Scholar 

  14. Podobnik, G., Strojan, P., Peterlin, P., Ibragimov, B., Vrtovec, T.: HaN-Seg: the head and neck organ-at-risk CT and MR segmentation dataset. Med. Phys. 50, 1917–1927 (2023). https://doi.org/10.1002/mp.16197

    Article  Google Scholar 

  15. Raudaschl, P.F., Zaffino, P., Sharp, G.C., et al.: Evaluation of segmentation methods on head and neck CT: auto-segmentation challenge 2015. Med. Phys. 44, 2020–2036 (2017). https://doi.org/10.1002/mp.12197

    Article  Google Scholar 

  16. Ronneberger, O., Fischer, P., Brox, T.: U-Net: convolutional networks for biomedical image segmentation. In: Navab, N., Hornegger, J., Wells, W.M., Frangi, A.F. (eds.) MICCAI 2015. LNCS, vol. 9351, pp. 234–241. Springer, Cham (2015). https://doi.org/10.1007/978-3-319-24574-4_28

    Chapter  Google Scholar 

  17. Sung, H., Ferlay, J., Siegel, R.L., et al.: Global cancer statistics 2020: GLOBOCAN estimates of incidence and mortality worldwide for 36 cancers in 185 countries. CA Cancer J. Clin. 71, 209–249 (2021). https://doi.org/10.3322/caac.21660

    Article  Google Scholar 

  18. Valada, A., Mohan, R., Burgard, W.: Self-supervised model adaptation for multimodal semantic segmentation. Int. J. Comput. Vis. 128, 1239–1285 (2018). https://doi.org/10.1007/s11263-019-01188-y

  19. Valindria, V.V., Pawlowski, N., Rajchl, M., et al.: Multi-modal learning from unpaired images: application to multi-organ segmentation in CT and MRI. In: 2018 IEEE Winter Conference on Applications of Computer Vision - WACV 2018, Lake Tahoe, NV, USA, pp. 547–556. IEEE (2018). https://doi.org/10.1109/WACV.2018.00066

  20. Valverde, F.R., Hurtado, J.V., Valada, A.: There is more than meets the eye: self-supervised multi-object detection and tracking with sound by distilling multimodal knowledge. In: 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition - CVPR 2021, Nashville, TN, USA, pp. 11607–11616. IEEE (2021). https://doi.org/10.1109/CVPR46437.2021.01144

  21. Yan, F., Knochelmann, H.M., Morgan, P.F., et al.: The evolution of care of cancers of the head and neck region: state of the science in 2020. Cancers 12, 1543 (2020). https://doi.org/10.3390/cancers12061543

    Article  Google Scholar 

  22. Yan, Y., et al.: Longitudinal detection of diabetic retinopathy early severity grade changes using deep learning. In: Fu, H., Garvin, M.K., MacGillivray, T., Xu, Y., Zheng, Y. (eds.) OMIA 2021. LNCS, vol. 12970, pp. 11–20. Springer, Cham (2021). https://doi.org/10.1007/978-3-030-87000-3_2

    Chapter  Google Scholar 

  23. Zhang, Y., et al.: Modality-aware mutual learning for multi-modal medical image segmentation. In: de Bruijne, M., et al. (eds.) MICCAI 2021. LNCS, vol. 12901, pp. 589–599. Springer, Cham (2021). https://doi.org/10.1007/978-3-030-87193-2_56

    Chapter  Google Scholar 

  24. Zhang, Y., Sidibé, D., Morel, O., Mériaudeau, F.: Deep multimodal fusion for semantic image segmentation: a survey. Image Vis. Comput. 105, 104042 (2021). https://doi.org/10.1016/j.imavis.2020.104042

    Article  Google Scholar 

  25. Zhou, T., Ruan, S., Canu, S.: A review: deep learning for medical image segmentation using multi-modality fusion. Array 3–4, 100004 (2019). https://doi.org/10.1016/j.array.2019.100004

    Article  Google Scholar 

Download references

Acknowledgments

This work was supported by the Slovenian Research Agency (ARRS) under grants J2-1732, P2-0232 and P3-0307, and partially by the Novo Nordisk Foundation under grant NFF20OC0062056.

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Gašper Podobnik .

Editor information

Editors and Affiliations

1 Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary material 1 (pdf 125 KB)

Rights and permissions

Reprints and permissions

Copyright information

© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Podobnik, G., Strojan, P., Peterlin, P., Ibragimov, B., Vrtovec, T. (2023). Multimodal CT and MR Segmentation of Head and Neck Organs-at-Risk. In: Greenspan, H., et al. Medical Image Computing and Computer Assisted Intervention – MICCAI 2023. MICCAI 2023. Lecture Notes in Computer Science, vol 14223. Springer, Cham. https://doi.org/10.1007/978-3-031-43901-8_71

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-43901-8_71

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-43900-1

  • Online ISBN: 978-3-031-43901-8

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics