Skip to main content
Springer Nature Link
Log in

Symmetric-Constrained Irregular Structure Inpainting for Brain MRI Registration with Tumor Pathology

  • Conference paper
  • First Online:
Brainlesion: Glioma, Multiple Sclerosis, Stroke and Traumatic Brain Injuries (BrainLes 2020)

Part of the book series: Lecture Notes in Computer Science ((LNIP,volume 12658))

Included in the following conference series:

Abstract

Deformable registration of magnetic resonance images between patients with brain tumors and healthy subjects has been an important tool to specify tumor geometry through location alignment and facilitate pathological analysis. Since tumor region does not match with any ordinary brain tissue, it has been difficult to deformably register a patient’s brain to a normal one. Many patient images are associated with irregularly distributed lesions, resulting in further distortion of normal tissue structures and complicating registration’s similarity measure. In this work, we follow a multi-step context-aware image inpainting framework to generate synthetic tissue intensities in the tumor region. The coarse image-to-image translation is applied to make a rough inference of the missing parts. Then, a feature-level patch-match refinement module is applied to refine the details by modeling the semantic relevance between patch-wise features. A symmetry constraint reflecting a large degree of anatomical symmetry in the brain is further proposed to achieve better structure understanding. Deformable registration is applied between inpainted patient images and normal brains, and the resulting deformation field is eventually used to deform original patient data for the final alignment. The method was applied to the Multimodal Brain Tumor Segmentation (BraTS) 2018 challenge database and compared against three existing inpainting methods. The proposed method yielded results with increased peak signal-to-noise ratio, structural similarity index, inception score, and reduced L1 error, leading to successful patient-to-normal brain image registration.

X. Liu and F. Xing—Contribute Equally.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

Notes

  1. 1.

    In the inpainting community, the \(1\times 1\) patch (in a feature map) is a widely used concept. The output of F1 \(\in \mathbb {R}^{256\times 60\times 60}\), while the original image is \(240\times 240\times 1\); therefore a \(1\times 1\) area in a feature map is not considered as a pixel.

  2. 2.

    https://www.med.upenn.edu/sbia/brats2018/data.html.

References

  1. Avants, B.B., Tustison, N.J., Song, G., Cook, P.A., Klein, A., Gee, J.C.: A reproducible evaluation of ants similarity metric performance in brain image registration. Neuroimage 54(3), 2033–2044 (2011)

    Article  Google Scholar 

  2. Ballester, C., Bertalmio, M., Caselles, V., Sapiro, G., Verdera, J.: Filling-in by joint interpolation of vector fields and gray levels. IEEE Trans. Image Process. 10(8), 1200–1211 (2001)

    Article  MathSciNet  Google Scholar 

  3. Barnes, C., Shechtman, E., Finkelstein, A., Goldman, D.B.: Patchmatch: a randomized correspondence algorithm for structural image editing. ACM Trans. Graph. 28(3), 24-1 (2009)

    Article  Google Scholar 

  4. Bauer, S., Wiest, R., Nolte, L.P., Reyes, M.: A survey of MRI-based medical image analysis for brain tumor studies. Phys. Med. Biol. 58(13), R97 (2013)

    Article  Google Scholar 

  5. Bertalmio, M., Sapiro, G., Caselles, V., Ballester, C.: Image inpainting. In: Proceedings of the 27th Annual Conference on Computer Graphics and Interactive Techniques, pp. 417–424 (2000)

    Google Scholar 

  6. Chen, T.Q., Schmidt, M.: Fast patch-based style transfer of arbitrary style. arXiv preprint arXiv:1612.04337 (2016)

  7. Criminisi, A., Pérez, P., Toyama, K.: Region filling and object removal by exemplar-based image inpainting. IEEE Trans. Image Process. 13(9), 1200–1212 (2004)

    Article  Google Scholar 

  8. Cuadra, M.B., et al.: Atlas-based segmentation of pathological brains using a model of tumor growth. In: Dohi, T., Kikinis, R. (eds.) MICCAI 2002. LNCS, vol. 2488, pp. 380–387. Springer, Heidelberg (2002). https://doi.org/10.1007/3-540-45786-0_47

    Chapter  Google Scholar 

  9. Dawant, B., Hartmann, S., Pan, S., Gadamsetty, S.: Brain atlas deformation in the presence of small and large space-occupying tumors. Comput. Aided Surg. 7(1), 1–10 (2002)

    Article  Google Scholar 

  10. DeAngelis, L.M.: Brain tumors. N. Engl. J. Med. 344(2), 114–123 (2001)

    Article  Google Scholar 

  11. Dosovitskiy, A., Brox, T.: Generating images with perceptual similarity metrics based on deep networks. In: Advances in Neural Information Processing Systems, pp. 658–666 (2016)

    Google Scholar 

  12. Efros, A.A., Freeman, W.T.: Image quilting for texture synthesis and transfer. In: Proceedings of the 28th Annual Conference on Computer Graphics And Interactive Techniques, pp. 341–346 (2001)

    Google Scholar 

  13. Gatys, L.A., Ecker, A.S., Bethge, M.: Image style transfer using convolutional neural networks. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2414–2423. IEEE (2016)

    Google Scholar 

  14. Gooya, A., Biros, G., Davatzikos, C.: Deformable registration of glioma images using EM algorithm and diffusion reaction modeling. IEEE Trans. Med. Imaging 30(2), 375–390 (2010)

    Article  Google Scholar 

  15. Iizuka, S., Simo-Serra, E., Ishikawa, H.: Globally and locally consistent image completion. ACM Trans. Graph. (TOG) 36(4), 107 (2017)

    Article  Google Scholar 

  16. Iizuka, S., Simo-Serra, E., Ishikawa, H.: Globally and locally consistent image completion. ACM Trans. Graph. (ToG) 36(4), 1–14 (2017)

    Article  Google Scholar 

  17. Isola, P., Zhu, J.Y., Zhou, T., Efros, A.A.: Image-to-image translation with conditional adversarial networks. In: Proceedings of the IEEE Conference On Computer Vision and Pattern Recognition, pp. 1125–1134 (2017)

    Google Scholar 

  18. Johnson, J., Alahi, A., Fei-Fei, L.: Perceptual losses for real-time style transfer and super-resolution. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9906, pp. 694–711. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-46475-6_43

    Chapter  Google Scholar 

  19. Lamecker, H., Pennec, X.: Atlas to image-with-tumor registration based on demons and deformation inpainting (2010)

    Google Scholar 

  20. Liu, G., Reda, F.A., Shih, K.J., Wang, T.C., Tao, A., Catanzaro, B.: Image inpainting for irregular holes using partial convolutions. In: Proceedings of the European Conference on Computer Vision (ECCV), pp. 85–100 (2018)

    Google Scholar 

  21. Liu, H., Jiang, B., Xiao, Y., Yang, C.: Coherent semantic attention for image inpainting. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 4170–4179 (2019)

    Google Scholar 

  22. Liu, X., et al.: Permutation-invariant feature restructuring for correlation-aware image set-based recognition. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 4986–4996 (2019)

    Google Scholar 

  23. Liu, X., Kumar, B.V., Ge, Y., Yang, C., You, J., Jia, P.: Normalized face image generation with perceptron generative adversarial networks. In: 2018 IEEE 4th International Conference on Identity, Security, and Behavior Analysis (ISBA), pp. 1–8. IEEE (2018)

    Google Scholar 

  24. Liu, X., et al.: Feature-level Frankenstein: eliminating variations for discriminative recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 637–646 (2019)

    Google Scholar 

  25. Marcos, D., Volpi, M., Tuia, D.: Learning rotation invariant convolutional filters for texture classification. In: ICPR (2016)

    Google Scholar 

  26. Marcus, D.S., Wang, T.H., Parker, J., Csernansky, J.G., Morris, J.C., Buckner, R.L.: Open access series of imaging studies (OASIS): cross-sectional MRI data in young, middle aged, nondemented, and demented older adults. J. Cogn. Neurosci. 19(9), 1498–1507 (2007)

    Article  Google Scholar 

  27. Mohamed, A., Zacharaki, E.I., Shen, D., Davatzikos, C.: Deformable registration of brain tumor images via a statistical model of tumor-induced deformation. Med. Image Anal. 10(5), 752–763 (2006)

    Article  Google Scholar 

  28. Oishi, K., Faria, A.V., Van Zijl, P.C., Mori, S.: MRI Atlas of Human White Matter. Academic Press (2010)

    Google Scholar 

  29. Oostenveld, R., Stegeman, D.F., Praamstra, P., van Oosterom, A.: Brain symmetry and topographic analysis of lateralized event-related potentials. Clin. Neurophysiol. 114(7), 1194–1202 (2003)

    Article  Google Scholar 

  30. Paszke, A., et al.: Automatic differentiation in pytorch (2017)

    Google Scholar 

  31. Pathak, D., Krahenbuhl, P., Donahue, J., Darrell, T., Efros, A.A.: Context encoders: feature learning by inpainting. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2536–2544 (2016)

    Google Scholar 

  32. Prados, F., et al.: Fully automated patch-based image restoration: application to pathology inpainting. In: Crimi, A., Menze, B., Maier, O., Reyes, M., Winzeck, S., Handels, H. (eds.) BrainLes 2016. LNCS, vol. 10154, pp. 3–15. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-55524-9_1

    Chapter  Google Scholar 

  33. Raina, K., Yahorau, U., Schmah, T.: Exploiting bilateral symmetry in brain lesion segmentation. arXiv preprint arXiv:1907.08196 (2019)

  34. Salimans, T., Goodfellow, I., Zaremba, W., Cheung, V., Radford, A., Chen, X.: Improved techniques for training GANs. In: Advances in Neural Information Processing Systems, pp. 2234–2242 (2016)

    Google Scholar 

  35. Sartor, K.: MR imaging of the brain: tumors. Eur. Radiol. 9(6), 1047–1054 (1999)

    Article  Google Scholar 

  36. Song, Y., et al.: Contextual-based image inpainting: infer, match, and translate. In: Proceedings of the European Conference on Computer Vision (ECCV), pp. 3–19 (2018)

    Google Scholar 

  37. Sotiras, A., Davatzikos, C., Paragios, N.: Deformable medical image registration: a survey. IEEE Trans. Med. Imaging 32(7), 1153–1190 (2013)

    Article  Google Scholar 

  38. Tang, Z., Wu, Y., Fan, Y.: Groupwise registration of MR brain images with tumors. Phys. Med. Biol. 62(17), 6853 (2017)

    Article  Google Scholar 

  39. Yang, C., Song, Y., Liu, X., Tang, Q., Kuo, C.C.J.: Image inpainting using block-wise procedural training with annealed adversarial counterpart. arXiv preprint arXiv:1803.08943 (2018)

  40. Zacharaki, E.I., Shen, D., Lee, S.K., Davatzikos, C.: Orbit: a multiresolution framework for deformable registration of brain tumor images. IEEE Trans. Med. Imaging 27(8), 1003–1017 (2008)

    Article  Google Scholar 

  41. Zhang, R., Isola, P., Efros, A.A., Shechtman, E., Wang, O.: The unreasonable effectiveness of deep features as a perceptual metric. arXiv preprint arXiv:1801.03924 (2018)

  42. Zheng, S., Song, Y., Leung, T., Goodfellow, I.: Improving the robustness of deep neural networks via stability training. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4480–4488 (2016)

    Google Scholar 

Download references

Acknowledgements

This work was supported by NIH R01DE027989, R01DC018511, R01AG061445, and P41EB022544.

Author information

Authors and Affiliations

  1. Gordon Center for Medical Imaging, Department of Radiology, Massachusetts General Hospital and Harvard Medical School, Boston, MA, 02114, USA

    Xiaofeng Liu, Fangxu Xing, Georges El Fakhri & Jonghye Woo

  2. Facebook Artificial Intelligence, Boston, MA, 02142, USA

    Chao Yang

  3. Ming Hsieh Department of Electrical and Computer Engineering, University of Southern California, Los Angeles, CA, 90007, USA

    C.-C. Jay Kuo

Authors
  1. Xiaofeng Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Fangxu Xing
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Chao Yang
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. C.-C. Jay Kuo
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Georges El Fakhri
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Jonghye Woo
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Xiaofeng Liu .

Editor information

Editors and Affiliations

  1. University of Zurich, Zurich, Switzerland

    Alessandro Crimi

  2. University of Pennsylvania, Philadelphia, PA, USA

    Spyridon Bakas

Rights and permissions

Reprints and permissions

Copyright information

© 2021 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Liu, X., Xing, F., Yang, C., Kuo, CC.J., El Fakhri, G., Woo, J. (2021). Symmetric-Constrained Irregular Structure Inpainting for Brain MRI Registration with Tumor Pathology. In: Crimi, A., Bakas, S. (eds) Brainlesion: Glioma, Multiple Sclerosis, Stroke and Traumatic Brain Injuries. BrainLes 2020. Lecture Notes in Computer Science(), vol 12658. Springer, Cham. https://doi.org/10.1007/978-3-030-72084-1_8

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-72084-1_8

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-72083-4

  • Online ISBN: 978-3-030-72084-1

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Societies and partnerships