Abstract
With the emergence of multimodal electronic health records, the evidence for an outcome may be captured across multiple modalities ranging from clinical to imaging and genomic data. Predicting outcomes effectively requires fusion frameworks capable of modeling fine-grained and multi-faceted complex interactions between modality features within and across patients. We develop an innovative fusion approach called MaxCorr MGNN that models non-linear modality correlations within and across patients through Hirschfeld-Gebelein-Reǹyi maximal correlation (MaxCorr) embeddings, resulting in a multi-layered graph that preserves the identities of the modalities and patients. We then design, for the first time, a generalized multi-layered graph neural network (MGNN) for task-informed reasoning in multi-layered graphs, that learns the parameters defining patient-modality graph connectivity and message passing in an end-to-end fashion. We evaluate our model an outcome prediction task on a Tuberculosis (TB) dataset consistently outperforming several state-of-the-art neural, graph-based and traditional fusion techniques.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Baltrušaitis, T., Ahuja, C., Morency, L.P.: Multimodal machine learning: a survey and taxonomy. IEEE Trans. Pattern Anal. Mach. Intell. 41(2), 423–443 (2018)
Cosmo, L., Kazi, A., Ahmadi, S.-A., Navab, N., Bronstein, M.: Latent-graph learning for disease prediction. In: Martel, A.L., et al. (eds.) MICCAI 2020. LNCS, vol. 12262, pp. 643–653. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-59713-9_62
Cozzo, E., de Arruda, G.F., Rodrigues, F.A., Moreno, Y.: Multiplex networks (2018). https://link.springer.com/10.1007/978-3-319-92255-3
DeLong, E.R., DeLong, D.M., Clarke-Pearson, D.L.: Comparing the areas under two or more correlated receiver operating characteristic curves: a nonparametric approach. Biometrics, 837–845 (1988)
Dsouza, N.S., Nebel, M.B., Crocetti, D., Robinson, J., Mostofsky, S., Venkataraman, A.: M-GCN: a multimodal graph convolutional network to integrate functional and structural connectomics data to predict multidimensional phenotypic characterizations. In: Medical Imaging with Deep Learning, pp. 119–130. PMLR (2021)
D’Souza, N.S., et al.: Fusing modalities by multiplexed graph neural networks for outcome prediction in tuberculosis. In: Wang, L., Dou, Q., Fletcher, P.T., Speidel, S., Li, S. (eds.) Medical Image Computing and Computer Assisted Intervention-MICCAI 2022: 25th International Conference, Singapore, 18–22 September 2022, Proceedings, Part VII, pp. 287–297. Springer, Cham (2022). https://doi.org/10.1007/978-3-031-16449-1_28
D’Souza, N.S., Nebel, M.B., Crocetti, D., Robinson, J., Mostofsky, S., Venkataraman, A.: A matrix autoencoder framework to align the functional and structural connectivity manifolds as guided by behavioral phenotypes. In: de Bruijne, M., et al. (eds.) MICCAI 2021. LNCS, vol. 12907, pp. 625–636. Springer, Cham (2021). https://doi.org/10.1007/978-3-030-87234-2_59
D’Souza, N.S., et al.: Deep sr-DDL: deep structurally regularized dynamic dictionary learning to integrate multimodal and dynamic functional connectomics data for multidimensional clinical characterizations. Neuroimage 241, 118388 (2021)
D’Souza, N.S., et al.: A deep-generative hybrid model to integrate multimodal and dynamic connectivity for predicting spectrum-level deficits in autism. In: Martel, A.L., et al. (eds.) MICCAI 2020. LNCS, vol. 12267, pp. 437–447. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-59728-3_43
Gabrielian, A., et al.: TB DEPOT (data exploration portal): a multi-domain tuberculosis data analysis resource. PLOS ONE 14(5), e0217410 (2019). https://doi.org/10.1371/journal.pone.0217410
Huang, G., Liu, Z., Van Der Maaten, L., Weinberger, K.Q.: Densely connected convolutional networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4700–4708 (2017)
Jones, P., et al.: InterProScan 5: genome-scale protein function classification. Bioinf. (Oxford, England) 30(9), 1236–1240 (2014). https://doi.org/10.1093/bioinformatics/btu031
Kipf, T.N., Welling, M.: Semi-supervised classification with graph convolutional networks. arXiv preprint arXiv:1609.02907 (2016)
Lahat, D., Adali, T., Jutten, C.: Multimodal data fusion: an overview of methods, challenges, and prospects. Proc. IEEE 103(9), 1449–1477 (2015)
Loshchilov, I., Hutter, F.: Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101 (2017)
Muñoz-Sellart, M., Cuevas, L., Tumato, M., Merid, Y., Yassin, M.: Factors associated with poor tuberculosis treatment outcome in the southern region of Ethiopia. Int. J. Tuberc. Lung Dis. 14(8), 973–979 (2010)
Seabolt, E.E., et al.: OMXWare, a cloud-based platform for studying microbial life at scale, November 2019. https://arxiv.org/abs/1911.02095
Subramanian, V., Do, M.N., Syeda-Mahmood, T.: Multimodal fusion of imaging and genomics for lung cancer recurrence prediction. In: 2020 IEEE 17th International Symposium on Biomedical Imaging (ISBI), pp. 804–808. IEEE (2020)
Subramanian, V., Syeda-Mahmood, T., Do, M.N.: Multi-modality fusion using canonical correlation analysis methods: application in breast cancer survival prediction from histology and genomics. arXiv preprint arXiv:2111.13987 (2021)
Veličković, P., Cucurull, G., Casanova, A., Romero, A., Lio, P., Bengio, Y.: Graph attention networks. arXiv preprint arXiv:1710.10903 (2017)
Wang, H., Subramanian, V., Syeda-Mahmood, T.: Modeling uncertainty in multi-modal fusion for lung cancer survival analysis. In: 2021 IEEE 18th International Symposium on Biomedical Imaging (ISBI), pp. 1169–1172. IEEE (2021)
Wang, H., Yushkevich, P.: Multi-atlas segmentation with joint label fusion and corrective learning-an open source implementation. Front. Neuroinform. 7, 27 (2013)
Wang, L., et al.: An efficient approach to informative feature extraction from multimodal data. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 33, pp. 5281–5288 (2019)
Xu, K., Hu, W., Leskovec, J., Jegelka, S.: How powerful are graph neural networks? arXiv preprint arXiv:1810.00826 (2018)
Yang, X., Liu, W., Liu, W., Tao, D.: A survey on canonical correlation analysis. IEEE Trans. Knowl. Data Eng. 33(6), 2349–2368 (2019)
Zheng, S., et al.: Multi-modal graph learning for disease prediction. IEEE Trans. Med. Imaging 41(9), 2207–2216 (2022)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2024 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
D’Souza, N.S. et al. (2024). MaxCorrMGNN: A Multi-graph Neural Network Framework for Generalized Multimodal Fusion of Medical Data for Outcome Prediction. In: Maier, A.K., Schnabel, J.A., Tiwari, P., Stegle, O. (eds) Machine Learning for Multimodal Healthcare Data. ML4MHD 2023. Lecture Notes in Computer Science, vol 14315. Springer, Cham. https://doi.org/10.1007/978-3-031-47679-2_11
Download citation
DOI: https://doi.org/10.1007/978-3-031-47679-2_11
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-47678-5
Online ISBN: 978-3-031-47679-2
eBook Packages: Computer ScienceComputer Science (R0)