skip to main content
10.1145/3664647.3681395acmconferencesArticle/Chapter ViewAbstractPublication PagesmmConference Proceedingsconference-collections
research-article
Open access

Mixed Prototype Correction for Causal Inference in Medical Image Classification

Published: 28 October 2024 Publication History

Abstract

The heterogeneity of medical images poses significant challenges to accurate disease diagnosis. To tackle this issue, the impact of such heterogeneity on the causal relationship between image features and diagnostic labels should be incorporated into model design, which however remains underexplored. In this paper, we propose a mixed prototype correction for causal inference (MPCCI) method, aimed at mitigating the impact of unseen confounding factors on the causal relationships between medical images and disease labels, so as to enhance the diagnostic accuracy of deep learning models. The MPCCI comprises a causal inference component based on front-door adjustment and an adaptive training strategy. The causal inference component employs a multi-view feature extraction (MVFE) module to establish mediators, and a mixed prototype correction (MPC) module to execute causal interventions. Moreover, the adaptive training strategy incorporates both information purity and maturity metrics to maintain stable model training. Experimental evaluations on four medical image datasets, encompassing CT and ultrasound modalities, demonstrate the superior diagnostic accuracy and reliability of the proposed MPCCI. The code will be available at https://github.com/Yajie-Zhang/MPCCI.

References

[1]
Samaneh Abbasi-Sureshjani, Ralf Raumanns, Britt EJ Michels, Gerard Schouten, and Veronika Cheplygina. 2020. Risk of training diagnostic algorithms on data with demographic bias. In Interpretable and Annotation-Efficient Learning for Medical Image Computing: Third International Workshop, iMIMIC 2020, Second International Workshop, MIL3ID 2020, and 5th International Workshop, LABELS 2020, Held in Conjunction with MICCAI 2020, Lima, Peru, October 4--8, 2020, Proceedings 3. Springer, 183--192.
[2]
Walid Al-Dhabyani, Mohammed Gomaa, Hussien Khaled, and Aly Fahmy. 2020. Dataset of breast ultrasound images. Data in Brief 28 (2020), 104863.
[3]
François Bertucci and Daniel Birnbaum. 2008. Reasons for breast cancer heterogeneity. Journal of Biology 7 (2008), 1--4.
[4]
Alceu Bissoto, Eduardo Valle, and Sandra Avila. 2021. Gan-based data augmentation and anonymization for skin-lesion analysis: a critical review. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 1847--1856.
[5]
Iain Carmichael, Andrew H Song, Richard J Chen, Drew FK Williamson, Tiffany Y Chen, and Faisal Mahmood. 2022. Incorporating intratumoral heterogeneity into weakly-supervised deep learning models via variance pooling. In International Conference on Medical Image Computing and Computer-Assisted Intervention. Springer, 387--397.
[6]
Irem Cetin, Maialen Stephens, Oscar Camara, and Miguel A González Ballester. 2023. Attri-VAE: attribute-based interpretable representations of medical images with variational autoencoders. Computerized Medical Imaging and Graphics 104 (2023), 102158.
[7]
Zhang Chen, Zhiqiang Tian, Jihua Zhu, Ce Li, and Shaoyi Du. 2022. C-cam: causal cam for weakly supervised semantic segmentation on medical image. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 11676--11685.
[8]
Junlong Cheng, Chengrui Gao, Fengjie Wang, and Min Zhu. 2023. Segnetr: rethinking the local-global interactions and skip connections in u-shaped networks. In International Conference on Medical Image Computing and Computer-Assisted Intervention. Springer, 64--74.
[9]
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. 2020. An image is worth 16x16 words: transformers for image recognition at scale. In International Conference on Learning Representations.
[10]
Jingcan Duan, Pei Zhang, Siwei Wang, Jingtao Hu, Hu Jin, Jiaxin Zhang, Haifang Zhou, and Xinwang Liu. 2023. Normality learning-based graph anomaly detection via multi-scale contrastive learning. In Proceedings of the ACM International Conference on Multimedia. 7502--7511.
[11]
Bradley J Erickson, Panagiotis Korfiatis, Zeynettin Akkus, and Timothy L Kline. 2017. Machine learning for medical imaging. Radiographics 37, 2 (2017), 505--515.
[12]
Andreas Fouras, Marcus John Kitchen, Stephen Dubsky, RA Lewis, Stuart Brian Hooper, and Kerry Hourigan. 2009. The past, present, and future of x-ray technology for in vivo imaging of function and form. Journal of Applied Physics 105, 10 (2009).
[13]
Haifan Gong, Guanqi Chen, Mingzhi Mao, Zhen Li, and Guanbin Li. 2022. Vqamix: conditional triplet mixup for medical visual question answering. IEEE Transactions on Medical Imaging 41, 11 (2022), 3332--3343. https://doi.org/10.1109/TMI. 2022.3185008
[14]
Simon Graham, Hao Chen, Jevgenij Gamper, Qi Dou, Pheng-Ann Heng, David Snead, YeeWah Tsang, and Nasir Rajpoot. 2019. MILD-Net: minimal information loss dilated network for gland instance segmentation in colon histology images. Medical Image Analysis 52 (2019), 199--211.
[15]
Albert Gu and Tri Dao. 2023. Mamba: linear-time sequence modeling with selective state spaces. arXiv preprint arXiv:2312.00752 (2023).
[16]
Kai Han, An Xiao, Enhua Wu, Jianyuan Guo, Chunjing Xu, and Yunhe Wang. 2021. Transformer in transformer. Advances in Neural Information Processing Systems 34 (2021), 15908--15919.
[17]
Along He, Tao Li, Ning Li, Kai Wang, and Huazhu Fu. 2020. CABNet: category attention block for imbalanced diabetic retinopathy grading. IEEE Transactions on Medical Imaging 40, 1 (2020), 143--153. https://doi.org/10.1109/TMI.2020.3023463
[18]
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 770--778.
[19]
Haokai Hong, Min Jiang, Liang Feng, Qiuzhen Lin, and Kay Chen Tan. 2022. Balancing exploration and exploitation for solving large-scale multiobjective optimization via attention mechanism. In 2022 IEEE Congress on Evolutionary Computation (CEC). IEEE, 1--8.
[20]
Yao Hu, Zhi-An Huang, Rui Liu, Xiaoming Xue, Xiaoyan Sun, Linqi Song, and Kay Chen Tan. 2023. Source free semi-supervised transfer learning for diagnosis of mental disorders on fmri scans. IEEE Transactions on Pattern Analysis and Machine Intelligence (2023). https://doi.org/10.1109/TPAMI.2023.3298332
[21]
Zhi-An Huang, Yao Hu, Rui Liu, Xiaoming Xue, Zexuan Zhu, Linqi Song, and Kay Chen Tan. 2022. Federated multi-task learning for joint diagnosis of multiple mental disorders on mri scans. IEEE Transactions on Biomedical Engineering 70, 4 (2022), 1137--1149. https://doi.org/10.1109/TBME.2022.3210940
[22]
Zhi-An Huang, Rui Liu, Zexuan Zhu, and Kay Chen Tan. 2022. Multitask learning for joint diagnosis of multiple mental disorders in resting-state fmri. IEEE Transactions on Neural Networks and Learning Systems (2022). https: //doi.org/10.1109/TNNLS.2022.3225179
[23]
Zhi-An Huang, Jia Zhang, Zexuan Zhu, Edmond Q Wu, and Kay Chen Tan. 2020. Identification of autistic risk candidate genes and toxic chemicals via multilabel learning. IEEE Transactions on Neural Networks and Learning Systems 32, 9 (2020), 3971--3984. https://doi.org/10.1109/TNNLS.2020.3016357
[24]
Nahid Ul Islam, Zongwei Zhou, Shiv Gehlot, Michael B Gotway, and Jianming Liang. 2024. Seeking an optimal approach for Computer-aided Diagnosis of Pulmonary Embolism. Medical image analysis 91 (2024), 102988.
[25]
Amirhossein Kazerouni, Ehsan Khodapanah Aghdam, Moein Heidari, Reza Azad, Mohsen Fayyaz, Ilker Hacihaliloglu, and Dorit Merhof. 2023. Diffusion models in medical imaging: a comprehensive survey. Medical Image Analysis (2023), 102846.
[26]
Ashnil Kumar, Jinman Kim, David Lyndon, Michael Fulham, and Dagan Feng. 2016. An ensemble of fine-tuned convolutional neural networks for medical image classification. IEEE Journal of Biomedical and Health Informatics 21, 1 (2016), 31--40. https://doi.org/10.1109/JBHI.2016.2635663
[27]
Xiong Li, Liyue Liu, Juan Zhou, and Che Wang. 2018. Heterogeneity analysis and diagnosis of complex diseases based on deep learning methods. Scientific Reports 8, 1 (2018), 6155.
[28]
Zihan Li, Yuan Zheng, Xiangde Luo, Dandan Shan, and Qingqi Hong. 2023. Scribblevc: scribble-supervised medical image segmentation with vision-class embedding. In Proceedings of the ACM International Conference on Multimedia. 3384--3393.
[29]
Wu Lin, Qiuzhen Lin, Liang Feng, and Kay Chen Tan. 2023. Ensemble of domain adaptation-based knowledge transfer for evolutionary multitasking. IEEE Transactions on Evolutionary Computation (2023). https://doi.org/10.1109/TEVC.2023. 3259067
[30]
Rui Liu, Zhi-An Huang, Yao Hu, Zexuan Zhu, Ka-ChunWong, and Kay Chen Tan. 2023. Spatial--temporal co-attention learning for diagnosis of mental disorders from resting-state fmri data. IEEE Transactions on Neural Networks and Learning Systems (2023). https://doi.org/10.1109/TNNLS.2023.3243000
[31]
Chenxiang Ma, Jibin Wu, Chenyang Si, and KC Tan. 2023. Scaling supervised local learning with augmented auxiliary networks. In The Twelfth International Conference on Learning Representations.
[32]
Neelu Madan, Radu Tudor Ionescu, Kamal Nasrollahi, Fahad Shahbaz Khan, Thomas B Moeslund, and Mubarak Shah. 2023. Selfsupervised masked convolutional transformer block for anomaly detection. IEEE Transactions on Pattern Analysis and Machine Intelligence (2023). https://doi.org/ 10.1109/TPAMI.2023.3322604
[33]
Maede Maftouni, Andrew Chung Chee Law, Bo Shen, Zhenyu James Kong Grado, Yangze Zhou, and Niloofar Ayoobi Yazdi. 2021. A robust ensemble-deep learning model for COVID-19 diagnosis based on an integrated CT scan images database. In IIE annual conference. Proceedings. Institute of Industrial and Systems Engineers (IISE), 632--637.
[34]
Juzheng Miao, Cheng Chen, Furui Liu, Hao Wei, and Pheng-Ann Heng. 2023. Caussl: causality-inspired semi-supervised learning for medical image segmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision. 21426--21437.
[35]
Yuhao Mo, Chu Han, Yu Liu, Min Liu, Zhenwei Shi, Jiatai Lin, Bingchao Zhao, Chunwang Huang, Bingjiang Qiu, Yanfen Cui, et al. 2023. Hover-trans: anatomyaware hover-transformer for roi-free breast cancer diagnosis in ultrasound images. IEEE Transactions on Medical Imaging (2023). https://doi.org/10.1109/TMI. 2023.3236011
[36]
Cheng Ouyang, Chen Chen, Surui Li, Zeju Li, Chen Qin, Wenjia Bai, and Daniel Rueckert. 2022. Causality-inspired single-source domain generalization for medical image segmentation. IEEE Transactions on Medical Imaging 42, 4 (2022), 1095--1106. https://doi.org/10.1109/TMI.2022.3224067
[37]
Judea Pearl et al. 2000. Models, reasoning and inference. Cambridge, UK: CambridgeUniversityPress 19, 2 (2000), 3.
[38]
Mattia Prosperi, Yi Guo, Matt Sperrin, James S Koopman, Jae S Min, Xing He, Shannan Rich, Mo Wang, Iain E Buchan, and Jiang Bian. 2020. Causal inference and counterfactual prediction in machine learning for actionable healthcare. Nature Machine Intelligence 2, 7 (2020), 369--375.
[39]
Alexandre Rame, Corentin Dancette, and Matthieu Cord. 2022. Fishr: invariant gradient variances for out-of-distribution generalization. In International Conference on Machine Learning. PMLR, 18347--18377.
[40]
Karen Simonyan and AndrewZisserman. 2014. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556 (2014).
[41]
ChongWang, Yuyuan Liu, Yuanhong Chen, Fengbei Liu, Yu Tian, Davis McCarthy, Helen Frazer, and Gustavo Carneiro. 2023. Learning support and trivial prototypes for interpretable image classification. In Proceedings of the IEEE/CVF International Conference on Computer Vision. 2062--2072.
[42]
Junxia Wang, Yuanjie Zheng, Jun Ma, Xinmeng Li, Chongjing Wang, James Gee, Haipeng Wang, and Wenhui Huang. 2023. Information bottleneck-based interpretable multitask network for breast cancer classification and segmentation. Medical Image Analysis 83 (2023), 102687.
[43]
Sanghyun Woo, Jongchan Park, Joon-Young Lee, and In So Kweon. 2018. Cbam: convolutional block attention module. In Proceedings of the European Conference on Computer Vision. 3--19.
[44]
Yixuan Wu, Jintai Chen, Jiahuan Yan, Yiheng Zhu, Danny Z Chen, and Jian Wu. 2023. GCL: gradient-guided contrastive learning for medical image segmentation with multi-perspective meta labels. In Proceedings of the ACM International Conference on Multimedia. 463--471.
[45]
Xingran Xie, Ting Jin, Boxiang Yun, Qingli Li, and Yan Wang. 2023. Exploring hyperspectral histopathology image segmentation from a deformable perspective. In Proceedings of the ACM International Conference on Multimedia. 242--251.
[46]
Jie Xing, Chao Chen, Qinyang Lu, Xun Cai, Aijun Yu, Yi Xu, Xiaoling Xia, Yue Sun, Jing Xiao, and Lingyun Huang. 2020. Using BI-RADS stratifications as auxiliary information for breast masses classification in ultrasound images. IEEE Journal of Biomedical and Health Informatics 25, 6 (2020), 2058--2070. https: //doi.org/10.1109/JBHI.2020.3034804
[47]
Yun Yang, Yuanyuan Hu, Xingyi Zhang, and Song Wang. 2021. Two-stage selective ensemble of cnn via deep tree training for medical image classification. IEEE Transactions on Cybernetics 52, 9 (2021), 9194--9207. https://doi.org/10.1109/ TCYB.2021.3061147
[48]
Chunyan Yu, Baoyu Gong, Meiping Song, Enyu Zhao, and Chein-I Chang. 2022. Multiview calibrated prototype learning for few-shot hyperspectral image classification. IEEE Transactions on Geoscience and Remote Sensing 60 (2022), 1--13. https://doi.org/10.1109/TGRS.2022.3225947
[49]
Lin Yue, Dongyuan Tian, Weitong Chen, Xuming Han, and Minghao Yin. 2020. Deep learning for heterogeneous medical data analysis. World Wide Web 23 (2020), 2715--2737.
[50]
Dong Zhang, Hanwang Zhang, Jinhui Tang, Xian-Sheng Hua, and Qianru Sun. 2020. Causal intervention for weakly-supervised semantic segmentation. Advances in Neural Information Processing Systems 33 (2020), 655--666.
[51]
Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. 2018. Mixup: beyond empirical risk minimization. In International Conference on Learning Representations.
[52]
Ling Zhang, Xiaosong Wang, Dong Yang, Thomas Sanford, Stephanie Harmon, Baris Turkbey, Bradford J Wood, Holger Roth, Andriy Myronenko, Daguang Xu, et al. 2020. Generalizing deep learning for medical image segmentation to unseen domains via deep stacked transformation. IEEE Transactions on Medical Imaging 39, 7 (2020), 2531--2540. https://doi.org/10.1109/TMI.2020.2973595
[53]
Bolei Zhou, Aditya Khosla, Agata Lapedriza, Aude Oliva, and Antonio Torralba. 2016. Learning deep features for discriminative localization. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2921--2929.
[54]
Kaiyang Zhou, Yongxin Yang, Yu Qiao, and Tao Xiang. 2020. Domain generalization with mixstyle. In International Conference on Learning Representations.

Cited By

View all
  • (2024)CRViT: Vision transformer advanced by causality and inductive bias for image recognitionApplied Intelligence10.1007/s10489-024-05910-355:1Online publication date: 2-Dec-2024

Index Terms

  1. Mixed Prototype Correction for Causal Inference in Medical Image Classification

    Recommendations

    Comments

    Information & Contributors

    Information

    Published In

    cover image ACM Conferences
    MM '24: Proceedings of the 32nd ACM International Conference on Multimedia
    October 2024
    11719 pages
    ISBN:9798400706868
    DOI:10.1145/3664647
    This work is licensed under a Creative Commons Attribution International 4.0 License.

    Sponsors

    Publisher

    Association for Computing Machinery

    New York, NY, United States

    Publication History

    Published: 28 October 2024

    Check for updates

    Author Tags

    1. causal inference
    2. disease diagnosis
    3. front-door adjustment
    4. multi-view prototype learning

    Qualifiers

    • Research-article

    Funding Sources

    • the National Natural Science Foundation of China
    • the Research Grants Council of the Hong Kong SAR
    • The Hong Kong Polytechnic University

    Conference

    MM '24
    Sponsor:
    MM '24: The 32nd ACM International Conference on Multimedia
    October 28 - November 1, 2024
    Melbourne VIC, Australia

    Acceptance Rates

    MM '24 Paper Acceptance Rate 1,150 of 4,385 submissions, 26%;
    Overall Acceptance Rate 2,145 of 8,556 submissions, 25%

    Contributors

    Other Metrics

    Bibliometrics & Citations

    Bibliometrics

    Article Metrics

    • Downloads (Last 12 months)241
    • Downloads (Last 6 weeks)76
    Reflects downloads up to 17 Feb 2025

    Other Metrics

    Citations

    Cited By

    View all
    • (2024)CRViT: Vision transformer advanced by causality and inductive bias for image recognitionApplied Intelligence10.1007/s10489-024-05910-355:1Online publication date: 2-Dec-2024

    View Options

    View options

    PDF

    View or Download as a PDF file.

    PDF

    eReader

    View online with eReader.

    eReader

    Login options

    Figures

    Tables

    Media

    Share

    Share

    Share this Publication link

    Share on social media