Abstract
Artificial Intelligence-based computer-aided diagnosis (CAD) has been widely applied to assist medical professionals in several medical applications. Although there are many studies on respiratory disease detection using Deep Learning (DL) approaches from radiographic images, the limited availability of public datasets limits their interpretation and generalization capacity. However, radiography images are available through different organizations in various countries. This condition is suited for Federated Learning (FL) training, which can collaborate with different institutes to use private data and train a global model. In FL, the local model on the client’s end is critical because there must be a balance between the model’s accuracy, communication cost, and client-side memory usage. The current DL or Vision Transformer (ViT)-based models have large parameters, making the client-side memory and communication costs a significant bottleneck when applied to FL training. The existing state-of-the-art (SOTA) FL techniques on respiratory disease detection either use small CNNs with insufficient accuracy or assume clients have sufficient processing capacity to train large models, which remains a significant challenge in practical applications. In this study, we tried to find one question: Is it possible to maintain higher accuracy while lowering the model parameters, leading to lower memory requirements and communication costs? To address this problem, we propose a federated multi-stage, light-weight ViT framework that combines the strengths of CNNs and ViTs to build an efficient FL framework. We conduct extensive experiments and show that the proposed framework outperforms a set of current SOTA models in FL training with higher accuracy while lowering communication costs and memory requirements. We adapted Grad-CAM for the infection localization and compared it with an experienced radiologist’s findings.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Dai, Z., Liu, H., Le, Q.V., Tan, M.: Coatnet: marrying convolution and attention for all data sizes. Adv. Neural. Inf. Process. Syst. 34, 3965–3977 (2021)
Dosovitskiy, A., et al.: An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929 (2020)
Feki, I., Ammar, S., Kessentini, Y., Muhammad, K.: Federated learning for Covid-19 screening from chest x-ray images. Appl. Soft Comput. 106, 107330 (2021)
Japkowicz, N., Shah, M.: Evaluating learning algorithms: a classification perspective. Cambridge University Press (2011)
Liu, B., Yan, B., Zhou, Y., Yang, Y., Zhang, Y.: Experiments of federated learning for covid-19 chest x-ray images. arXiv preprint arXiv:2007.05592 (2020)
Liu, Z., et al.: Swin transformer: hierarchical vision transformer using shifted windows. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 10012–10022 (2021)
McMahan, B., Moore, E., Ramage, D., Hampson, S., y Arcas, B.A.: Communication-efficient learning of deep networks from decentralized data. In: Artificial intelligence and statistics, pp. 1273–1282. PMLR (2017)
Mehta, S., Rastegari, M.: Mobilevit: light-weight, general-purpose, and mobile-friendly vision transformer. arXiv preprint arXiv:2110.02178 (2021)
Mooney, P.: Chest x-ray images (pneumonia). kaggle, Marzo (2018)
Palmal, S., Arya, N., Saha, S., Tripathy, S.: A multi-modal graph convolutional network for predicting human breast cancer prognosis. In: International Conference on Neural Information Processing, pp. 187–198. Springer (2022). https://doi.org/10.1007/978-981-99-1648-1_16
Palmal, S., Saha, S., Tripathy, S.: HIV-1 protease cleavage site prediction using stacked autoencoder with ensemble of classifiers. In: 2022 International Joint Conference on Neural Networks (IJCNN), pp. 1–8. IEEE (2022)
Palmal, S., Saha, S., Tripathy, S.: Multi-objective optimization with majority voting ensemble of classifiers for prediction of HIV-1 protease cleavage site. Soft Comput. 27, 12211–1221 (2023). https://doi.org/10.1007/s00500-023-08431-2
Rahman, T., et al.: Reliable tuberculosis detection using chest x-ray with deep learning, segmentation and visualization. IEEE Access 8, 191586–191601 (2020)
Sahoo, P., Saha, S., Mondal, S., Chowdhury, S., Gowda, S.: Computer-aided Covid-19 screening from chest CT-scan using a fuzzy ensemble-based technique. In: 2022 International Joint Conference on Neural Networks (IJCNN), pp. 1–8. IEEE (2022)
Sahoo, P., Saha, S., Mondal, S., Chowdhury, S., Gowda, S.: Vision transformer-based federated learning for covid-19 detection using chest x-ray. In: Neural Information Processing: 29th International Conference, ICONIP 2022, Virtual Event, November 22–26, 2022, Proceedings, Part VII, pp. 77–88. Springer (2023). https://doi.org/10.1007/978-981-99-1648-1_7
Sahoo, P., Saha, S., Mondal, S., Gowda, S.: Vision transformer based Covid-19 detection using chest CT-scan images. In: 2022 IEEE-EMBS International Conference on Biomedical and Health Informatics (BHI), pp. 01–04. IEEE (2022)
Sahoo, P., Saha, S., Mondal, S., Sharma, N.: Covid-19 detection from lung ultrasound images using a fuzzy ensemble-based transfer learning technique. In: 2022 26th International Conference on Pattern Recognition (ICPR), pp. 5170–5176. IEEE (2022)
Sandler, M., Howard, A., Zhu, M., Zhmoginov, A., Chen, L.C.: Mobilenetv 2: inverted residuals and linear bottlenecks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4510–4520 (2018)
Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: visual explanations from deep networks via gradient-based localization. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 618–626 (2017)
Toraman, S., Alakus, T.B., Turkoglu, I.: Convolutional capsnet: a novel artificial neural network approach to detect Covid-19 disease from x-ray images using capsule networks. Chaos, Solitons Fractals 140, 110122 (2020)
Varshni, D., Thakral, K., Agarwal, L., Nijhawan, R., Mittal, A.: Pneumonia detection using CNN based feature extraction. In: 2019 IEEE International Conference on Electrical, Computer and Communication Technologies (ICECCT), pp. 1–7. IEEE (2019)
Vaswani, A., et al.: Attention is all you need. In: Advances in Neural Information Processing Systems 30 (2017)
Wang, L., Lin, Z.Q., Wong, A.: Covid-net: a tailored deep convolutional neural network design for detection of Covid-19 cases from chest x-ray images. Sci. Rep. 10(1), 1–12 (2020)
WHO: who coronavirus (Covid-19) dashboard. https://covid19.who.int/ (2022)
Xiao, T., Singh, M., Mintun, E., Darrell, T., Dollár, P., Girshick, R.: Early convolutions help transformers see better. Adv. Neural. Inf. Process. Syst. 34, 30392–30400 (2021)
Zhao, A., Aboutalebi, H., Wong, A., Gunraj, H., Terhljan, N., et al.: Covidx cxr-2: chest x-ray images for the detection of Covid-19 (2021)
Acknowledgement
Dr. Sriparna Saha gratefully acknowledges the Young Faculty Research Fellowship (YFRF) Award, supported by Visvesvaraya Ph.D. Scheme for Electronics and IT, Ministry of Electronics and Information Technology (MeitY), Government of India, being implemented by Digital India Corporation (formerly Media Lab Asia) for carrying out this research.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2024 The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd.
About this paper
Cite this paper
Sahoo, P., Sharma, S.K., Saha, S., Mondal, S. (2024). A Federated Multi-stage Light-Weight Vision Transformer for Respiratory Disease Detection. In: Luo, B., Cheng, L., Wu, ZG., Li, H., Li, C. (eds) Neural Information Processing. ICONIP 2023. Communications in Computer and Information Science, vol 1964. Springer, Singapore. https://doi.org/10.1007/978-981-99-8141-0_23
Download citation
DOI: https://doi.org/10.1007/978-981-99-8141-0_23
Published:
Publisher Name: Springer, Singapore
Print ISBN: 978-981-99-8140-3
Online ISBN: 978-981-99-8141-0
eBook Packages: Computer ScienceComputer Science (R0)