skip to main content
research-article

A Multimodal, Multimedia Point-of-Care Deep Learning Framework for COVID-19 Diagnosis

Published: 31 March 2021 Publication History

Abstract

In this article, we share our experiences in designing and developing a suite of deep neural network–(DNN) based COVID-19 case detection and recognition framework. Existing pathological tests such as RT-PCR-based pathogen RNA detection from nasal swabbing seem to display low detection rates during the early stages of virus contraction. Moreover, the reliance on a few overburdened laboratories based around an epicenter capable of supplying large numbers of RT-PCR tests makes this testing method non-scalable when the rate of infections is high. Similarly, finding an effective drug or vaccine with which to combat COVID-19 requires a long time and many clinical trials. The development of pathological COVID-19 tests is hindered by shortages in the supply chain of chemical reagents necessary for testing on a large scale. This diminishes the speed of diagnosis and the ability to filter out COVID-19 positive patients from uninfected patients on a national level. Existing research has shown that DNN has been successful in identifying COVID-19 from radiological media such as CT scans and X-ray images, audio media such as cough sounds, optical coherence tomography to identify conjunctivitis and pink eye symptoms on the ocular surface, body temperature measurement using smartphone fingerprint sensors or thermal cameras, the use of live facial detection to identify safe social distancing practices from camera images, and face mask detection from camera images. We also investigate the utility of federated learning in diagnosis cases where private data can be trained via edge learning. These point-of-care modalities can be integrated with DNN-based RT-PCR laboratory test results to assimilate multiple modalities of COVID-19 detection and thereby provide more dimensions of diagnosis. Finally, we will present our initial test results, which are encouraging.

References

[1]
M. S. Hossain, S. U. Amin, M. Alsulaiman, and G. Muhammad. 2019. Applying deep learning for epilepsy seizure detection and brain mapping visualization. ACM Trans. Multimed. Comput. Commun. Appl 15, 1 (2019), 1–17.
[2]
M. A. Rahman and M. S. Hossain. 2021. An internet of medical things-enabled edge computing framework for tackling COVID-19. IEEE IoT J.
[3]
T. Phan. 2020. Novel coronavirus: From discovery to clinical diagnostics. Infect. Genet. Evol. 79 (2020), 104211.
[4]
M. Shen et al. 2020. Recent advances and perspectives of nucleic acid detection for coronavirus. J. Pharm. Anal. 10, 2 (2020), 97--101.
[5]
M. Shorfuzzaman and M. S. Hossain. 2021. MetaCOVID: A Siamese neural network framework with contrastive loss for n-shot diagnosis of COVID-19 patients. Pattern Recognition 113 (2021), 107700.
[6]
M. Shorfuzzaman, M. S. Hossain, and M. F. Alhamid. 2021. Towards the sustainable development of smart cities through mass video surveillance: A response to the COVID-19 pandemic. Sustainable Cities and Society 64 (2021), 102582.
[7]
C. Biscayart, P. Angeleri, S. Lloveras, T. do S. S. Chaves, P. Schlagenhauf, and A. J. Rodríguez-Morales. 2020. The next big threat to global health? 2019 novel coronavirus (2019-nCoV): What advice can we give to travellers? Interim recommendations January 2020, from the Latin-American society for Travel Medicine (SLAMVI). Travel Med. Infect. Dis. 33 (2020), 17–20.
[8]
M. Hoffmann et al. 2020. SARS-CoV-2cell entry depends on ACE2 and TMPRSS2 and is blocked by a 1clinically-proven protease inhibitor. Cell (2020), 1–10.
[9]
X. Liu and X.-J. Wang. 2020. Potential inhibitors against 2019-ncov coronavirus m protease from clinically approved medicines. J. Genet. Genom. (2020).
[10]
X. Li, M. Geng, Y. Peng, L. Meng, and S. Lu. 2020. Molecular immune pathogenesis and diagnosis of COVID-19. J. Pharm. Anal. (2020).
[11]
J. Chen. 2020. Pathogenicity and transmissibility of 2019-nCoV—A quick overview and comparison with other emerging viruses. Microbes Infect. 2019—2021.
[12]
A. K. Cordes and A. Heim. 2020. Rapid random access detection of the novel SARS-coronavirus-2 (SARS-CoV-2, previously 2019-nCoV) using an open access protocol for the panther fusion. J. Clin. Virol 125 (2020), 104305.
[13]
H. Shi et al. 2020. Radiological findings from 81 patients with COVID-19 pneumonia in wuhan, china: A descriptive study. Lancet. Infect. Dis. 3099, 20 (2020), 1–10.
[14]
R. Lu et al. 2020. Genomic characterisation and epidemiology of 2019 novel coronavirus: Implications for virus origins and receptor binding. Lancet 395, 10224 (2020), 565–574.
[15]
J. A. Al-tawfiq, A. H. Al-homoud, and Z. A. Memish. 2020. Remdesivir as a possible therapeutic option for the COVID-19. Travel Med. Infect. Dis. (March 2020), 101615.
[16]
A. Lopez-Rincon, A. Tonda, L. Mendoza-Maldonado, E. Claassen, J. Garssen, and A. D. Kraneveld. 2020. Accurate identification of SARS-CoV-2 from viral genome sequences using deep learning. bioRxiv, 2020.03.13.990242
[17]
C. Huang, P. Lee, and P. Hsueh. 2020. Arguments in favor of remdesivir for treating SARS-CoV-2 infections. Int. J. Antimicrob. Agents (2020) 105933.
[18]
G. S. Randhawa, M. P. M. Soltysiak, H. El Roz, C. P. E. de Souza, K. A. Hill, and L. Kari. 2020. Machine learning using intrinsic genomic signatures for rapid classification of novel pathogens: COVID-19 case study. bioRxiv. 2020.02.03.932350.
[19]
K. C. Santosh. 2020. AI-driven tools for coronavirus outbreak: Need of active learning and cross-population train/test models on multitudinal/multimodal data. J. Med. Syst. 44, 5 (2020), 93.
[20]
H. Hou et al. 2020. Artificial intelligence distinguishes COVID-19 from community acquired pneumonia on chest CT. RSNA Radiol. (2020).
[21]
M. S. Hossain and G. Muhammad. 2019. Emotion recognition using deep learning approach from audio-visual emotional big data. Inf. Fus. 49 (2019), 69–78.
[22]
Y. Song et al. 2020. Deep learning enables accurate diagnosis of novel coronavirus (COVID-19) with CT images. MedRxiv. 2020.02.23.20026930.
[23]
S. K. Sundararajan and S. P. D. 2020. Detection of conjunctivitis with deep learning algorithm in medical image processing. In Proceedings of the 3rd International Conference on the Internet of Things in Social, Mobile, Analytics and Cloud (I-SMAC’20). 714–717.
[24]
M. A. Rahman et al. 2020. Secure and provenance enhanced Internet of health things framework: A blockchain managed federated learning approach. IEEE Access 8 (2020), 205071--205087.
[25]
G. Muhammad, M. S. Hossain, and N. Kumar. 2021. EEG-Based pathology detection for home health monitoring. IEEE J. Sel. Areas Commun. 39, 2 (2021), 603--610.
[26]
G. A. Kaissis, M. R. Makowski, D. Rückert, and R. F. Braren. 2020. Secure, privacy-preserving and federated machine learning in medical imaging. Nat. Mach. Intell 2, 6 (2020), 305–311.
[27]
N. D. Lane, P. Georgiev, and L. Qendro. 2015. DeepEar: Robust smartphone audio sensing in unconstrained acoustic environments using deep learning. In Proceedings of the 2015 ACM International Joint Conference on Pervasive Ubiquitous Computing (UbiComp’15), 283–294.
[28]
R. S. Hande and P. S. Deshpande. 2017. An integrated computerized cough analysis by using wavelet for pneumonia diagnosis. Int. J. Innov. Sci. Res. Technol 2, 9 (2017), 283–287.
[29]
M. S. Hossain and G. Muhammad. 2020. Deep learning based pathology detection for smart connected healthcares. IEEE Network. 34, 6 (2020), 120--125.
[30]
M. S. Hossain, G. Muhammad, and N. Guizani. 2020. Explainable ai and mass surveillance system-based healthcare framework to combat COVID-I9 like pandemics. IEEE Netw. 34, 4 (2020), 126–132.
[31]
C. Pham. 2016. MobiCough: Real-time cough detection and monitoring using low-cost mobile devices. In Proceedings of the 8th Asian Conference on Intelligent Information and Database Systems (ACIIDS’16). 300–309.
[32]
P. Porter et al. 2019. A prospective multicentre study testing the diagnostic accuracy of an automated cough sound centred analytic system for the identification of common respiratory disorders in children. Respir. Res 20, 1 (2019), 1–10.
[33]
F. Guo et al. 2018. Yanbao: A mobile app using the measurement of clinical parameters for glaucoma screening. IEEE Access. 6 (2018), 77414–77428.
[34]
X. Sun et al. 2020. The infection evidence of SARS-COV-2 in ocular surface: A single-center cross-sectional study. MedRxiv. 8197033356.
[35]
H. Zhang et al. 2020. Corona virus international public health emergencies: Implications for radiology management. Acad. Radiol. (2020), 1–5.
[36]
Y.-H. Xu et al. 2020. Clinical and computed tomographic imaging features of novel coronavirus pneumonia caused by SARS-CoV-2. J. Infect. (2020).
[37]
J. A. Al-tawfiq and Z. A. Memish. 2020. Diagnosis of SARS-CoV-2 infection based on CT scan vs. RT-PCR: Reflecting on experience from MERS-CoV. J. Hosp. Infect. (2020).
[38]
L. Wang and A. Wong. 2020. COVID-Net: A tailored deep convolutional neural network design for detection of COVID-19 cases from chest radiography images. arXiv:2003.09871v4. Retrieved from https://arxiv.org/abs/2003.09871v4.
[39]
C. Lin et al. 2020. Asymptomatic novel coronavirus pneumonia patient outside WuHan: The value of CT images in the course of the disease. Clin. Imag. (2020), 153057.
[40]
S. Kooraki, M. Hosseiny, L. Myers, and A. Gholamrezanezhad. 2020. Coronavirus outbreak: What the department of radiology should know. J. Am. Coll. Radiol. (2020), 1–5.
[41]
H. S. Maghdid, K. Z. Ghafoor, A. S. Sadiq, K. Curran, and K. Rabie. 2020. A novel ai-enabled framework to diagnose coronavirus COVID 19 using smartphone embedded sensors: Design study. arXiv:2003.07434. Retrieved from https://arxiv.org/abs/2003.07434.
[42]
H. A. Rothan and S. N. Byrareddy. 2020. The epidemiology and pathogenesis of coronavirus disease (COVID-19) outbreak. J. Autoimmun. PG-102433. (2020), 102433.
[43]
M. A. Rahman et al. 2019. Blockchain and IoT-based cognitive edge framework for sharing economy services in a smart city. IEEE Access. 7 (2019), 18611–18621.
[44]
M. A. Rahman et al. 2018. Blockchain-based mobile edge computing framework for secure therapy applications. IEEE Access. 6 (2018), 72469–72478.
[45]
Y. Qian et al. 2020. Blockchain-based privacy-aware content caching in cognitive internet of vehicles. IEEE Netw. 34, 2 (2020), 46–51.
[46]
M. A. Rahman et al. 2019. A blockchain-based non-invasive cyber-physical occupational therapy framework: BCI perspective. IEEE Access 7 (2019), 34874–34884.
[47]
A. Alimadadi, S. Aryal, I. Manandhar, P. B. Munroe, B. Joe, and X. Cheng. 2020. Artificial intelligence and machine learning to fight COVID-19. Physiol. Genom. (2020), 1–9.
[48]
Y. Abdulsalam and M. S. Hossain. 2020. COVID-19 networking demand: An auction-based mechanism for automated selection of edge computing services. IEEE Transactions on Network Science and Engineering.
[49]
Q. Lin et al. 2019. A conceptual model of the outbreak of novel coronavirus (2019-nCoV) in Wuhan, China, with human reaction and holiday effects. Int. J. Infect. Dis. (Feb. 2019), 1–9.
[50]
A. Wilder-smith, C. J. Chiew, and V. J. Lee. 2020. Personal View Can we contain the COVID-19 outbreak with the same measures as for SARS ? Lancet Infect. Dis. 3099 (2020), 20.
[51]
Kaggle. COVID-19 Image Dataset. Retrieved from https://www.kaggle.com/paultimothymooney/chest-xray-pneumonia/download.
[52]
H. Nishiura, N. M. Linton, and A. R. Akhmetzhanov. 2020. Serial interval of novel coronavirus (COVID-19) infections. Int. J. Infect. Dis. 93 (2020), 284–286.
[53]
J. P. Cohen, P. Morrison, and L. Dao. 2020. COVID-19 Image data collection. arXiv:2003.11597. Retrieved from https://arxiv.org/abs/2003.11597.
[54]
O. Gozes, M. Frid-Adar, H. Greenspan, P. D. Browning, A. Bernheim, and E. Siegel. 2020. Rapid ai development cycle for the coronavirus (COVID-19) pandemic: initial results for automated detection & patient monitoring using deep learning CT image analysis. arXiv:2003.05037. Retrieved from https://arxiv.org/abs/2003.05037.
[55]
J. Zhao, Y. Zhang, X. He, and P. Xie. 2020. COVID-CT-Dataset: A CT Scan dataset about COVID-19. arXiv:2003.13865. Retrieved from https://arxiv.org/abs/2003.13865.
[56]
A. Narin, C. Kaya, and Z. Pamuk. 2020. Automatic detection of coronavirus disease (COVID-19) using x-ray images and deep convolutional neural networks. arXiv:2003.10849. Retrieved from https://arxiv.org/abs/2003.10849.
[57]
J. Bullock, Alexandra, Luccioni, K. H. Pham, C. S. N. Lam, and M. Luengo-Oroz. 2020. Mapping the landscape of artificial intelligence applications against COVID-19. arXiv:2003.11336. Retrieved from https://arxiv.org/abs/2003.11336.
[58]
J. M. Shuai Wang and Bo Kang. 2020. A deep learning algorithm using CT images to screen for corona virus disease (COVID-19). MedRxiv.
[59]
L. O. Hall, R. Paul, D. B. Goldgof, and G. M. GOLDGOF. 2020. Finding covid-19 from chest X-rays using deep learning on a small dataset. arXiv:2004.02060. Retrieved from https://arxiv.org/abs/2004.02060.
[60]
B. D. Killeen et al. 2020. A county-level dataset for informing the united states’ response to COVID-19. arXiv:2004.00756. Retrieved from https://arxiv.org/abs/2004.00756.
[61]
P. Afshar, S. Heidarian, F. Naderkhani, A. Oikonomou, K. N. Plataniotis, and A. Mohammadi. 2020. COVID-CAPS: A capsule network-based framework for identification of COVID-19 cases from X-ray Images. arXiv:2004.02696. Retrieved from https://arxiv.org/abs/2004.02696.
[62]
S. Chaganti et al. 2020. Quantification of tomographic patterns associated with COVID-19 from chest CT. 5. arXiv:2004.01279. Retrieved from https://arxiv.org/abs/2004.01279.
[63]
M. A. Rahman, M. S. Hossain, N. A. Alrajeh, and N. Guizani. 2020. B5G and explainable deep learning assisted healthcare vertical at the edge: COVID-I9 perspective. IEEE Netw. 34, 4 (2020), 98–105.
[64]
M. A. Rahman and M. S. Hossain. 2021. An Internet of medical things-enabled edge computing framework for tackling COVID-19. IEEE Internet of Things J. (2021) 1--8.
[65]
M. A. Rahman, M. S. Hossain, N. Alrajeh, and F. Alsolami. 2020. Adversarial examples – security threats to COVID-19 deep learning systems in medical iot devices. IEEE IoT J.
[66]
F. Shan et al. 2020. Lung infection quantification of COVID-19 in CT images with deep learning. arXiv:2003.04655. Retrieved from https://arxiv.org/abs/2003.04655.
[67]
M. Farooq and A. Hafeez. 2020. COVID-ResNet: A deep learning framework for screening of COVID19 from radiographs. arXiv:2003.14395. Retrieved from https://arxiv.org/abs/2003.14395.
[68]
P. Kumar and S. Kumari. 2020. Detection of coronavirus disease (COVID-19) based on deep features.
[69]
B. Ghoshal and A. Tucker. 2020. Estimating uncertainty and interpretability in deep learning for coronavirus (COVID-19). Detection (2020), 1–14.
[70]
M. Loey, F. Smarandache, and N. E. M. Khalifa. 2020. Within the lack of COVID-19 benchmark dataset : A novel GAN with deep transfer learning for corona- virus detection in chest x-ray images (unpublished).
[71]
Y. Ye et al. 2020. α-Satellite: An ai-driven system and benchmark datasets for hierarchical community-level risk assessment to help combat COVID-19. arXiv:2003.12232. Retrieved from https://arxiv.org/abs/2003.12232.
[72]
N. E. M. Khalifa and M. H. N. Taha, Detection of Coronavirus (COVID-19) associated pneumonia based on generative adversarial networks and a fine-tuned deep transfer learning model using chest x-ray dataset. arXiv:2004.01184. Retrieved from https://arxiv.org/abs/2004.01184.
[73]
Y. Liang and P. Xie. 2020. Identifying radiological findings related to COVID-19 from medical literature. arXiv:2004.01862. Retrieved from https://arxiv.org/abs/2004.01862.
[74]
A. Abbas, M. M. Abdelsamea, and M. M. Gaber. 2020. Classification of COVID-19 in chest X-ray images using DeTraC deep convolutional neural network. medRxiv.
[75]
R. R. Selvaraju, M. Cogswell, A. Das, R. Vedantam, D. Parikh, and D. Batra. 2020. Grad-CAM: Visual explanations from deep networks via gradient-based localization. Int. J. Comput. Vis 128, 2 (2020), 336–359.
[76]
F. Song et al. 2020. Emerging 2019 novel coronavirus (2019-NCoV) pneumonia. Radiology 295, 1 (2019), 210–217.
[77]
Q. Guo, J. Ye, Y. Hu, G. Zhang, X. Li, and H. Li. 2020. MultiPAD: A multivariant partition-based method for audio adversarial examples detection. IEEE Access. 8 (2020), 63368–63380.
[78]
R. Taori, A. Kamsetty, B. Chu, and N. Vemuri. 2019. Targeted adversarial examples for black box audio systems. In Proceedings of the 2019 IEEE Symposium on Security and Privacy Workshops (SPW’19). 15–20.
[79]
W. Zhang. 2019. Generating adversarial examples in one shot with image-to-image translation GAN. IEEE Access. 7 (2019), 151103–151119.
[80]
M. Sharif, S. Bhagavatula, L. Bauer, and M. K. Reiter. 2019. A general framework for adversarial examples with objectives. ACM Trans. Priv. Secur 22, 3 (2019).

Cited By

View all
  • (2024)From CNNs to Transformers in Multimodal Human Action Recognition: A SurveyACM Transactions on Multimedia Computing, Communications, and Applications10.1145/366481520:8(1-24)Online publication date: 13-May-2024
  • (2024)A novel lightweight CNN for chest X-ray-based lung disease identification on heterogeneous embedded systemApplied Intelligence10.1007/s10489-024-05420-254:6(4756-4780)Online publication date: 1-Mar-2024
  • (2023)Car Emotion Labeling Based on Color-SSL Semi-Supervised Learning Algorithm by Color AugmentationInternational Journal of Intelligent Systems10.1155/2023/43318382023Online publication date: 1-Jan-2023
  • Show More Cited By

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Transactions on Multimedia Computing, Communications, and Applications
ACM Transactions on Multimedia Computing, Communications, and Applications  Volume 17, Issue 1s
January 2021
353 pages
ISSN:1551-6857
EISSN:1551-6865
DOI:10.1145/3453990
Issue’s Table of Contents
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 31 March 2021
Accepted: 01 August 2020
Revised: 01 July 2020
Received: 01 April 2020
Published in TOMM Volume 17, Issue 1s

Permissions

Request permissions for this article.

Check for updates

Author Tags

  1. Deep Learning
  2. COVID-19 pandemic
  3. point-of-care

Qualifiers

  • Research-article
  • Refereed

Funding Sources

  • KACST, Saudi Arabia, Research Fund

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)81
  • Downloads (Last 6 weeks)15
Reflects downloads up to 17 Jan 2025

Other Metrics

Citations

Cited By

View all
  • (2024)From CNNs to Transformers in Multimodal Human Action Recognition: A SurveyACM Transactions on Multimedia Computing, Communications, and Applications10.1145/366481520:8(1-24)Online publication date: 13-May-2024
  • (2024)A novel lightweight CNN for chest X-ray-based lung disease identification on heterogeneous embedded systemApplied Intelligence10.1007/s10489-024-05420-254:6(4756-4780)Online publication date: 1-Mar-2024
  • (2023)Car Emotion Labeling Based on Color-SSL Semi-Supervised Learning Algorithm by Color AugmentationInternational Journal of Intelligent Systems10.1155/2023/43318382023Online publication date: 1-Jan-2023
  • (2023)DiRaC-I: Identifying Diverse and Rare Training Classes for Zero-Shot LearningACM Transactions on Multimedia Computing, Communications, and Applications10.1145/360314720:1(1-23)Online publication date: 31-May-2023
  • (2023)Blockchain-Based Mechanism for Smart Record Monitoring During and After the COVID-19 PandemicIT Professional10.1109/MITP.2023.331065025:5(20-28)Online publication date: 1-Sep-2023
  • (2023)Long Short-Term Memory-based Deep Learning Model for COVID-19 Detection using Coughing SoundSN Computer Science10.1007/s42979-023-01934-74:5Online publication date: 30-Jun-2023
  • (2023)Evaluating the Sustainable COVID-19 Vaccination Framework of India Using Recurrent Neural NetworksWireless Personal Communications: An International Journal10.1007/s11277-023-10751-3133:1(73-91)Online publication date: 1-Nov-2023
  • (2023)A Novel Optimized Perturbation-Based Machine Learning for Preserving Privacy in Medical DataWireless Personal Communications: An International Journal10.1007/s11277-023-10363-x130:3(1905-1927)Online publication date: 11-Mar-2023
  • (2023)A Systematic Review of Multimodal Deep Learning Approaches for COVID-19 DiagnosisImage Analysis and Processing - ICIAP 2023 Workshops10.1007/978-3-031-51026-7_13(140-151)Online publication date: 11-Sep-2023
  • (2022)Diagnosing COVID-19 from CT Image of Lung Segmentation & Classification with Deep Learning Based on Convolutional Neural NetworksWireless Personal Communications: An International Journal10.1007/s11277-021-09076-w127:3(2483-2499)Online publication date: 1-Dec-2022
  • Show More Cited By

View Options

Login options

Full Access

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

HTML Format

View this article in HTML Format.

HTML Format

Media

Figures

Other

Tables

Share

Share

Share this Publication link

Share on social media