Improving prediction for medical institution with limited patient data: Leveraging hospital-specific data based on multicenter collaborative research network

Highlights

• A multisource deep transfer learning model is proposed for improving predictive performance on a single institution with limited patient data.

• The proposed approach enables better feature adaptation by incorporating hospital-specific features across source data and limited target data.

• The unlabeled target data is integrated into the model updating approach to enhance the training process for single center with insufficient labels.

• The case study shows the better discrimination and calibration ability of proposed model learning process than baseline models with limited EHR data.

Abstract

Background and objective

Clinical decision support assisted by prediction models usually faces the challenges of limited clinical data and a lack of labels when the model is developed with data from a single medical institution. Accordingly, research on multicenter clinical collaborative networks, which can provide external medical data, has received increasing attention. With the increasing availability of machine learning techniques such as transfer learning, leveraging large-scale patient data from multiple hospitals to build data-driven predictive models with clinical application potential provides an alternative solution to address the problem of limited patient data.

Methods

A multicenter hybrid semi-supervised transfer learning model (MHSTL) is proposed in this study on the basis of unified common data model to ensure multicenter data standardized representation. Then the hospital-specific features, along with the co-occurrence features across domains, are aligned through a representation learning architecture that is built based on deep neural networks and the newly proposed neural decision forest model. In this process, limited patient data from the target hospital, both labeled and unlabeled, are incorporated during the feature adaptation process, thereby contributing to better model performance. Without patient-level data sharing, the proposed model learning strategy which overcomes feature misalignment and distribution divergence, enables the multi-source transfer learning process in the case of insufficient and unlabeled patient data at target hospital.

Results

The effectiveness of the proposed transfer learning model was evaluated on a collaborative research network of colorectal cancer patients in the US and China. The results demonstrate that the proposed model can achieve much better performance for predicting target risk with limited resources on patient data than baseline models      . Better discrimination and calibration ability are also observed when sufficient labeled data are not available in the target hospital for prognosis prediction tasks      . Further exploratory experiments show that the proposed approach exhibits good model generalizability regardless of the data heterogeneity. With the help of the SHapley Additive exPlanations for model interpretation, the effectiveness of incorporating hospital-specific features in the transfer learning model is shown.

Conclusions

In this study, the proposed method can develop prediction models from multiple source hospitals and exhibit good performance by leveraging cross-domain hospital-specific feature information, therefore enhancing the model prediction when applied to single medical institution with limited patient data.

Introduction

The quality of healthcare decision-making depends heavily on the validity and reliability of clinical decision support (CDS) models [1], [2], while the statistical significance outcome of which hinges upon the quality of medical data that support model establishment [3]. Therefore, a data-driven prediction model developed based on high-quality large-scale electronic health record (EHR) patient data can provide a more accurate basis for clinical decision-makers than other models. However, single clinical institution is often unable to collect sufficient sample sizes and adequate amounts of labeled data due to the limited scale. Poor-quality data have led to the inadequacy of single-center models to support a persuasive and powerful conclusion, along with an inability to obtain improved predictive modeling results of patient outcomes [4], [5].

To overcome this challenge, an increasing amount of research has sought to expand the scope from one institution to multiple centers to mitigate the limitation caused by the lack of available training samples [6], [7], [8]. The accessibility of multi-source data has been improved by using health information exchange standards such as Fast Healthcare Interoperability Resources (FHIR) [9] or the Observational Medical Outcomes Partnership (OMOP) common data model (CDM) [10], which solve the interoperability problem of healthcare information systems caused by the diversity of clinical decision support systems (CDSS) and EHR [11], [12]. After the patient-level data are accessed through a standardized information sharing interface by concurrently using the ‘same language’, the data from multiple sources can be used to support the clinical research in the multicenter clinical research network.

In multicenter collaborative modeling, the recently proposed transfer learning (TL) technique [13] has demonstrated its effectiveness in transferring model (s) learned from external institution(s) to local system. Transfer learning is a machine learning method that leverages the evidence from the ‘sources’ in one or more related tasks to a different but related task in the ‘target’. TL aims to reduce the distribution discrepancy across domains, thereby ensuring effective prediction on the target domain after the transfer. It appears to be a widely used solution in various applications, such as image annotation [14] and concept extraction [15], etc.

However, there are two aspects that distinguish healthcare-related problems from applications in other domains during model transfer. First, most research on transfer learning has assumed that all data are under the same feature space [16], [17], [18]; however, in medical practice, such an assumption does not always hold because the features often lie in the distinct but overlapping space [19] (see Fig. 1). Many of the observational variables can be hospital-specific due to factors such as different healthcare facility levels among different medical institutions [20] or missing records in the electronic medical system, etc., which are all included in the EHR data. The discrepancy of feature space is commonly neglected in multicenter studies [21], which prevents the hospital-specific features from the source and target domains from being effectively used and thus lowers the potential predictive capabilities of the model.

Another characteristic in model transfer for medical-related issues is that the available labeled data is typically limited. A large amount of unlabeled data is often found in the training data from the target hospital [22]. Collecting labeled patient data is difficult and expensive, especially when the presence of the outcome of interest is hardly observable. For example, in some fast-growing malignant cancers, symptom detection is arduous and the progression of the disease is rapid, so consistent data collection and expensive expert labeling in prognosis prediction are needed [23], [24]. This will impose difficulty on model learning in the fully supervised fashion and thus hinder the model from achieving good performance when predicting on its own cohort [25].

In this study, we propose a multicenter hybrid semi-supervised transfer learning model (MHSTL) to improve prediction performance for medical institution with limited patient data. The hospital-specific data are used through hybrid transfer learning to implement the feature alignment and further solve the problems arising from feature space heterogeneity and data distribution divergence. Meanwhile, a model updating approach that integrates the unlabeled target data is used to enhance the training process and resolve the issue of poor prediction due to the insufficient labels on a single center. The efficient SHapley Additive exPlanations (SHAP) approach [26] was used in this study to improve the interpretability of predictive models. The predictive model acquired from multicenter collaborative model learning provides an alternative solution to enhance the predictive power of CDS models for medical institutions with limited patient data, thereby overcoming the challenges for clinical data analytics in low-resource scenarios.