Ada-CCFNet: Classification of multimodal direct immunofluorescence images for membranous nephropathy via adaptive weighted confidence calibration fusion network

Highlights

• An adaptive weighted confidence calibrated fusion network (Ada-CCFNet) is designed to classify early, late and non-membranous nephropathy.

• Classify membranous nephropathy stages by multi-modal direct immunofluorescence images.

• A preprocessing method to segment glomerular region effectively and standardize luminance.

• An adaptive weighted method by Excepted Calibration Error reductions is proposed for fusion.

• Our method provides higher classification accuracy than the compared methods.

Abstract

In the pathological diagnosis of early, late and non-membranous nephropathy, direct immunofluorescence is highly likely to present potentially specific lesions, while it is often overlooked due to the difficulty of screening with naked eyes. With the advanced progress of deep learning, they have shown powerful abilities in detecting potential lesions. In this paper, we propose an adaptive weighted confidence calibration fusion framework (Ada-CCFNet) consisting of a preprocessing module, an adaptive weighted confidence calibration fusion (Ada-CCF) module and a classification module for diagnosis of membranous nephropathy by classifying the multimodal direct immunofluorescence images. In the preprocessing module, we use the well-known U-Net to segment individual glomeruli and standardize their luminance appearance by the average luminance difference method, allowing the subsequent modules to focus more on the diseased glomerular region. Subsequently, in the Ada-CCF module, six confidence calibration methods are utilized for two main direct immunofluorescence images, IgG and C3, and the comprehensive calibration scores are obtained based on the adaptive weighted fusion of six confidence calibration methods to obtain more reliable confidence level, in which the adaptive weights are related with expected calibration error reductions. For the classification module, the weighted probability scores of IgG and C3 are jointly fed into the module to achieve the classification by random forest. Experimental results showed that Ada-CCFNet achieves the classification accuracy of 73.52%, surpassing the methods of using single IgG or C3 images and positive grade indicator with 8.24%, 8.94% and 22.76%, and outperforming the compared methods in the classification of membranous nephropathy.

Introduction

Membranous nephropathy (MN), as the second most common chronic kidney disease (CKD) in China, accounts for about 23.4% of all CKD patients (Xu et al., 2016), in which approximately 30% of patients may develop uremia within 10 years (Gañán et al., 2022). Since the pathology of MN is characterized by the presence of deposits of subepithelial immune complexes and the corresponding structural changes (Gañán et al., 2022), the diagnosis of MN is usually made by a renal pathology examination known as the gold standard of diagnosis. The examination of renal pathology consists mainly of electron microscopy, light microscopy and immunofluorescence (Gañán et al., 2022). And immunofluorescence examination of MN often adopts direct immunofluorescence (DIF), in which antibody-containing fluorescein is combined with antigen to obtain an immunofluorescent sample for analysis under the fluorescence microscope and preservation (Jain et al., 2021). Immunofluorescence in MN typically shows high intensity granular deposition of antibody IgG and complement C3 along the capillary wall (Ronco and Debiec, 2015). In the early stage of MN, deposits of immune complexes are just forming, and the subepithelial immune complex deposits are not yet separated by the projection spikes of the glomerular basement membrane (GBM), which manifest as subepithelial immune complex deposits in the glomerulus. In the late stage of MN, the formation of projection spikes becomes increasingly evident, and the deposits are eventually incorporated completely into the GBM, many of which may be partially or completely reabsorbed (Huang et al., 2013). The traditional diagnosis is often based on the severity of GBM lesions in images under electron microscopy or light microscopy, while DIF images of the kidney are often considered unimportant and slighted by pathologists during renal biopsy analysis of MN because DIF images of the kidney rarely show specificity in the pathological staging of MN. However, DIF can distinguish the early stage of MN from non-MN such as minimal change glomerulonephropathy by positive antibody fluorescence presentation, although it is similar in clinical presentation or other pathological examinations. In the early stage of MN pathology, the patient’s clinical manifestations are not obvious and usually cannot be detected by clinical data alone, and the glomeruli are not detected by electron microscopic, while immunofluorescence lesions are significant, and immunoproteins and complements are seen to be scattered along the GBM, showing a positive reaction, so DIF images are more sensitive to early MN lesions. Meanwhile, in the late stage of MN, the electronic accumulations under electron microscopy are often contained in the GBM and are too difficult to discern, whereas immunofluorescence often shows the negative reaction. Besides, in the medical diagnosis of MN, multiple direct immunofluorescence images from renal biopsy are used primarily for comprehensive analysis, which ensures the reliability of diagnosis. Clinically, IgG and C3, as the two main immunofluorescence images from renal biopsy, are often analyzed together to obtain complementary information about the lesion. In summary, DIF images can also be helpful in the classification of early MN, late MN and non-MN pathology to some extent.

In recent years, deep learning (DL) and machine learning (ML) methods have proven to be very effective in advancing the field of medical image analysis. They can analyze images and find relevant lesions that are difficult to detect with the naked eyes by computer techniques. Several articles have been published describing the applications of DL and ML algorithms in cancer (Esteva et al., 2017, Xu et al., 2019, Zhou et al., 2020, Cuocolo et al., 2019), Alzheimer’s disease (Venugopalan et al., 2021, Qiu et al., 2020), eye diseases (Sarki et al., 2020, Asaoka et al., 2019, Jammal et al., 2020), and other medical imaging fields. As a class of important diseases, renal diseases have also appeared in a large number of related researches. However, there are few studies on DIF of renal biopsies, and mostly for the analysis of certain specific lesions. Ligabue et al. (2020) described the “appearance”, “distribution”, “location” and “intensity” of glomerular deposits in the DIF renal biopsy and achieved more than 90% classification accuracy in “fine particles” and “continuous regular capillary wall”, but had a high rate of missed detections in, for example, “segmental”, which may have serious implications for medical analysis. Zhang et al. (2022) used a multiple attention network to identify the location and pattern of antibody protein depositions within the glomerulus in renal biopsy DIF, but it is problematic for a more refined classification due to the small amount of data. Pollastri et al. (2021) introduced a temperature scaling calibration technique to classify the location of intra-glomerular antibody depositions in renal biopsy of DIF, which did not improve the low accuracy of the original model, although it effectively enhanced the interpretability of the model. Direct diagnosis of disease using DIF images is even more difficult. Only Kitamura et al. (2020) achieved the identification of diabetic nephropathy using six types of DIF images and comprehensive analysis by convolutional neural network feature fusion, but its accuracy still needs to be improved. Similar to diabetic nephropathy, the diagnosis of MN requires a comprehensive analysis using multiple modal immunofluorescence images, in which IgG and C3, as common diagnostic indicators in medicine, show more obvious lesion characteristics than other modal data and are significant for the staging of MN. However, there have been no studies on the DIF-based classification of early, late and non-MN before. Therefore, there is still considerable room for research on whether DL and ML can use multi-modal DIF images to classify early MN, late MN and non-MN and to detect lesions that are difficult to detect by human physicians.

In this paper, we study the classification of early MN, late MN and non-MN based on two different modalities of DIF images, IgG and C3, which are the main and important data in clinical diagnosis and have not been jointly analyzed by any researcher before to the best of our knowledge. To address this problem, we propose an adaptive weighted confidence calibration fusion network (Ada-CCFNet) framework for the MN classification of IgG and C3 DIF images. The proposed Ada-CCFNet comprises three modules that involve preprocessing, adaptive weighted confidence calibration fusion (Ada-CCF), and classification. In the preprocessing module, we use a segmentation network such as the U-Net (Ronneberger et al., 2015) to extract individual glomeruli from DIF images and correct the luminance of origin glomeruli as standardized images by using the average luminance difference method to eliminate the relative effect of background luminance on glomerular luminance. In the Ada-CCF module, we first generate the initial class probability vectors for IgG and C3 using pretrained ResNeSt-50 (Zhang et al., 2020), respectively, and then calibrate the initial probabilities by multiple confidence calibration methods and adaptive weighted these confidence calibration probabilities according to the expected calibration error (ECE) (Guo et al., 2017) reductions of different calibration methods to obtain a comprehensive probability score that simulates the true judgment. In the final classification module, the classification is performed on the comprehensive probability scores of IgG and C3 using the classical random forest (Breiman, 2001) algorithm. The main contributions of this paper are summarized as follows.

• An adaptive weighted confidence calibrated fusion network (Ada-CCFNet) is designed using adaptive weighted confidence calibration fusion method based on multimodal DIF images, which achieves significant classification accuracy close to the real judgment level of MN.

• A preprocessing method that includes an effective segmentation of the glomerular region and standardization of the luminance is utilized to eliminate the influence of background luminance and interference on the glomerular itself.

• The adaptive weighted confidence level calibration method uses the ECE reductions of different calibration methods as weighting parameters to obtain a comprehensive confidence level calibration, which can extreme the advantage of various confidence level calibration methods.

• The proposed method can effectively and accurately classify early MN, late MN and non-MN based on two main DIF images of renal biopsy, greatly assisting pathologists in diagnosis, and providing support for more timely and effective treatment.

The remainder of this paper is organized as follows. A brief synopsis of related work is provided in Section 2, and Section 3 detailed the proposed adaptive weighted confidence calibration fusion network (Ada-CCFNet). The results of ablation experiments and comprehensive experiments are presented in Section 4. Finally, Section 5 summarizes the work of this paper.