Abstract:
In real-world scenarios, the information quality provided by RGB and thermal (RGB-T) sensors often varies across samples. This variation will negatively impact the perfor...Show MoreMetadata
Abstract:
In real-world scenarios, the information quality provided by RGB and thermal (RGB-T) sensors often varies across samples. This variation will negatively impact the performance of semantic segmentation models in utilizing complementary information from RGB-T modalities, resulting in a decrease in accuracy and fusion credibility. Dynamically estimating the uncertainty of each modality for different samples could help the model perceive such information quality variation and then provide guidance for a reliable fusion. With this in mind, we propose a novel uncertainty-guided trustworthy fusion network (UTFNet) for RGB-T semantic segmentation. Specifically, we design an uncertainty estimation and evidential fusion (UEEF) module to quantify the uncertainty of each modality and then utilize the uncertainty to guide the information fusion. In the UEEF module, we introduce the Dirichlet distribution to model the distribution of the predicted probabilities, parameterized with evidence from each modality and then integrate them with the Dempster–Shafer theory (DST). Moreover, illumination evidence gathering (IEG) and multiscale evidence gathering (MEG) modules by considering illumination and target multiscale information, respectively, are designed to gather more reliable evidence. In the IEG module, we calculate the illumination probability and model it as the illumination evidence. The MEG module can collect evidence for each modality across multiple scales. Both qualitative and quantitative results demonstrate the effectiveness of our proposed model in accuracy, robustness, and trustworthiness. The code will be accessible at https://github.com/KustTeamWQW/UTFNet.
Published in: IEEE Geoscience and Remote Sensing Letters ( Volume: 20)