PMT: Partial-Modality Translation Based on Diffusion Models for Prostate Magnetic Resonance and Ultrasound Image Registration

Abstract

In diagnosing prostate cancer, urologists frequently rely on the Magnetic Resonance Image (MRI) to pinpoint suspicious areas, followed by biopsies guided by ultrasound (US). Aligning MRI and US images is necessary during the procedure, and this task is currently performed manually by highly skilled experts only. Additionally, the notable differences between MR and US images also pose challenges for AI algorithms in the registration process. Therefore, this paper proposes a novel method aimed at bridging the modality gap between MR and US images, thereby easing the registration problem associated with them. We combine the strengths of a 3D Diffusion Model and a Generative Adversarial Network (GAN) to autonomously translate both MR and US images into an intermediate pseudo modality. This marks the initial endeavor to achieve such partial modality translation between 3D MR and US images. Compared to existing state-of-the-art fully modality translation techniques, our method visibly preserves the original image details while producing images with more similar textures. This improvement is objectively evident in our 33.67% reduction in the Fréchet Inception Distance (FID), which is more than double the 15.86% reduction achieved by the existing method. Additionally, our method achieves an impressive 54.90% reduction in the Kernel Inception Distance (KID), surpassing the 17.65% reduction attained by existing methods by more than threefold. Furthermore, we also provide evidence to illustrate that these enhancements significantly improve the effectiveness of the downstream registration task. Additionally, by exclusively employing modality-translated results to derive the warping map, along with conducting the actual warping on the original images, we effectively address the well-known “hallucination issue” in AI-generated medical images.