Authors:
Petra Takacs
1
;
Richard Zsamboki
1
;
Elod Kiss
1
and
Ferdinand Dhombres
2
Affiliations:
1
GE HealthCare Hungary, Budapest, Hungary
;
2
Sorbonne University, INSERM Limics, GRC26, Armand Trousseau Hospital, APHP Paris, France
Keyword(s):
AI, Ultrasound, Image Similarity Search, Comparison, Vision Transformer, ResNet.
Abstract:
Querying similar images from a database to a reference image is an important task with multiple possible use-cases in healthcare industry, including improving labelling processes, and enhancing diagnostic support to medical professionals. The aim of this work is to measure the performance of different artificial neural networks, comparing their ability to identify clinically relevant similar images based on their generated feature sets. To measure the clinical relevance, metrics using expert labels of organs and diagnoses on the images were calculated, and image similarity was further confirmed by pixel metrics. Images with organ and diagnosis labels were selected from a dataset of early-stage pregnancy and 2nd -3rd trimester pregnancy ultrasound images respectively for the measurements. The networks were chosen from state-of-the-art foundational models trained on natural images, DINO and DINOv2, SAM2, and DreamSim. The best performing model based on our experiments is DreamSim for o
rgan matches, and DINO for diagnosis matches. A simple ResNet trained on the mentioned early pregnancy dataset for organ classification was also added to the selection. ResNet performs best for early pregnancy organ matches, therefore finetuning a robust encoder on our own dataset is a promising future step to further enhance medically relevant similar image search.
(More)