Inter- and Intra-Observer Variability of Radiologists Evaluating CBIR Systems

Abstract

The purpose of the study is to evaluate the inter- and intra-observer variability of the radiologists in evaluation of the similarity between the query and retrieved ulatasound images containing breast masses by the Content-Based Image Retrieval (CBIR) CADx system. Three radiologists rated the similarity between the query masses and the computer-retrieved (ED-CBIR) masses. Three CBIR systems based on each radiologist’s subjective similarity ratings (R-CBIRs) were formed and compared with the ED-CBIR. The intra-observer variability was smaller than the inter-observer variability for all three radiologists. The radiologists’ performance with the R-CBIRs produced similarity ratings results close to the radiologists’ performance with the ED-CBIR. The average difference in classification accuracy (Az) between the ED-CBIR and the R-CBIRs was slightly lower than the average difference in Az between the R-CBIRs. The relatively large intra- and inter-observer variability may make more difficult to evaluate the effect of the CBIR CADx systems on radiologists’ performance.