Today's medical institutions produce enormous amounts of data on patients, including multimedia data, which is increasingly produced in digital form. These data in their clinical context contain much information and experience that is currently not being used up to its full potential. Through the digital form the data has become accessible for automatic analysis and treatment for a variety of applications. At the same time, the variety of images produced can be confusing even for trained specialists causing an information overload exists for many medical doctors. This suggests that content-based image retrieval can be a valuable tool for helping manage these data and access the right information at the right time. This article gives a short state of the art of content-based medical image retrieval followed by a description of the medGIFT project on image retrieval with its main components. Then, several challenges are used to illustrate areas where much more work is currently needed to advance biomedical image retrieval. This shows that we have now progresses beyond the phase, where medical doctors transfer a database to computer scientists to only evaluate their algorithms. We conclude that visual information retrieval can have a real impact in the medical field if the techniques can adapt to this rapidly changing field and get integrated into the workflow in radiology and other medical fields.