Skip to main content
SpringerLink
Log in

A fully automatic end-to-end method for content-based image retrieval of CT scans with similar liver lesion annotations

  • Original Article
  • Published:
International Journal of Computer Assisted Radiology and Surgery Aims and scope Submit manuscript

Abstract

Purpose

The goal of medical content-based image retrieval (M-CBIR) is to assist radiologists in the decision-making process by retrieving medical cases similar to a given image. One of the key interests of radiologists is lesions and their annotations, since the patient treatment depends on the lesion diagnosis. Therefore, a key feature of M-CBIR systems is the retrieval of scans with the most similar lesion annotations. To be of value, M-CBIR systems should be fully automatic to handle large case databases.

Methods

We present a fully automatic end-to-end method for the retrieval of CT scans with similar liver lesion annotations. The input is a database of abdominal CT scans labeled with liver lesions, a query CT scan, and optionally one radiologist-specified lesion annotation of interest. The output is an ordered list of the database CT scans with the most similar liver lesion annotations. The method starts by automatically segmenting the liver in the scan. It then extracts a histogram-based features vector from the segmented region, learns the features’ relative importance, and ranks the database scans according to the relative importance measure. The main advantages of our method are that it fully automates the end-to-end querying process, that it uses simple and efficient techniques that are scalable to large datasets, and that it produces quality retrieval results using an unannotated CT scan.

Results

Our experimental results on 9 CT queries on a dataset of 41 volumetric CT scans from the 2014 Image CLEF Liver Annotation Task yield an average retrieval accuracy (Normalized Discounted Cumulative Gain index) of 0.77 and 0.84 without/with annotation, respectively.

Conclusions

Fully automatic end-to-end retrieval of similar cases based on image information alone, rather that on disease diagnosis, may help radiologists to better diagnose liver lesions.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6

Similar content being viewed by others

References

  1. Horton KM, Bluemke DA, Hruban RH, Soyer P, Fishman EK (1999) CT and MR imaging of benign hepatic and biliary tumors. Radio Gr 19:431–451

    CAS  Google Scholar 

  2. Pandey P, Lewis H, Pandey A, Schmidt C, Dillhoff M, Kamel IR, Pawlik TM (2017) Updates in hepatic oncology imaging. Surg Oncol 26:195–206

    Article  PubMed  Google Scholar 

  3. DeVita VT, Lawrence TSRS (2011) DeVita, Hellman, and Rosenberg’s cancer: principles and practice of oncology. Lippincott Williams & Wilkins, Philadelphia

    Google Scholar 

  4. Assy N, Nasser G, Djibre A, Beniashvili Z, Elias S, Zidan J (2009) Characteristics of common solid liver lesions and recommendations for diagnostic workup. World J Gastroenterol 15:3217–27

    Article  PubMed  PubMed Central  Google Scholar 

  5. Oliver JH, Baron RL (1996) Helical biphasic contrast-enhanced CT of the liver: technique, indications, interpretation, and pitfalls. Radiology 201:1–14

    Article  PubMed  Google Scholar 

  6. MacDonald SL, Cowan IA, Floyd R, Mackintosh S, Graham R, Jenkins E, Hamilton R (2013) Measuring and managing radiologist workload: application of lean and constraint theories and production planning principles to planning radiology services in a major tertiary hospital. J Med Imaging Radiat Oncol 57:544–550

    Article  PubMed  Google Scholar 

  7. Zhao B, Tan Y, Bell DJ, Marley SE, Guo P, Mann H, Scott MLJ, Schwartz LH, Ghiorghiu DC (2013) Exploring intra- and inter-reader variability in uni-dimensional, bi-dimensional, and volumetric measurements of solid tumors on CT scans reconstructed at different slice intervals. Eur J Radiol 82:959–968

    Article  PubMed  Google Scholar 

  8. Suzuki C, Torkzad MR, Jacobsson H, Åström G, Sundin A, Hatschek T, Fujii H, Blomqvist L (2010) Interobserver and intraobserver variability in the response evaluation of cancer therapy according to RECIST and WHO-criteria. Acta Oncol (Madr) 49:509–514

    Article  Google Scholar 

  9. Dankerl P, Cavallaro A, Tsymbal A, Costa MJ, Suehling M, Janka R, Uder M, Hammon M (2013) A retrieval-based computer-aided diagnosis system for the characterization of liver lesions in CT scans. Acad Radiol 20:1526–34

    Article  PubMed  Google Scholar 

  10. Müller H, Michoux N, Bandon D, Geissbuhler A (2004) A review of content-based image retrieval systems in medical applications-clinical benefits and future directions. Int J Med Inform 73:1–23

    Article  PubMed  Google Scholar 

  11. Akgül CB, Rubin DL, Napel S, Beaulieu CF, Greenspan H, Acar B (2011) Content-based image retrieval in radiology: current status and future directions. J Digit Imaging 24:208–22

    Article  PubMed  Google Scholar 

  12. Liu Y, Zhang D, Lu G, Ma WY (2007) A survey of content-based image retrieval with high-level semantics. Pattern Recognit Lett 40:262–282

    Article  Google Scholar 

  13. Grosky WI (2002) Narrowing the semantic gap—improved text-based web document retrieval using visual features. IEEE Trans Multimed 4:189–200

    Article  Google Scholar 

  14. Deselaers T, Deserno TM, Müller H (2008) Automatic medical image annotation in ImageCLEF 2007: overview, results, and discussion. Pattern Recognit Lett 29:1988–1995

    Article  Google Scholar 

  15. van Ginneken B, Schaefer-Prokop CM, Prokop M (2011) Computer-aided diagnosis: how to move from the laboratory to the clinic. Radiology 261:719–32

    Article  PubMed  Google Scholar 

  16. Ponciano-Silva M, Souza JP, Bugatti PH, Bedo MVN, Kaster DS, Braga RT V, Bellucci \({\hat{A}}\)D, Azevedo-Marques PM, Traina C, Traina AJM (2013) Does a CBIR system really impact decisions of physicians in a clinical environment? In: Proceedings of the 26th IEEE international symposium on computer based medical system CBMS 2013, pp 41–46

  17. An C, Rakhmonova G, Choi JY, Kim MJ (2016) Liver imaging reporting and data system (LI-RADS) version 2014: understanding and application of the diagnostic algorithm. Clin Mol Hepatol 22:296–307

    Article  PubMed  PubMed Central  Google Scholar 

  18. Mitchell DG, Bruix J, Sherman M, Sirlin CB (2015) LI-RADS (Liver Imaging Reporting and Data System): Summary, discussion, and consensus of the LI-RADS Management Working Group and future directions. Hepatology 61:1056–1065

    Article  PubMed  Google Scholar 

  19. Napel SA, Beaulieu CF, Rodriguez C, Cui J, Xu J, Gupta A, Korenblum D, Greenspan H, Ma Y, Rubin DL (2010) Automated retrieval of CT images of liver lesions on the basis of image similarity: method and preliminary results. Radiology 256:243–252

    Article  PubMed  PubMed Central  Google Scholar 

  20. Depeursinge A, Kurtz C, Beaulieu C, Napel S, Rubin D (2014) Predicting visual semantic descriptive terms from radiological image data: preliminary results with liver lesions in CT. IEEE Trans Med Imaging 33:1669–1676

    Article  PubMed  PubMed Central  Google Scholar 

  21. Kumar A, Dyer S, Kim J, Li C, Leong PHW, Fulham M, Feng D (2016) Adapting content-based image retrieval techniques for the semantic annotation of medical images. Comput Med Imag Gr 49:37–45

    Article  Google Scholar 

  22. Xu J, Napel S, Greenspan H, Beaulieu CF, Agrawal N, Rubin DL (2012) Quantifying the margin sharpness of lesions on radiological images for content-based image retrieval. Med Phys 39:5405

    Article  PubMed  PubMed Central  Google Scholar 

  23. Spanier AB, Joskowicz L (2014) Towards content-based image retrieval?: From computer generated features to semantic descriptions of liver CT scans. In: CLEF online work, notes. pp 438–447

  24. Spanier AB, Joskowicz L (2017) Automatic Atlas-free multi-organ segmentation of contrast-enhanced CT scans. In: Hanbury A, Müller H, Langs G (eds) Cloud-Based Benchmarking Med Image Anal. Springer, Berlin, pp 145–164

    Chapter  Google Scholar 

  25. Achanta R, Shaji A, Smith K, Lucchi A, Fua P, Süsstrunk S (2011) SLIC superpixels compared to state-of-the-art superpixel methods. Pattern Anal Mach Intell IEEE Trans 34:2274–2282

    Article  Google Scholar 

  26. Cha SH, Srihari SN (2002) On measuring the distance between histograms. Pattern Recognit 35:1355–1370

    Article  Google Scholar 

  27. Marvasti NB, Kökciyan N, Türkay R, Yazici A, Yolum P, Üsküdarl S, Acar B (2014) ImageCLEF liver CT image annotation task 2014. In: CLEF (working notes), pp 329–340

  28. Yolum P, Üsküdarl S, Acar B (2014) ImageCLEF liver CT image annotation task 2014. In: CLEF online work, notes, pp 329–340

  29. Pedregosa F, Grisel O, Weiss R, Passos A, Brucher M (2011) Scikit-learn?: Machine Learning in Python. J Mach Res 12:2825–2830

    Google Scholar 

  30. Kalervo J, Kekäläinen J (2002) Cumulated gain-based evaluation of IR techniques. ACM Trans Inf Syst 20(4):422–446

  31. Spanier AB, Cohen D, Joskowicz L (2017) A new method for the automatic retrieval of medical cases based on the RadLex ontology. Int J Comput Assist Radiol Surg 12:471–484

    Article  CAS  PubMed  Google Scholar 

  32. Chapelle O, Haffner P, Vapnik VN (1999) Support vector machines for histogram-based image classification. IEEE Trans Neural Netw 10:1055–1064

    Article  CAS  PubMed  Google Scholar 

  33. Echegaray S, Gevaert O, Shah R, Kamaya A, Louie J, Kothary N, Napel S (2015) Core samples for radiomics features that are insensitive to tumor segmentation: method and pilot study using CT images of hepatocellular carcinoma. J Med Imaging 2:41011

    Article  Google Scholar 

  34. Järvelin K, Kekäläinen J (2002) Cumulated gain-based evaluation of IR techniques. ACM Trans Inf Syst 20:422–446

    Article  Google Scholar 

Download references

Acknowledgements

This research was supported in part by Israel Ministry of Science, Technology and Space, Grant 53681, 2016-19, and by the Oppenheimer Applied Research Grant, The Hebrew University, TUBITAK ARDEB Grant No. 110E264, 2015-16.

Author information

Authors and Affiliations

  1. The Rachel and Selim Benin School of Computer Science and Engineering, The Hebrew University of Jerusalem Givat Ram Campus, Jerusalem, 91904, Israel

    A. B. Spanier & L. Joskowicz

  2. Alexander Grass Center for Bioengineering, The Hebrew University of Jerusalem, Jerusalem, Israel

    A. B. Spanier

  3. Department of Radiology, Hadassah Hebrew University Medical Center, Jerusalem, Israel

    N. Caplan & J. Sosna

  4. Department of Electrical-Electronics Engineering, Bogazici University, Istanbul, Turkey

    B. Acar

Authors
  1. A. B. Spanier
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. N. Caplan
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. J. Sosna
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. B. Acar
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. L. Joskowicz
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to A. B. Spanier.

Ethics declarations

Conflict of interest

None of the authors has any conflict of interest. The authors have no personal financial or institutional interest in any of the materials, software or devices described in this article.

Human and animal rights

No animals or humans were involved in this research. All scans were anonymized before delivery to the researchers.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Spanier, A.B., Caplan, N., Sosna, J. et al. A fully automatic end-to-end method for content-based image retrieval of CT scans with similar liver lesion annotations. Int J CARS 13, 165–174 (2018). https://doi.org/10.1007/s11548-017-1687-1

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11548-017-1687-1

Keywords

Search

Navigation