Skip to main content
SpringerLink
Log in

Interactive cross and multimodal biomedical image retrieval based on automatic region-of-interest (ROI) identification and classification

  • Regular Paper
  • Published:
International Journal of Multimedia Information Retrieval Aims and scope Submit manuscript

Abstract

In biomedical articles, authors often use annotation markers such as arrows, letters, or symbols overlaid on figures and illustrations to highlight ROIs. These annotations are then referenced and correlated with concepts in the caption text or figure citations in the article text. This association creates a bridge between the visual characteristics of important regions within an image and their semantic interpretation. Identifying these assists in extracting ROIs that are likely to be highly relevant to the discussion in the article text. The aim of this work is to perform semantic search without knowing the concept keyword or the specific name of the visual pattern or appearance. We consider the problem of cross and multimodal retrieval of images from articles which contains components of text and images. Our proposed method localizes and recognizes the annotations by utilizing a combination of rule-based and statistical image processing techniques. The image regions are then annotated for classification using biomedical concepts obtained from a glossary of imaging terms. Similar automatic ROI extraction can be applied to query images, or to interactively mark an ROI. As a result, visual characteristics of the ROIs can be mapped to text concepts and then used to search image captions. In addition, the system can toggle the search process from purely perceptual to a conceptual one (crossmodal) based on utilizing user feedback or integrate both perceptual and conceptual search in a multimodal search process. The hypothesis, that such approaches would improve biomedical image retrieval, was validated through experiments on a biomedical article dataset of thoracic CT scans.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15

Similar content being viewed by others

Notes

  1. http://www.ncbi.nlm.nih.gov/pubmed/.

  2. http://www.nlm.nih.gov/research/umls/.

  3. http://www.rsna.org/RadLex.aspx.

  4. http://openi.nlm.nih.gov/.

  5. http://www.ncbi.nlm.nih.gov/pmc/.

  6. http://imageclef.org.

  7. http://goldminer.arrs.org.

  8. http://krauthammerlab.med.yale.edu/imagefinder/.

  9. http://imageclef.org.

References

  1. Demner-Fushman D, Antani SK, Thoma GR (2007) Automatically finding images for clinical decision support. In: Proceedings of workshop on data mining in medicine, 7th IEEE international conference on data mining, pp 139–144

  2. Demner-Fushman D, Antani SK, Simpson M, Thoma GR (2009) Annotation and retrieval of clinically relevant images. Int J Med Inform 78(12):e59–e67

    Article  Google Scholar 

  3. Datta R, Joshi D, Li J, Wang JZ (2008) Image retrieval: ideas, influences, and trends of the new age. ACM Comput Surv 40(2):1–60

    Article  Google Scholar 

  4. Simpson M, You D, Rahman MM, Demner-Fushman D, Antani SK, Thoma GR (2012) Towards the creation of a visual ontology of biomedical imaging entities. In: AMIA 2012 Annual Symposium, Nov. 3–7, Chicago, IL

  5. Pallisa E, Sanz P, Roman A, Maj J, Andreu J, Cceres J (2002) Lymphangioleiomyomatosis: pulmonary and abdominal findings with pathologic correlation. Radiographics 22:S185–S198

    Article  Google Scholar 

  6. Rahman MM, Antani SK, Thoma GR (2009) A medical image retrieval framework in correlation enhanced visual concept feature space. In: 22nd IEEE international symposium on computer-based medical systems (CBMS), Albuquerque, New Mexico, USA

  7. Zhou XS, Huang TS (2003) Relevance feedback for image retrieval: a comprehensive review. Multimed Syst 8(6):536–544

    Article  Google Scholar 

  8. Rui Y, Huang TS (1999) Relevance feedback: a power tool for interactive content-based image retrieval. IEEE Circuits Syst Video Technol 8(5):644–655

    Google Scholar 

  9. Demner-Fushman D, Antani SK, Simpson M, Thoma GR (2012) Design and development of a multimodal biomedical information retrieval system. JCSE 6(2):168–177

    Google Scholar 

  10. Mori Y, Takahashi H, Oka R (1999) Image-to-word transformation based on dividing and vector quantizing images with words. In: MISRM’99 first international workshop on multimedia intelligent storage and retrieval management

  11. Duygulu P, Barnard K, de Freitas N, Forsyth D (2002) Object recognition as machine translation: learning a lexicon for a fixed image vocabulary. In: Seventh European conference on computer vision, pp 97–112

  12. Yang Y, Zhuang Y, Wu F, Pan Y (2008) Harmonizing hierarchical manifolds for multimedia document semantics understanding and cross-media retrieval. IEEE Trans Multimed 10(3):437–446

    Article  Google Scholar 

  13. Jeon J, Lavrenko V, Manmatha R (2003) Automatic image annotation and retrieval using cross-media relevance models. In: Proceeding SIGIR ’03 Proceedings of the 26th annual international ACM SIGIR conference on research and development in informaion retrieval, pp 119–126

  14. Rasiwasia N, Costa Pereira J, Coviello E, Doyle G, Lanckriet GRG, Levy R, Vasconcelos N (2010) A new approach to cross-modal multimedia retrieval. In: Proceedings of the international conference on multimedia, MM ’10, pp 251–260

  15. Datta R, Joshi D, Li J, Wang J (2008) Image retrieval: ideas, influences, and trends of the new age. ACM Comput Surv (CSUR) 40(2):1–60

  16. Chen N (2006) A survey of indexing and retrieval of multimodal documents: text and images. Technical Report, 2006–505, Queen’s University

  17. Kalpathy-Cramer J, Müller H, Bedrick S, Eggel I, Garcia Seco de Herrera A, Tsikrika T (2011) Overview of the CLEF 2011 medical image classification and retrieval tasks. CLEF 2011 Working Notes, Amsterdam, The Netherlands

  18. Müller H, Herrera A, Kalpathy-Cramer J, Demner-Fushman D, Antani S, Ivan E (2012) Overview of the ImageCLEF 2012 medical image retrieval and classification tasks. In the Working Notes for the CLEF 2012 Labs and Workshop, 17–20 September. Rome, Italy, Amsterdam

  19. Müller H, Kalpathy-Cramer J, Eggel I, Bedrick S, Reisetter J Jr., Kahn CE Jr., Hersh WR (2010) Overview of the CLEF 2010 medical image retrieval track. CLEF 2012 Evaluation Labs and Workshop, Online Working Notes, Padua, Italy, September 20–23

  20. You D, Antani SK, Demner-Fushman D, Rahman MM, Govindaraju V, Thoma GR (2011) Biomedical article retrieval using multimodal features and image annotations in region-based CBIR. In: Document recognition and retrieval XVIII. Proceedings of SPIE, vol 7874, p 7874 0K

  21. Zhang D, Lu G (2003) A comparative study on shape retrieval using fourier descriptors with different shape signatures. J Vis Commun Image Represent 14(1):41–60

    Article  Google Scholar 

  22. Sluimer IC, Schilham S, Prokop M, van Ginneken B (2006) Computer analysis of computed tomography scans of the lung: a survey. IEEE Trans Med Imaging 25(4):385–405

  23. Sluimer IC, van Waes PF, Viergever MA, van Ginneken B (2003) Computeraided diagnosis in high resolution ct of the lungs. Med Phys 30(12):3081–3090

    Article  Google Scholar 

  24. Haralick RM, Shanmugam K, Dinstein I (1973) Textural features for image classification. IEEE Trans Syst Man Cybern 3:610–621

    Article  Google Scholar 

  25. Srinivasan GN, Shobha G (2008) Statistical texture analysis. Proc World Acad Sci Eng Technol 36. ISSN 2070–3740

  26. Zhang D, Wong A, Indrawan M, Lu G (2000) Content-based image retrieval using gabor texture features. IEEE Trans Pattern Anal Mach Intell 13–15

  27. Tamura H, Mori S, Yamawaki T (1978) Textural features corresponding to visual perception. IEEE Trans Syst Man Cybern 8(6):460–472

    Article  Google Scholar 

  28. Chatzichristofis SA, Boutalis YS (2008) Cedd: color and edge directivity descriptor. A compact descriptor for image indexing and retrieval. In: 6th international conference on computer vision systems, ICVS 2008. LNCS, vol 5008, pp 312–322

  29. Chatzichristofis SA, Boutalis YS (2008) Fcth: fuzzy color and texture histogram a low level feature for accurate image retrieval. In: Proceedings of the 9th international workshop on image analysis for multimedia interactive services, WIAMIS, pp 191–196. doi:10.1109/WIAMIS.2008.24

  30. Lux M, Chatzichristofis SA (2008) Lire: lucene image retrieval: an extensible java CBIR library. In: Proceedings of the 16th ACM international conference on multimedia, Vancouver, British Columbia, Canada. doi:10.1145/1459359.1459577

  31. Yates RB, Neto BR (1999) Modern information retrieval, 1st edn. Addison Wesley, Reading

  32. Krishnam MS, Suh RD, Tomasian A, Goldin JG, Lai C, Brown K et al (2007) Postoperative complications of lung transplantation: radiologic findings along a time continuum. Radiographics 27(4):957–974

  33. Hansell DM, Bankier AA, MacMahon H, McLoud TC, Müller NL, Remy J (2008) Fleischner society: glossary of terms for thoracic imaging. Radiology 246(3):697–722

    Article  Google Scholar 

  34. Uzuner Ö, South BR, Shen S, DuVall SL (2011) 2010 i2b2/VA challenge on concepts, assertions, and relations in clinical text. J Am Med Inform Assoc 18(5):552–556

  35. Vapnik V (1998) Statistical learning theory. Wiley, New York

    MATH  Google Scholar 

  36. Wu TF, Lin CJ, Weng RC (2004) Probability estimates for multi-class classification by pairwise coupling. J Mach Learn Res 5:975–1005

Download references

Acknowledgments

This research was supported by the Intramural Research Program of the National Institutes of Health (NIH), the National Library of Medicine (NLM), and the Lister Hill National Center for Biomedical Communications (LHNCBC).

Author information

Authors and Affiliations

  1. US National Library of Medicine, National Institutes of Health, Bethesda, MD, USA

    Md Mahmudur Rahman, Daekeun You, Matthew S. Simpson, Sameer K. Antani, Dina Demner-Fushman & George R. Thoma

Authors
  1. Md Mahmudur Rahman
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Daekeun You
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Matthew S. Simpson
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Sameer K. Antani
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Dina Demner-Fushman
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. George R. Thoma
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Md Mahmudur Rahman.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Rahman, M.M., You, D., Simpson, M.S. et al. Interactive cross and multimodal biomedical image retrieval based on automatic region-of-interest (ROI) identification and classification. Int J Multimed Info Retr 3, 131–146 (2014). https://doi.org/10.1007/s13735-014-0057-9

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s13735-014-0057-9

Keywords

Search

Navigation