Skip to main content
SpringerLink
Log in

Multimodal medical image retrieval system

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

In this paper we depict an implemented system for medical image retrieval. Our system performs retrieval based on both textual and visual content, separately and combined, using advanced encoding and quantization techniques. The text-based retrieval subsystem uses textual data acquired from an image’s corresponding article to generate a suitable representation. Using a vector space model, the generated representations structure is altered to increase performance. Query expansion with pseudo-relevance feedback is applied to fine-tune the results. The content-based retrieval subsystem performs retrieval based on visual features extracted from the images. A Gaussian Mixture Model is constructed from the extracted visual features, in our case - RGB histograms, and is used in encoding the same features into Fisher Vectors. With scalability and speed in mind, we utilized a product quantization technique over the generated vectors, which provides fast response times over large image collections. Product quantization drastically reduces the size of the image representation at almost no cost to accuracy, thus improving the scalability factor of our system. Our system uses modality classification to further improve retrieval results. This subsystem labels the image modality based on their visual content. The images are described using state-of-the-art opponentSIFT visual features. Classification was performed using Support Vector Machines (SVMs). The predictions from the SVMs are used for re-ranking the resulting images based on their modality and the modality of the query. The system was evaluated against the standardized ImageCLEF 2013, 2012 and 2011 medical datasets and it reported state-of-the-art performance for all datasets.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6

Similar content being viewed by others

Notes

  1. http://www.nlm.nih.gov/research/umls/

  2. http://lucene.apache.org

  3. http://194.149.136.27/images/home

  4. http://194.149.136.27/images/home

References

  1. Amati G, Van Rijsbergen CJ (2002) Probabilistic models of information retrieval based on measuring the divergence from randomness. ACM Trans Inf Syst (TOIS) 20 (4):357–389

    Article  Google Scholar 

  2. Atrey PK, Hossain MA, El Saddik A, Kankanhalli MS (2010) Multimodal fusion for multimedia analysis: a survey. Multimedia Systems 16(6):345–379

    Article  Google Scholar 

  3. Brazier H, Begley CM (1996) Selecting a database for literature searches in nursing: Medline or cinahl? J Adv Nurs 24(4):868–875

    Article  Google Scholar 

  4. Chang C-C, Lin C-J (2001) LIBSVM: a library for support vector machines, software available at, http://www.csie.ntu.edu.tw/cjlin/libsvm

  5. Chatfield K, Lempitsky V, Vedaldi A, Zisserman A The devil is in the details: an evaluation of recent feature encoding methods

  6. Clough P, Sanderson M, Müller H The clef cross language image retrieval track (imageclef) 2004. In: Image and Video Retrieval, Springer, 2004, pp. 243–251

  7. de Herrera AGS, Markonis D, Eggel I, Müller H (2012) The medgift group in imageclefmed 2012. In: CLEF (Online Working Notes/Labs/Workshop)

  8. de Herrera AGS, Kalpathy-Cramer J, Fushman DD, Antani S, Müller H (2013) Overview of the imageclef 2013 medical tasks. In: Working notes of CLEF 2013

  9. Dimitrovski I, Kocev D, Kitanovski I, Loskovska S, DŻeroski S (2015) Improved medical image modality classification using a combination of visual and textual features. Comput Med Imaging Graph 39:14–26

    Article  Google Scholar 

  10. Douze M, Ramisa A, Schmid C Combining attributes and fisher vectors for efficient image retrieval. In: 2011 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), IEEE, 2011, pp 745–752

  11. Dye C, Reeder JC, Terry RF (2013) Research for universal health coverage. World Health Organization

  12. El-Naqa I, Yang Y, Galatsanos NP, Nishikawa RM, Wernick MN (2004) A similarity learning approach to content-based image retrieval: application to digital mammography, Medical Imaging. Trans IEEE 23(10):1233–1244

    Google Scholar 

  13. Escalante HJ, Hérnadez CA, Sucar LE, Montes M (2008) Late fusion of heterogeneous methods for multimedia image retrieval. In: Proceedings of the 1st ACM international conference on Multimedia information retrieval, ACM, pp 172–179

  14. Ghosh P, Antani S, Long LR, Thoma GR Review of medical image retrieval systems and future directions. In: 2011 24th International Symposium on Computer-Based Medical Systems (CBMS), IEEE, 2011, pp. 1–6

  15. Gonalves N, Oje E, Ricardo V Document mining combining image exploration and text characterization, note

  16. Guld MO, Kohnen M, Keysers D, Schubert H, Wein B. B, Bredno J, Lehmann T. M Quality of DICOM header information for image categorization. In: SPIE vol. 4685 - Medical Imaging 2002: PACS and Integrated Medical Information Systems: Design and Evaluation, 2002, pp. 280–287

  17. Hearst MA, Divoli A, Guturu H, Ksikes A, Nakov P, Wooldridge MA, Ye J (2007) Biotext search engine: beyond abstract search. Bioinformatics 23(16):2196–2197

    Article  Google Scholar 

  18. Ide NC, Loane RF, Demner-Fushman D (2007) Essie: a concept-based search engine for structured biomedical text. J Am Med Inform Assoc 14(3):253–263

    Article  Google Scholar 

  19. Jegou H, Douze M, Schmid C (2011) Product quantization for nearest neighbor search. IEEE Trans Pattern Anal Mach Intell 33(1):117–128

    Article  Google Scholar 

  20. Kahn Jr CE, Thao C (2007) Goldminer: a radiology image search engine. AJR Am J Roentgenol 188(6):1475–1478

    Article  Google Scholar 

  21. Kalpathy-Cramer J, Müller H, Bedrick S, Eggel I, de Herrera AGS, Tsikrika T (2011) Overview of the clef 2011 medical image classification and retrieval tasks. In: CLEF (Notebook Papers/Labs/Workshop)

  22. Kalpathy-Cramer J, Hersh W (2008) Effectiveness of global features for automatic medical image classification and retrieval–the experiences of ohsu at imageclefmed. Pattern Recogn Lett 29(15):2032–2038

    Article  Google Scholar 

  23. Kitanovski I, Dimitrovski I, Loskovska S (2013) Fcse at medical tasks of imageclef2013. In: CLEF (Online Working Notes/Labs/Workshop)

  24. Kitanovski I, Trojacanec K, Dimitrovski I, Loshkovska S (2013) Merging words and concepts for medical articles retrieval. In: Proceedings of the 10th Conference on Open Research Areas in Information Retrieval, LE CENTRE DE HAUTES ETUDES INTERNATIONALES D’INFORMATIQUE DOCUMENTAIRE, pp 25–28

  25. Kitanovski I, Trojacanec K, Dimitrovski I, Loskovska S (2013) Multimodal medical image retrieval. In: ICT Innovations 2012, Springer, pp 81–89

  26. Kitanovski I, Dimitrovski I, Madjarov G, Loskovska S (2014) Medical image retrieval using multimodal data. In: Discovery Science, Springer, pp 144–155

  27. Kumar A, Kim J, Cai W, Fulham M, Feng D (2013) Content-based medical image retrieval: A survey of applications to multidimensional and multimodality data. J Digit Imaging 26(6):1025–1039

    Article  Google Scholar 

  28. Lehmann TM, Wein BB, Dahmen J, Bredno J, Vogelsang F, Kohnen M Content-based image retrieval in medical applications: a novel multistep approach. In: Proceedings of SPIE: Storage and Retrieval for Media Databases, Vol. 3972, 2000, pp. 312–320

  29. Lin H-T, Lin C-J, Weng RC (2007) A note on Platt’s probabilistic outputs for support vector machines. Mach Learn 68:267–276

    Article  Google Scholar 

  30. Lowe DG (2004) Distinctive image features from scale-invariant keypoints. Int J Comput Vis 60(2):91–110

    Article  Google Scholar 

  31. Macdonald C, Plachouras V, He B, Lioma C, Ounis I (2006) University of glasgow at webclef 2005: Experiments in per-field normalisation and language specific stemming. In: Accessing Multilingual Information Repositories, Springer, p 898–907

  32. Mazin B, Delon J, Gousseau Y (2012) Combining color and geometry for local image matching. In: 2012 21st International Conference on Pattern Recognition (ICPR), IEEE, pp 2667–2680

  33. Montague M, Aslam JA (2001) Relevance score normalization for metasearch. In: Proceedings of the tenth international conference on Information and knowledge management, ACM, pp 427–433

  34. Müller H, Kalpathy-Cramer J, Kahn Jr JCE, Hersh W Comparing the quality of accessing medical literature using content-based visual and textual information retrieval. In: SPIE Medical Imaging, International Society for Optics and Photonics, 2009, pp. 726405–726405

  35. Müller H, de Herrera AGS, Kalpathy-Cramer J, Demner-Fushman D, Antani S, Eggel I (2012) Overview of the imageclef 2012 medical image retrieval and classification tasks. In: CLEF (Online Working Notes/Labs/Workshop)

  36. Medical retrieval task. http://www.imageclef.org/node/104/, accessed: 2014-07-03

  37. Névéol A, Deserno TM, Darmoni SJ, Güld MO, Aronson AR (2009) Natural language processing versus content-based image analysis for medical document retrieval. J Am Soc Inf Sci Technol 60(1):123–134

    Article  Google Scholar 

  38. Okan Ozturkmenoglu NMC, Alpkocak A (2013) Demir at imageclefmed 2013: The effects of modality classification to information retrieval. In: CLEF (Online Working Notes/Labs/Workshop)

  39. Ounis I, Amati G, Plachouras V, He B, Macdonald C, Johnson D (2005) Terrier information retrieval platform. In: Advances in Information Retrieval, Springer, pp 517–519

  40. Pubmed. http://www.ncbi.nlm.nih.gov/pubmed, accessed: 2015-03-30

  41. Rahman MM, You D, Simpson MS, Antani SK, Demner-Fushman D, Thoma GR (2013) Multimodal biomedical image retrieval using hierarchical classification and modality fusion. International Journal of Multimedia Information Retrieval 2(3):159–173

    Article  Google Scholar 

  42. Science direct. http://www.elsevier.com/online-tools/sciencedirect, accessed: 2015-03-30

  43. Simonyan K, Modat M, Ourselin S, Cash D, Criminisi A, Zisserman A (2012) Immediate roi search for 3-d medical images. In: MICCAI International Workshop on Content-Based Retrieval for Clinical Decision Support

  44. Spyridon Stathopoulos AK, Ismini Lourentzou, Kalamboukis T (2013) Ipl at clef 2013 medical retrieval task. In: CLEF (Online Working Notes/Labs/Workshop)

  45. Tommasi T, Orabona F, Caputo B (2008) Discriminative cue integration for medical image annotation. Pattern Recogn Lett 29(15):1996–2002

    Article  Google Scholar 

  46. van de Sande K, Gevers T, Snoek C (2010) Evaluating color descriptors for object and scene recognition. IEEE Trans Pattern Anal Mach Intell 32(9):1582–1596

    Article  Google Scholar 

  47. van Gemert JC, Veenman CJ, Smeulders AWM, Geusebroek JM Visual word ambiguity, IEEE Transactions on Pattern Analysis and Machine Intelligence 99 (1)

  48. Weiss Y, Torralba A, Fergus R (2009) Spectral hashing. In: Advances in neural information processing systems, pp 1753–1760

  49. Xu S, McCusker J, Krauthammer M (2008) Yale image finder (yif): a new search engine for retrieving biomedical images. Bioinformatics 24(17):1968–1970

    Article  Google Scholar 

  50. Zheng L, Wetzel AW, Gilbertson J, Becich MJ (2003) Design and analysis of a content-based pathology image retrieval system. IEEE Trans Inf Technol Biomed 7(4):249–255

    Article  Google Scholar 

Download references

Acknowledgments

We would like to acknowledge the support of the European Commission through the project MAESTRA - Learning from Massive, Incompletely annotated, and Structured Data (Grant number ICT-2013-612944).

Author information

Authors and Affiliations

  1. Faculty of Computer Science and Engineering, University “Ss. Cyril and Methodius”, Skopje, Macedonia

    Ivan Kitanovski, Gjorgji Strezoski, Ivica Dimitrovski, Gjorgji Madjarov & Suzana Loskovska

Authors
  1. Ivan Kitanovski
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Gjorgji Strezoski
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Ivica Dimitrovski
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Gjorgji Madjarov
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Suzana Loskovska
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Ivan Kitanovski.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Kitanovski, I., Strezoski, G., Dimitrovski, I. et al. Multimodal medical image retrieval system. Multimed Tools Appl 76, 2955–2978 (2017). https://doi.org/10.1007/s11042-016-3261-1

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-016-3261-1

Keywords

Search

Navigation