Skip to main content
SpringerLink
Log in

IFM3IRS: Information fusion retrieval system with knowledge-assisted text and visual features based on medical conceptual model

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

The technology of medical data production has been rapidly changed over the past few years. Modern computer technology has created the possibility of creating multi-modal medical images. Medical data often contain multi-modal information such as visual information (image) as well as textual information. Both types of information are important for medical retrieval system (MRS). Due to the information limitation at different levels of sources, the application of information fusion becomes a real need in medical application. In this research, an information fusion framework was built to develop the multi-modality medical image retrieval system (IFM3IRS). The framework utilizes two sources of information involving text and visual-based retrieval process. The application is based on sequential order where the result from text-based process will automatically be the input in visual-based process. The main contributions of this paper are the development of a new ranking model called MedHieCon ranking model which applies semantic concepts of modality, anatomy and pathology in text-based process and also the learning approach of medical images using medical concept model in visual-based process. ImageCLEFmed 2010 data collection was used to evaluate IFM3IRS and it shows that our information fusion framework is in top list among other researchers. Although text-based retrieval system has proven to be a better performance in MRS; it is significant to determine the overall performance improvements which include the fusion of text and image.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12

Similar content being viewed by others

Notes

  1. http://www.nlm.nih.gov/research/umls/

  2. http://www.nlm.nih.gov/mesh/meshhome.html

  3. http://www.ebi.uniprot.org/index.shtml

  4. http://www.geneontology.org/

  5. http://www.imageclef2010.org/

  6. http://www.xmlfiles.com/dtd/

References

  1. Adiego J, de la Puente P, Navarro G (2004) Merging prediction by partial matching with structural contexts model. Data Compression Conference, 2004. Proceedings. DCC 2004, pp. 522

  2. Apweiler R, Bairoch A, Wu CH, Barker WC, Boeckmann B, Ferro S et al (2004) UniProt: the universal protein knowledgebase. Nucleic Acids Res 32(suppl 1):D115–D119

    Article  Google Scholar 

  3. Bedrick S, Kalpathy-Cramer J (2010) Improving retrieval using external annotations: OHSU at ImageCLEF 2010. CLEF (Notebook Papers/LABs/Workshops)'10

  4. Bodenreider O (2004) The unified medical language system (UMLS): integrating biomedical terminology. Nucleic Acids Res 32(suppl 1):D267–D270

    Article  Google Scholar 

  5. Bostrom H et al (2007) On the definition of information fusion as a field of research. Humanities

  6. Cavnar WB, Trenkle JM (1994) N-gram-based text categorization. Ann Arbor MI 48113(2):161–175

    Google Scholar 

  7. Chbeir R, Amghar Y, Flory A (2000) MIMS: a prototype for medical image retrieval. Recherche d’Informations Assistee Par Ordinateur

  8. Chevallet JP, Lim JH, Radhouani S, France CI (2005) Using ontology dimensions and negative expansion to solve precise queries in clef medical task. Working Notes of the 2005 CLEF Workshop,(Vienna, Austria)

  9. Depeursinge A, Müller H (2010) Fusion techniques for combining textual and visual information retrieval. ImageCLEF 95–114

  10. Dıaz-Galiano M, Martın-Valdivia M, Montejo-Raez A, Garcıa-Cumbreras M (2009) SINAI at ImageCLEF 2010 medical task

  11. Díaz-Galiano M, Martín-Valdivia M, Ureņa-López L (2009) Query expansion with a medical ontology to improve a multimodal information retrieval system. Comput Biol Med 39(4):396–403

    Article  Google Scholar 

  12. Durao F, Bayyapu KR, Xu G, Dolog P, Lage R (2012) Expanding user’s query with tag-neighbors for effective medical information. Multimed Tools and Appl

  13. Hall DL, Llinas J (1997) An introduction to multisensor data fusion. Proc IEEE 85(1):6–23

    Article  Google Scholar 

  14. Haralick RM, Shanmugam K, Dinstein IH (1973) Textural features for image classification. IEEE Trans Syst Man Cybern 3(6):610–621

    Article  Google Scholar 

  15. Harris C, Stephens M (1988) A combined corner and edge detector. Alvey Vision Conference 15:50

    Google Scholar 

  16. Hearst MA, Divoli A, Guturu H, Ksikes A, Nakov P, Wooldridge MA et al (2007) BioText search engine: beyond abstract search. Bioinformatics 23(16):2196

    Article  Google Scholar 

  17. Hiemstra D (2000) A probabilistic justification for using tf × idf term weighting in information retrieval. Int J Digit Libr 3(2):131–139

    Article  Google Scholar 

  18. Hsu W, Antani S, Long LR, Neve L, Thoma GR (2009) SPIRS: a web-based image retrieval system for large biomedical databases. Int J Med Inform 78:S13–S24

    Article  Google Scholar 

  19. Hu MK (1962) Visual pattern recognition by moment invariants. IRE Trans Inf Theory 8(2):179–187

    Article  MATH  Google Scholar 

  20. Ibrahim R, Arafa W (2010) Medical image retrieval: ISSR at CLEF

  21. Lehmann TM, Schubert H, Keysers D, Kohnen M, Wein BB (2003) The IRMA code for unique classification of medical images. Proc SPIE 5033:109–117

    Google Scholar 

  22. Müller H, Kalpathy–Cramer J, Eggel I, Bedrick S, Radhouani S, Bakke B et al (2011) Overview of the CLEF 2009 medical image retrieval track. Multilingual Information Access Evaluation II. Multimedia Exp 72–84

  23. Müller H, Kalpathy-Cramer J, Eggel I, Bedrick S, Reisetter J Jr, C.E.K., Hersh WR (2010) Overview of the CLEF 2010 medical image retrieval track. CLEF (Notebook Papers/LABs/Workshops)'10

  24. Muller H, Michoux N, Bandon D, Geissbuhler A (2004) A review of content-based image retrieval systems in medical applications–clinical benefits and future directions. Int J Med Inform 73(1):1–23

    Article  Google Scholar 

  25. Ozer S, Langer DL, Liu X, Haider MA, van der Kwast TH, Evans AJ, Yetik IS (2010) Supervised and unsupervised methods for prostate cancer segmentation with multispectral MRI. Med Phys 37:1873

    Article  Google Scholar 

  26. Parikshit Sondhi JS (2010) Medical case-based retrieval by leveraging medical ontology and physician feedback: UIUC-IBM at ImageCLEF 2010. CLEF (Notebook Papers/LABs/Workshops) 2010

  27. Peters C (2001) Cross-language information retrieval and evaluation: Workshop of the cross-language evaluation forum, CLEF 2000, Lisbon, Portugal, September 21–22, 2000: Revised papers Springer-Verlag New York Inc

  28. Prasad M, Sowmya A, Wilson P (2009) Multi-level classification of emphysema in HRCT lung images. Pattern Anal Applic 12(1):9–20

    Article  MathSciNet  Google Scholar 

  29. Rahman M, Antani S, Fushman D, Thoma G (2012) Biomedical image retrieval using multimodal context and concept feature spaces. Medical Content-Based Retrieval for Clinical Decision Support, pp 24–35

  30. Salton G, Wong A, Yang CS (1975) A vector space model for automatic indexing. Commun ACM 18(11):613–620

    Article  MATH  Google Scholar 

  31. Stougiannis A, Gkanogiannis A, Kalamboukis T (2010) IPL at ImageCLEF 2010. CLEF (Notebook Papers/LABs/Workshops)

  32. Vapnik VN (1999) An overview of statistical learning theory. IEEE Trans Neural Netw 10(5):988–999

    Article  Google Scholar 

  33. Wang W, Paliwal J (2006) Spectral data compression and analyses techniques to discriminate wheat classes. Trans ASABE 49(5):1607–1612

    Article  Google Scholar 

  34. Zhou X, Depeursinge A, Müller H (2010) Information fusion for combining visual and textual image retrieval. Proceedings of the 20th International Conference on Recognizing Patterns in Signals, Speech, Images, and Videos, pp 1590–1593

Download references

Author information

Authors and Affiliations

  1. Multimedia Department, Faculty of Computer Science & Information Technology, University Putra Malaysia, Serdang, Malaysia

    Hizmawati Madzin

  2. Artificial Intelligence Department, Faculty of Computer Science & Information Technology, University of Malaya, Kuala Lumpur, Malaysia

    Roziati Zainuddin

  3. Computer Science Department, Faculty of Computer Science & Information Technology, University Putra Malaysia, Serdang, Malaysia

    Nurfadhlina Mohd Sharef

Authors
  1. Hizmawati Madzin
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Roziati Zainuddin
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Nurfadhlina Mohd Sharef
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Hizmawati Madzin.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Madzin, H., Zainuddin, R. & Sharef, N.M. IFM3IRS: Information fusion retrieval system with knowledge-assisted text and visual features based on medical conceptual model. Multimed Tools Appl 74, 3651–3674 (2015). https://doi.org/10.1007/s11042-013-1792-2

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-013-1792-2

Keywords

Search

Navigation