Skip to main content
SpringerLink
Log in

Semantic medical image retrieval in a medical social network

  • Original Article
  • Published:
Social Network Analysis and Mining Aims and scope Submit manuscript

Abstract

Medical social networks have become an exchange of opinions between patients and health professionals. However, patients are anxious to quickly find a reliable analysis and a concise explanation of their medical images and express their queries through a textual description or a visual description or both sets. For this, we present in this paper a multimodal research model to research medical images based on multimedia information that is extracted from a radiological collaborative social network. Indeed, the opinions shared on a medical image in a medico-social network are a textual description which in most cases requires cleaning by using a medical thesaurus. In addition, we describe the textual description and medical image in a TF-IDF weight vector using an approach of “bag of words”. We use latent semantic analysis to establish relationships between textual terms and visual terms in shared opinions on the medical image. The multimodal modeling researches the medical information through multimodal queries. Our model is evaluated against the ImageCLEFMed’2015 baseline, which is the ground truth for our experiments. We have conducted numerous experiments with different descriptors and many combinations of modalities. The analysis of results shows that the model based on two methods can increase the performance of a research system based on a single modality, visual or textual.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6

Similar content being viewed by others

Notes

  1. http://www.lemurproject.org/.

  2. Singular value decomposition (SVD).

References

  • Barnard K, Duygulu P, Freitas JFG, Blei DM, Jordan MI (2003) Matching words and pictures. J Mach Learn Res 3:1107–1135

    MATH  Google Scholar 

  • Bouslimi R, Akaichi J (2015) Automatic medical image annotation on social network of physician collaboration. J Netw Model Anal Health Inform Bioinform 4(10):219–228

    Google Scholar 

  • Bouslimi R, Akaichi J, Ayadi MG, Hedhli H (2016) A medical collaboration network for medical image analysis. J Netw Model Anal Health Inform Bioinform 5(10):145–165

    Google Scholar 

  • Chang YC, Chen HH (2008) Using an image-text parallel corpus and the web for query expansion in cross-language image retrieval. In: Workshop of the Cross-Language Evaluation Forum for European Languages CLEF 2007: Advances in multilingual and multimodal information retrieval, Lecture notes in Computer Science (LNCS), vol 5152. Springer, Heidelberg, pp 504–511

  • Clinchant S, Csurka G, Ah-Pine J (2011) Semantic combination of textual and visual information in multimedia retrieval. In: Proceedings of 1st ACM International Conference on multimedia retrieval, New York, NY, USA

  • Csurka G, Dance C, Fan L, Willamowski J, Bray C (2004) Visual categorization with bags of keypoints. In: ECCV’04 workshop on statistical learning in computer vision, pp 59–74

  • de Herrera AGS, Muller H, Bromuri S (2015) Overview of the ImageCLEF 2015 medical classification task. In: Working notes of CLEF 2015 (Cross Language Evaluation Forum)

  • Duan L, Yuan B, Wu C, Li J, Guo Q (2014) Text-image separation and indexing in historic patent document image based on extreme learning machine. In: Proceedings of ELM-2014 Volume 2, Volume 4 of the series Proceedings in Adaptation, Learning and Optimization, pp 299–307

  • Duygulu P, Barnard K, Freitas JFG, Forsyth DA (2002) Object recognition as machine translation: learning a lexicon for a fixed image vocabulary. In Proceedings of ECCV’02, Springer-Verlag, London, UK, pp 97–112

  • Gordo A, Rusinol M, Karatzas D, Bagdanov AD (2013) Document classification and page stream segmentation for digital mailroom applications. In: 2013 12th international conference on document analysis and recognition (ICDAR), IEEE, pp 621–625

  • Gorgevik D, Cakmakov D (2005) Handwritten digit recognition by combining SVM classifiers. In: The international conference on computer as a tool, EUROCON 2005, IEEE, vol 2, pp 1393–1396

  • Guillaumin M, Mensink T, Verbeek JJ, Schmid C (2009) TagProp: discriminative metric learning in nearest neighbor models for image auto-annotation. In: Proceedings of ICCV, pp 309–316

  • Ho TK, Hull JJ, Srihari SN (1994) Decision combination in multiple classifier systems. IEEE Trans Pattern Anal Mach Intell 16(1):66–75

    Article  Google Scholar 

  • Hofmann T (1998) Learning and representing topic. A hierarchical mixture model for word occurrences in document databases. In: Proceedings of CONALD’98, Pittsburgh

  • Jeon J, Lavrenko V, Manmatha R (2003) Automatic image annotation and retrieval using cross-media relevance models. In: Proceedings SIGIR’03 (ACM, New York, NY, USA, 2003), pp 119–126

  • Joachims T (2002) Learning to classify text using support vector machines. In: The springer international series in engineering and computer science, vol 668. Springer, US

  • Jurie F, Triggs W (2005) Creating efficient codebooks for visual recognition. In: ICCV

  • Kittler J, Hatef M, Duin RP, Matas J (1998) On combining classifiers. IEEE Trans Pattern Anal Mach Intell 20(3):226–239

    Article  Google Scholar 

  • Larlus D, Dorkó G, Jurie F (2006) Création de vocabulaires visuels efficaces pour la catégorisation d’images. In: Reconnaissance des Formes et Intelligence Artificielle, Tours

  • Lavrenko V, Manmatha R, Jeon J (2004) A model for learning the semantics of pictures. NIPS, Mathura, pp 553–560

    Google Scholar 

  • Lin WC, Chang YC, Chen HH (2007) Integrating textual and visual information for cross-language image retrieval: a trans-media dictionary approach. J Inf Process Manag 43(2):488–502

    Article  Google Scholar 

  • Liu B (2012) Sentiment analysis and opinion mining. Synth Lect Hum Lang Technol 5(1):1–167

    Article  Google Scholar 

  • Lowe D (2004) Distinctive image features from scale-invariant keypoints. Int J Comput Vis 60(2):91–110

    Article  Google Scholar 

  • Makadia A, Pavlovic V, Kumar S (2010) Baselines for image annotation. Int J Comput Vis 90:88–105

    Article  Google Scholar 

  • Matas J, Chum O, Martin U, Pajdla T (2002) Robust wide baseline stereo from maximally stable extremal regions. In: Proceedings of the British machine vision conference, BMVA, pp 384–393

  • Metzler D, Manmatha R (2004), An inference network approach to image retrieval. In: CIVR’04, pp 42–50

  • Meüller H, Clough P, Deselaers T, Caputo B (2010) ImageCLEF: experimental evaluation in visual information retrieval, vol 32. Springer, Berlin

    Book  MATH  Google Scholar 

  • Monay F, Gatica-Perez D (2003) On image auto-annotation with latent space models. In: Proceedings of MULTIMEDIA’03 (ACM, New York, NY, USA), pp 275–278

  • Mori Y, Takahashi H, Oka R (1999) Image-to-word transformation based on dividing and vector quantizing images with words. In: Proceedings of first international workshop multimedia intelligent storage and retrieval management (Orlando, Florida, USA, 1999), pp 405–409

  • Priyatharshini R, Chitrakala S (2012) Association based image retrieval: a survey. second international joint conference, AIM/CCPE 2012, Bangalore, India, pp 17–26

  • Rahman NA, Mabni Z, Omar N, Fairuz H, Hanum M, Amirah N Nur, Rahim TM (2015) A parallel latent semantic indexing (LSI) algorithm for malay hadith translated document retrieval. In: First international conference, SCDS 2015, Putrajaya, Malaysia, pp 154–163

  • Robertson S, Walker S, Hancock-Beaulieu M, Gull A, Lau M, Okapi (1994) at trec-3. In: Text REtrieval conference, pp 21–30

  • Rusinol M, Lladós J (2009) Logo spotting by a bag-of-words approach for document categorization. In: 2009 10th International conference on document analysis and recognition, IEEE, pp 111–115

  • Rusinol M, Karatzas D, Bagdanov AD, Lladós J (2012) Multipage document retrieval by textual and visual representations. In: 2012 21st international conference on pattern recognition (ICPR), IEEE, pp 521–524

  • Salton G, Wong A, Yang C (1975) A vector space model for automatic indexing. Commun ACM 18(11):613–620

    Article  MATH  Google Scholar 

  • Sebastiani F (2002) Machine learning in automated text categorization. ACM Comput Surv 34(1):1– 47

    Article  Google Scholar 

  • Shekhar R, Jawahar C (2012) Word image retrieval using bag of visual words. In: 2012 10th IAPR international workshop on document analysis systems (DAS), IEEE, pp 297–301

  • Song W, Uchida S, Liwicki M (2011) Look Inside the world of parts of handwritten characters. In: 2011 international conference on document analysis and recognition (ICDAR), IEEE, pp 784–788

  • Terrades OR, Valveny E, Tabbone S (2009) Optimal classifier fusion in a non-bayesian probabilistic framework. IEEE Trans Pattern Anal Mach Intell 31(9):1630–1644

    Article  Google Scholar 

  • Valle E, Cord M, (2009) Advanced techniques in CBIR: local descriptors, visual dictionaries and bags of features. In: Tutorials of the XXII Brazilian symposium on computer graphics and image processing, IEEE, pp 72–78

  • Vidal-Naquet M, Ullman S (2003) Object recognition with informative features and linear classification. In: ICCV, pp 281–288

  • Wang S, Pan P, Lu Y, Xie L (2013) Improving cross-modal and multi-modal retrieval combining content and semantics similarities with probabilistic model. J Multimed Tools Appl 74(6):2009–2032

    Article  Google Scholar 

  • Yang J, Jiang YG, Hauptmann A, Ngo CW (2007) Evaluating bag-of-visual-words representations in scene classification. In: MIR '07: Proceedings of the international workshop on Workshop on multimedia information retrieval, pp 197–206

  • Ye G, Liu D, Jhuo I.-H., Chang S.-F (2012) Robust late fusion with rank minimization. In: 2012 IEEE conference on computer vision and pattern recognition (CVPR), IEEE, pp 3021–3028

  • Zhai C (2001) Notes on the lemur TFIDF model. Technical report. Carnegie Mellon University

Download references

Author information

Authors and Affiliations

  1. Higher Institute of Technological Studies, Jendouba, Tunisia

    Riadh Bouslimi & Mouhamed Gaith Ayadi

  2. Computer Science Department, BESTMOD Lab, ISG-University of Tunis, Bardo, Tunisia

    Riadh Bouslimi, Mouhamed Gaith Ayadi & Jalel Akaichi

Authors
  1. Riadh Bouslimi
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Mouhamed Gaith Ayadi
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Jalel Akaichi
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Riadh Bouslimi.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Bouslimi, R., Ayadi, M.G. & Akaichi, J. Semantic medical image retrieval in a medical social network. Soc. Netw. Anal. Min. 7, 2 (2017). https://doi.org/10.1007/s13278-016-0420-3

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s13278-016-0420-3

Keywords

Search

Navigation