Abstract
A medical report contains many elements such as medical images accompanied by text descriptions. We present in this paper a new approach for semantic automatic annotation of medical images. The proposed approach uses the bag of words model to represent the visual content of the medical image combined with text descriptors based on term frequency–inverse document frequency technique and reduced by latent semantic to extract the co-occurrence between text and visual terms. In a first phase, we are interested in indexing texts and extracting all relevant terms using a thesaurus containing medical subject headings and concepts. In a second phase, medical images are indexed while recovering areas of interest which are invariant to change in scale such as light and tilt. To annotate a new medical image, we use the bag of words model to recover the feature vector. Indeed, we use the vector space model to retrieve similar medical images from the training database. The computation of the relevance value of an image according to a query image is based on the cosine function. To evaluate the performance of our proposed approach, we present an experiment carried out on five types of radiological imaging. The results showed that our approach works efficiently, especially with more images taken from the radiology of the skull.
Similar content being viewed by others
References
Albatal R, Mulhem P, Chiaramella Y (2010) Phrases Visuelles pour l’Annotation Automatiques d’Images. CORIA’10. In: 7e Conférence en Recherche d’Information et Applications, pp 3–18
Avni U, Konen E, Sharon M, Goldberger J (2011) X-ray categorization and retrieval on the organ and pathology level, using patch-based visual words. IEEE Trans Med Image 3:30
Caicedo JC, Cruz-Roa A, Gonzalez FA (2009) Histopathology image classification using bag of features and kernel functions. In: Conference on artificial intelligence in medicine, ser. lecture notes in computer science, vol 5651, pp 126–135
Caicedo JC, Ben-Abdallah J, González FA, Nasraoui O (2012) Multimodal representation, indexing, automated annotation and retrieval of image collections via non-negative matrix factorization. Neurocomputing 76(1):50–60
Cao Y, Wang C, Li Z, Zhang L (2010) Spatial-bag-of-features. In: CVPR’10: 23rd IEEE conference on computer vision and pattern recognition
Csurka G, Dance C, Fan L, Willamowski J, Brayn C (2004) Visual categorization with bags of keypoints. In: ECCV’04 workshop on statistical learning in computer vision, pp 59–74
Deerwester S, Dumais S, Furnas G, Landauer T, Harshman R (1990) Indexing by latent semantic analysis. J Am Soc Inf Sci 41(6):391–407
Diday E (1971) Une Nouvelle Méthode en Classification Automatique et Reconnaissance de Formes: La Méthode des Nuées Dynamiques. Revue de Statistique Appliquée, 19(2):199–33
Jégou H, Douze M, Schmid C (2010) Improving bag-of-features for large scale image search. Int J Comput Vision 87(3):316–336
Jurie F, Triggs B (2005) Creating efficient codebooks for visual recognition. In: ICCV’05 10th IEEE international conference on computer vision, pp 604–610
Kim J, Kumar A, Cai TW, Feng DD (2011) Multi-modal content based image retrieval in healthcare: current applications and future challenges 44:59. Medical Information Science Reference, Pennsylvania
Landauer T, Foltz P, Laham T (1998) Introduction to latent semantic indexing. Discourse Process 25(5):259–284
Lazebnik S, Schmid C, Ponce J (2006) Beyond bags of features: “spatial pyramid matching for recognizing natural scene categories”. In: CVPR’06: IEEE computer society conference on computer vision and pattern recognition, pp 2169–2178
Lemur (2013) http://www.lemurproject.org/
Lowe DG (2004) Distinctive image features from scale-invariant keypoints. Int J Comput Vision 60(2):91–110
MacQueen J (1967) Some methods for classification and analysis of multivariate observations. In: 5th Berkeley symposium on mathematical statistics and probability, pp 281–297
Matas J, Chum O, Martin U, Pajdla T Robust (2002) Wide baseline stereo from maximally stable external regions. In: Proceedings of the British machine vision conference (BMVA), pp 384–393
Mikolajczyk K, Schmid C (2004) Scale & affine invariant interest point detectors. Int J Comput Vision 60(1):63–86
Mikolajczyk K, Tuytelaars T, Schmid C, Zisserman A, Matas J, Schaffalitzky F, Kadir T, Van Gool L (2005) A comparison of affine region detectors. Int J Comput Vision 65:43–72
Nowak E, Jurie F, Triggs B (2006) Sampling strategies for bag-of-features image classification. In: ECCV’06: 9th European conference on computer vision: workshop on statistical learning in computer vision, pp 490–503
Rahman MM, Antani SK, Thoma GR (2011) Biomedical cbir using “bag of keypoints” in a modified inverted index. In: International symposium on computer-based medical systems, ser. CBMS’11, pp 1–6
Robertson S, Walker S, Hancock-Beaulieu M, Gull, Lau M (1994) Okapi at Trec-3. In: Text retrieval conference, pp 21–30
Salton G, Wong A, Yang CS (1975) A vector space model for automatic indexing. Commun ACM 18(11):613–620
Selvi SM, Kavitha C (2014) Radiographic medical image retrieval system for both organ and pathology level using bag of visual words. Int J Eng Sci Emerg Technol 6(4):410–416
Sivic J, Zisserman A (2003) Video Google: a text retrieval approach to object matching in videos. In: Proceedings of the international conference on computer vision, vol 2, pp 1470–1477
Wang XY, Feng D (2009) Image registration for biomedical information integration. In: Data mining and medical knowledge management: cases and applications, vol 122, Information Science Reference, Hershey, p 136
Wang JY, Li YP, Zhang Y, Wang C, Xie HL, Chen GL, Xiao X (2011a) Bag-of-features based medical image retrieval via multiple assignment and visual words weighting. IEEE Trans Med Image 3:30
Wang J, Li Y, Zhang Y, Xie H, Wang C (2011) Boosted learning of visual word weighting factors for bag-of-features based medical image retrieval. In: Image and graphics (ICIG), 2011 6th international conference on, pp 1035–1040
Wu M, Sun Q, Wang J (2012) Medical image retrieval based on combination of visual semantic and local features. Int J Signal Process Image Process Pattern Recognit 5(4):43–56
Yang J, Jiang YG, Hauptmann A, Ngo CW (2007) Evaluating bag-of-visual-words representations in scene classification. In: MIR’07: international workshop on multimedia information retrieval, pp 197–206
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Akaichi, J. Bag of words for semantic automatic medical image annotation. Netw Model Anal Health Inform Bioinforma 3, 61 (2014). https://doi.org/10.1007/s13721-014-0061-2
Received:
Revised:
Accepted:
Published:
DOI: https://doi.org/10.1007/s13721-014-0061-2