Abstract
Purpose
The bag of visual words (BoVW) model is a powerful tool for feature representation that can integrate various handcrafted features like intensity, texture, and spatial information. In this paper, we propose a novel BoVW-based method that incorporates texture and spatial information for the content-based image retrieval to assist radiologists in clinical diagnosis.
Methods
This paper presents a texture-specific BoVW method to represent focal liver lesions (FLLs). Pixels in the region of interest (ROI) are classified into nine texture categories using the rotation-invariant uniform local binary pattern method. The BoVW-based features are calculated for each texture category. In addition, a spatial cone matching (SCM)-based representation strategy is proposed to describe the spatial information of the visual words in the ROI. In a pilot study, eight radiologists with different clinical experience performed diagnoses for 20 cases with and without the top six retrieved results. A total of 132 multiphase computed tomography volumes including five pathological types were collected.
Results
The texture-specific BoVW was compared to other BoVW-based methods using the constructed dataset of FLLs. The results show that our proposed model outperforms the other three BoVW methods in discriminating different lesions. The SCM method, which adds spatial information to the orderless BoVW model, impacted the retrieval performance. In the pilot trial, the average diagnosis accuracy of the radiologists was improved from 66 to 80% using the retrieval system.
Conclusion
The preliminary results indicate that the texture-specific features and the SCM-based BoVW features can effectively characterize various liver lesions. The retrieval system has the potential to improve the diagnostic accuracy and the confidence of the radiologists.
![](http://media.springernature.com/m312/springer-static/image/art%3A10.1007%2Fs11548-017-1671-9/MediaObjects/11548_2017_1671_Fig1_HTML.gif)
![](http://media.springernature.com/m312/springer-static/image/art%3A10.1007%2Fs11548-017-1671-9/MediaObjects/11548_2017_1671_Fig2_HTML.gif)
![](http://media.springernature.com/m312/springer-static/image/art%3A10.1007%2Fs11548-017-1671-9/MediaObjects/11548_2017_1671_Fig3_HTML.gif)
![](http://media.springernature.com/m312/springer-static/image/art%3A10.1007%2Fs11548-017-1671-9/MediaObjects/11548_2017_1671_Fig4_HTML.gif)
![](http://media.springernature.com/m312/springer-static/image/art%3A10.1007%2Fs11548-017-1671-9/MediaObjects/11548_2017_1671_Fig5_HTML.jpg)
![](http://media.springernature.com/m312/springer-static/image/art%3A10.1007%2Fs11548-017-1671-9/MediaObjects/11548_2017_1671_Fig6_HTML.gif)
![](http://media.springernature.com/m312/springer-static/image/art%3A10.1007%2Fs11548-017-1671-9/MediaObjects/11548_2017_1671_Fig7_HTML.gif)
![](http://media.springernature.com/m312/springer-static/image/art%3A10.1007%2Fs11548-017-1671-9/MediaObjects/11548_2017_1671_Fig8_HTML.gif)
![](http://media.springernature.com/m312/springer-static/image/art%3A10.1007%2Fs11548-017-1671-9/MediaObjects/11548_2017_1671_Fig9_HTML.gif)
![](http://media.springernature.com/m312/springer-static/image/art%3A10.1007%2Fs11548-017-1671-9/MediaObjects/11548_2017_1671_Fig10_HTML.gif)
![](http://media.springernature.com/m312/springer-static/image/art%3A10.1007%2Fs11548-017-1671-9/MediaObjects/11548_2017_1671_Fig11_HTML.gif)
![](http://media.springernature.com/m312/springer-static/image/art%3A10.1007%2Fs11548-017-1671-9/MediaObjects/11548_2017_1671_Fig12_HTML.gif)
![](http://media.springernature.com/m312/springer-static/image/art%3A10.1007%2Fs11548-017-1671-9/MediaObjects/11548_2017_1671_Fig13_HTML.gif)
![](http://media.springernature.com/m312/springer-static/image/art%3A10.1007%2Fs11548-017-1671-9/MediaObjects/11548_2017_1671_Fig14_HTML.gif)
Similar content being viewed by others
References
Chi Y, Zhou J, Venkatesh SK, Tian Q, Liu J (2013) Content-based image retrieval of multiphase CT images for focal liver lesion characterization. Med Phys 40(10):103502
Diamant I, Goldberger J, Klang E, Amitai M (2015) Multiphase liver lesions classification using relevant visual words based on mutual information. In: 2015 IEEE 12th international symposium on biomedical imaging (ISBI). IEEE, pp 407–410
Diamant I, Hoogi A, Beaulieu CF, Safdari M, Klang E, Amitai M, Rubin DL (2016) Improved patch-based automated liver lesion classification by separate analysis of the interior and boundary regions. IEEE J Biomed Health Inform 20(6):1585–1594
Napel SA, Beaulieu CF, Rodriguez C, Cui J, Xu J, Gupta A, Korenblum D, Greenspan H, Ma Y, Rubin DL (2010) Automated retrieval of CT images of liver lesions on the basis of image similarity: method and preliminary results 1. Radiology 256(1):243–252
Akgül CB, Rubin DL, Napel S, Beaulieu CF, Greenspan H, Acar B (2011) Content-based image retrieval in radiology: current status and future directions. J Digit Imaging 24(2):208–222
Yu M, Feng Q, Yang W, Gao Y, Chen W (2012) Extraction of lesion-partitioned features and retrieval of contrast-enhanced liver images. Comput Math Methods Med 2012
Roy S, Chi Y, Liu J, Venkatesh SK, Brown MS (2014) Three-dimensional spatiotemporal features for fast content-based retrieval of focal liver lesions. IEEE Trans Biomed Eng 61(11):2768–2778
Yang W, Lu Z, Yu M, Huang M, Feng Q, Chen W (2012) Content-based retrieval of focal liver lesions using bag-of-visual-words representations of single-and multiphase contrast-enhanced CT images. J Digit Imaging 25(6):708–719
Xu Y, Lin L, Hu H, Yu H, Jin C, Wang J, Han X, Chen YW (2016) Combined density, texture and shape features of multi-phase contrast-enhanced CT images for CBIR of focal liver lesions: a preliminary study. In: Chen YW, Toro C, Tanaka S, Howlett RJ, Jain LC (eds) Innovation in medicine and healthcare 2015. Springer, Berlin, pp 215–224
Ojala T, Pietikainen M, Harwood D (1994) Performance evaluation of texture measures with classification based on Kullback discrimination of distributions. In: Proceedings of the 12th IAPR international conference on Pattern recognition, 1994. Vol. 1-conference a: computer vision & image processing. IEEE, pp 582–585
Burner A, Donner R, Mayerhoefer M, Holzer M, Kainberger F, Langs G (2011) Texture bags: anomaly retrieval in medical images based on local 3D-texture similarity. In: MICCAI international workshop on medical content-based retrieval for clinical decision support. Springer, Berlin, pp 116–127
Banerji S, Sinha A, Liu C (2013) A new bag of words LBP (BoWL) descriptor for scene image classification. In: International conference on computer analysis of images and patterns. Springer, Berlin, pp 490–497
Asherov M, Diamant I, Greenspan H (2014) Lung texture classification using bag of visual words. In: SPIE medical imaging. International society for optics and photonics, pp 90352K–90352K
Hofmann T (2001) Unsupervised learning by probabilistic latent semantic analysis. Mach Learn 42(1):177–196
Foncubierta-Rodríguez A, García Seco de Herrera A, Müller H (2013) Medical image retrieval using bag of meaningful visual words: unsupervised visual vocabulary pruning with PLSA. ACM international workshop on multimedia indexing and information retrieval for healthcare, pp 75–82
del Toro OAJ, Foncubiertarodríguez A, Depeursinge A, Müller H (2015) Texture classification of anatomical structures in CT using a context-free machine learning approach. In: SPIE medical imaging, vol 9414, pp 94140W–94140W-14
Depeursinge A, Foncubierta–Rodriguez A, Van de Ville D, Müller H (2012) Multiscale lung texture signature learning using the Riesz transform. Medical image computing & computer-assisted intervention: Miccai international conference on medical image computing & computer-assisted intervention, vol 15, p 517
Depeursinge A, Foncubierta-Rodriguez A, Van de Ville D, Müller H (2011) Lung texture classification using locally–oriented Riesz components. In: Medical image computing and computer-assisted intervention (MICCAI 2011). Springer, Berlin
Csurka G, Perronnin F (2010) Fisher vectors: beyond bag-of-visual-words image representations. In: International conference on computer vision, imaging and computer graphics. Springer, Berlin pp 28–42
Gadermayr M, Kogler H, Uhl A, Vécsei A (2015) Comparing endoscopic imaging configurations in computer-aided celiac disease diagnosis. In: 2015 International conference on image processing theory, tools and applications (IPTA). IEEE, pp 446–451
Gadermayr M, Kogler H, Karla M, Merhof D, Uhl A, Vécsei A (2016) Computer-aided texture analysis combined with experts’ knowledge: improving endoscopic celiac disease diagnosis. World J Gastroenterol 22(31):7124
Lei B, Tan EL, Chen S, Zhuo L, Li S, Ni D, Wang T (2015) Automatic recognition of fetal facial standard plane in ultrasound image via fisher vector. PLoS ONE 10(5):e0121838
Greenspan H, van Ginneken B, Summers RM (2016) Guest editorial deep learning in medical imaging: overview and future promise of an exciting new technique. IEEE Trans Med Imaging 35(5):1153–1159
Roth HR, Lu L, Seff A, Cherry KM, Hoffman J, Wang S, Liu J, Turkey E, Summers RM (2014) A new 2.5 D representation for lymph node detection using random sets of deep convolutional neural network observations. In: International conference on medical image computing and computer-assisted intervention. Springer, Berlin, pp 520–527
Akselrod-Ballin A, Karlinsky L, Alpert S, Hasoul S, Ben-Ari R, Barkan E (2016) A region based convolutional network for tumor detection and classification in breast mammography. In: International workshop on large-scale annotation of biomedical data and expert label synthesis. Springer, Berlin, pp 197–205
de Brebisson A, Montana G (2015) Deep neural networks for anatomical brain segmentation. In: Proceedings of the IEEE conference on computer vision and pattern recognition workshops, pp 20–28
Brosch T, Tang LY, Yoo Y, Li DK, Traboulsee A, Tam R (2016) Deep 3D convolutional encoder networks with shortcuts for multiscale feature integration applied to multiple sclerosis lesion segmentation. IEEE Trans Med Imaging 35(5):1229–1239
Anavi Y, Kogan I, Gelbart E, Geva O, Greenspan H (2015) A comparative study for chest radiograph image retrieval using binary texture and deep learning classification. In: 2015 37th Annual international conference of the IEEE engineering in medicine and biology society (EMBC). IEEE, pp 2940–2943
Bar Y, Diamant I, Wolf L, Greenspan H (2015) Deep learning with non-medical training used for chest pathology identification. In: SPIE medical imaging, international society for optics and photonics, pp 94140V–94140V
Bar Y, Diamant I, Wolf L, Lieberman S, Konen E, Greenspan H (2015) Chest pathology detection using deep learning with non-medical training. In: 2015 IEEE 12th international symposium on biomedical imaging (ISBI). IEEE, pp 294–297
Andrearczyk V, Whelan PF (2016) Deep learning for biomedical texture image analysis. In: Proceedings of the 18th Irish machine vision and image processing conference (IMVIP2016)
Li W, Jia F, Hu Q (2015) Automatic segmentation of liver tumor in CT images with deep convolutional neural networks. J Comput Commun 3:146–151
Vivanti R (2015) Automatic liver tumor segmentation in follow-up CT studies using convolutional neural networks. In: Proceedings of the MICCAI2015
Todoroki Y, Han X, Iwamoto Y, Lin L, Hu H, Chen YW (2017) Detection of liver tumor candidates from CT images using deep convolutional neural networks. In: Proceedings of the international conference on innovation in medicine and healthcare, pp 140–145
Hu P, Wu F, Peng J, Liang P, Kong D (2016) Automatic 3D liver segmentation based on deep learning and globally optimized surface evolution. Phys Med Biol 61(24):8676
Zhao LJ, Tang P, Huo LZ (2014) Land-use scene classification using a concentric circle-structured multiscale bag-of-visual-words model. IEEE J Sel Top Appl Earth Obs Remote Sens 7(12):4620–4631
Lazebnik S, Schmid C, Ponce J (2006) Beyond bags of features: spatial pyramid matching for recognizing natural scene categories. In: 2006 IEEE computer society conference on computer vision and pattern recognition (CVPR’06). IEEE, vol 2, pp 2169–2178
Zhou L, Zhou Z, Hu D (2013) Scene classification using a multi-resolution bag-of-features model. Pattern Recognit 46(1):424–433
Marvasti NB, Kökciyan N, Türkay R, Yazici A, Yolum P, Üsküdarli S, Acar B (2014) ImageCLEF liver CT image annotation task 2014. In: CLEF (working notes), pp 329–340
Dong C, Chen YW, Lin L, Hu H, Jin C, Yu H, Han X, Tateyama T (2016) Simultaneous segmentation of multiple organs using random walks. J Inf Process 24(2):320–329
Joachims T (1998) Text categorization with support vector machines: learning with many relevant features. In: European conference on machine learning. Springer, Berlin, pp 137–142
Ojala T, Pietikainen M, Maenpaa T (2002) Multiresolution gray-scale and rotation invariant texture classification with local binary patterns. IEEE Trans Pattern Anal Mach Intell 24(7):971–987
Barla A, Odone F, Verri A (2003) Histogram intersection kernel for image classification. In: Proceedings of the 2003 international conference on image processing (ICIP 2003), vol 3. IEEE, p III-513
Shan C, Gong S, McOwan PW (2005) Robust facial expression recognition using local binary patterns. In: IEEE international conference on image processing 2005, vol 2. IEEE, p II-370
Acknowledgements
This research was supported in part by the National Key Basic Research Program of China under the Grant No. 2015CB352400, in part by the National Key Research and Development Program of China under the Grant No. 2016YFB1200203-03, in part by the Recruitment Program of Global Experts (HAIOU Program) from Zhejiang Province, China, in part by the Grant-in Aid for Scientific Research from the Japanese Ministry for Education, Science, Culture and Sports (MEXT) under the Grant No. 15H01130, No. 15K00253 and No.16H01436, and in part by the MEXT Support Program for the Strategic Research Foundation at Private Universities (2013–2017).
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Conflict of interest
The authors declare that they have no conflict of interest.
Ethical standard
All procedures performed in studies involving human participants were in accordance with the ethical standards of the institutional and/or national research committee and with the 1964 Helsinki Declaration and its later amendments or comparable ethical standards. For this type of study, formal consent is not required.
Rights and permissions
About this article
Cite this article
Xu, Y., Lin, L., Hu, H. et al. Texture-specific bag of visual words model and spatial cone matching-based method for the retrieval of focal liver lesions using multiphase contrast-enhanced CT images. Int J CARS 13, 151–164 (2018). https://doi.org/10.1007/s11548-017-1671-9
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11548-017-1671-9