Skip to main content
SpringerLink
Log in

Endoscopic Image Classification and Retrieval using Clustered Convolutional Features

  • Image & Signal Processing
  • Published:
Journal of Medical Systems Aims and scope Submit manuscript

Abstract

With the growing use of minimally invasive surgical procedures, endoscopic video archives are growing at a rapid pace. Efficient access to relevant content in such huge multimedia archives require compact and discriminative visual features for indexing and matching. In this paper, we present an effective method to represent images using salient convolutional features. Convolutional kernels from the first layer of a pre-trained convolutional neural network (CNN) are analyzed and clustered into multiple distinct groups, based on their sensitivity to colors and textures. Dominant features detected by each cluster are collected into a single, layout-preserving feature map using a spatial maximal activator pooling (SMAP) approach. A moving window based structured pooling method then captures spatial layout features and global shape information from the aggregated feature map to populate feature histograms. Finally, individual histograms for each cluster are combined into a single comprehensive feature histogram. Clustering convolutional feature space allow extraction of color and texture features of varying strengths. Further, the SMAP approach enable us to select dominant discriminative features. The proposed features are compact and capable of conveniently outperforming several existing features extraction approaches in retrieval and classification tasks on endoscopy images dataset.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10

Similar content being viewed by others

References

  1. Sainju, S., Bui, F.M., and Wahid, K.A., Automated bleeding detection in capsule endoscopy videos using statistical features and region growing. J. Med. Syst. 38:25, 2014.

    Article  PubMed  Google Scholar 

  2. Ahmad, J., Sajjad, M., Mehmood, I., Rho, S., and Baik, S.W., Saliency-weighted graphs for efficient visual content description and their applications in real-time image retrieval systems. J. Real-Time Image Proc. 1–17, 2016.

  3. Murala, S., Maheshwari, R., and Balasubramanian, R., Directional binary wavelet patterns for biomedical image indexing and retrieval. J. Med. Syst. 36:2865–2879, 2012.

    Article  PubMed  Google Scholar 

  4. Smeulders, A.W., Worring, M., Santini, S., Gupta, A., and Jain, R., Content-based image retrieval at the end of the early years. Pattern Analysis and Machine Intelligence, IEEE Transactions on. 22:1349–1380, 2000.

    Article  Google Scholar 

  5. Nowaková, J., Prílepok, M., and Snášel, V., Medical image retrieval using vector quantization and fuzzy S-tree. J. Med. Syst. 41:18, 2017.

    Article  PubMed  Google Scholar 

  6. Messing, D. S., Van Beek, P., and Errico, J. H., The mpeg-7 colour structure descriptor: Image description using colour and local spatial information. In:  IEEE International Conference on Image Processing (ICIP), Thessaloniki, Greece, pp. 670–673, 2001. http://dx.doi.org/10.1109/ICIP.2001.959134.

  7. Liu, G.-H., and Yang, J.-Y., Content-based image retrieval using color difference histogram. Pattern Recogn. 46:188–198, 2013.

    Article  Google Scholar 

  8. Liu, G.-H., Zhang, L., Hou, Y.-K., Li, Z.-Y., and Yang, J.-Y., Image retrieval based on multi-texton histogram. Pattern Recogn. 43:2380–2389, 2010.

    Article  Google Scholar 

  9. Liu, G.-H., Li, Z.-Y., Zhang, L., and Xu, Y., Image retrieval based on micro-structure descriptor. Pattern Recogn. 44:2123–2133, 2011.

    Article  Google Scholar 

  10. Wang, X., and Wang, Z., A novel method for image retrieval based on structure elements’ descriptor. J. Vis. Commun. Image Represent. 24:63–74, 2013.

    Article  Google Scholar 

  11. Ahmad, J., Sajjad, M., Rho, S., and Baik, S.W., Multi-scale local structure patterns histogram for describing visual contents in social image retrieval systems. Multimed. Tools Appl. 75:12669–12692, 2016.

    Article  Google Scholar 

  12. Lowe, D.G., Distinctive image features from scale-invariant keypoints. Int. J. Comput. Vis. 60:91–110, 2004.

    Article  Google Scholar 

  13. Bay, H., Ess, A., Tuytelaars, T., and Van Gool, L., Speeded-up robust features (SURF). Comput. Vis. Image Underst. 110:346–359, 2008. http://dx.doi.org/10.1016/j.cviu.2007.09.014.

  14. Li, T., Mei, T., Kweon, I.-S., and Hua, X.-S., Contextual bag-of-words for visual categorization. IEEE Trans. Circ. Syst. Video Technol. 21:381–392, 2011.

    Article  Google Scholar 

  15. Haas, S., Donner, R., Burner, A., Holzer, M., and Langs, G., Superpixel-based interest points for effective bags of visual words medical image retrieval. In: MICCAI International Workshop on Medical Content-Based Retrieval for Clinical Decision Support, pp. 58–68. Berlin, Heidelberg: Springer, 2011.

  16. Yang, J., Jiang, Y.-G., Hauptmann, A. G., and Ngo, C.-W., Evaluating bag-of-visual-words representations in scene classification. In: Proceedings of the international workshop on multimedia information retrieval, Augsburg, Bavaria, Germany, pp. 197–206, 2007.

  17. Wang, S., Lu, S., Dong, Z., Yang, J., Yang, M., and Zhang, Y., Dual-tree complex wavelet transform and twin support vector machine for pathological brain detection. Appl. Sci. 6:169, 2016.

    Article  Google Scholar 

  18. Zhang, Y.-D., Zhao, G., Sun, J., Wu, X., Wang, Z.-H., Liu, H.-M., et al., Smart pathological brain detection by synthetic minority oversampling technique, extreme learning machine, and Jaya algorithm. Multimed. Tools Appl. 1–20, 2017. http://dx.doi.org/10.1007/s11042-017-5023-0.

  19. Wang, P., Krishnan, S. M., Kugean, C., and Tjoa, M., Classification of endoscopic images based on texture and neural network. In: Engineering in Medicine and Biology Society, 2001. Proceedings of the 23rd Annual International Conference of the IEEE, Istanbul, Turkey, pp. 3691–3695, 2001.

  20. Wang, S.-H., Du, S., Zhang, Y., Phillips, P., Wu, L.-N., Chen, X.-Q., et al., Alzheimer’s disease detection by Pseudo Zernike moment and linear regression classification. CNS Neurol. Disord. Drug Targets (Formerly Current Drug Targets-CNS & Neurological Disorders). 16:11–15, 2017.

  21. Krizhevsky, A., Sutskever, I., and Hinton, G. E., Imagenet classification with deep convolutional neural networks. In: Advances in neural information processing systems, Lake Tahoe, Nevada, pp. 1097–1105. Curran Associates, Inc., USA, 2012.

  22. Ahmad, J., Sajjad, M., Mehmood, I., and Baik, S.W., SiNC: Saliency-injected neural codes for representation and efficient retrieval of medical radiographs. PLoS One. 12:e0181707, 2017.

    Article  PubMed  PubMed Central  Google Scholar 

  23. Krizhevsky, A., and Hinton, G. E., Using very deep autoencoders for content-based image retrieval. In: Proceedings of the 19th European Symposium on Artificial Neural Networks, Bruges, Belgium, pp. 489–494, 2011.

  24. Zhang, Y.-D., Zhang, Y., Hou, X.-X., Chen, H., and Wang, S.-H., Seven-layer deep neural network based on sparse autoencoder for voxelwise detection of cerebral microbleed. Multimed. Tools Appl. 1–18, 2017. http://dx.doi.org/10.1007/s11042-017-4554-8.

  25. Qi, Y., Song, Y.-Z., Zhang, H., and Liu, J., Sketch-based image retrieval via Siamese convolutional neural network. In: IEEE International Conference on Image Processing (ICIP), Phoenix, AZ, USA, pp. 2460–2464, 2016.

  26. Vishnuvarthanan, A., Rajasekaran, M.P., Govindaraj, V., Zhang, Y., and Thiyagarajan, A., An automated hybrid approach using clustering and nature inspired optimization technique for improved tumor and tissue segmentation in magnetic resonance brain images. Appl. Soft Comput. 57:399–426, 2017.

    Article  Google Scholar 

  27. Lu, S., Wang, S., and Zhang, Y., A note on the marker-based watershed method for X-ray image segmentation. Comput. Methods Prog. Biomed. 141:1–2, 2017.

    Article  Google Scholar 

  28. Pons, J., and Serra, X., Designing efficient architectures for modeling temporal features with convolutional neural networks. In: IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP 2017), New Orleans, USA, pp. 2472–2476, 2017.

  29. Zeiler, M. D., and Fergus, R., Visualizing and understanding convolutional networks. In: Fleet, D., Pajdla, T., Schiele, B., and Tuytelaars, T. (Eds), Computer Vision – ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6–12, 2014, Proceedings, Part I, pp. 818–833. Cham: Springer International Publishing, 2014. http://dx.doi.org/10.1007/978-3-319-10590-1_53.

  30. Babenko, A., Slesarev, A., Chigorin, A., and Lempitsky, V., Neural codes for image retrieval. In: Fleet, D., Pajdla, T., Schiele, B., and Tuytelaars, T. (Eds.), Computer Vision – ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6–12, 2014, Proceedings, Part I, pp. 584–599. Cham: Springer International Publishing, 2014. http://dx.doi.org/10.1007/978-3-319-10590-1_38.

  31. Razavian, A. S., Azizpour, H., Sullivan, J., and Carlsson, S., CNN features off-the-shelf: an astounding baseline for recognition. In: IEEE Conference on Computer Vision and Pattern Recognition Workshops, 23–28 June, Columbus, OH, USA, pp. 512–519, 2014. http://dx.doi.org/10.1109/CVPRW.2014.131.

  32. Ahmad, J., Mehmood, I., Rho, S., Chilamkurti, N., and Baik, S.W., Embedded deep vision in smart cameras for multi-view objects representation and retrieval. Comput. Electr. Eng. 61C:297–311, 2017.

    Article  Google Scholar 

  33. Ahmad, J., Mehmood, I., and Baik, S.W., Efficient object-based surveillance image search using spatial pooling of convolutional features. J. Vis. Commun. Image Represent. 45:62–76, 2017.

    Article  Google Scholar 

  34. Li, C., Huang, Y., and Zhu, L., Color texture image retrieval based on Gaussian copula models of Gabor wavelets. Pattern Recogn. 64:118–129, 2017.

    Article  Google Scholar 

  35. Pogorelov, K., Randel, K. R., Griwodz, C., Eskeland, S. L., de Lange, T., Johansen, D., et al., Kvasir: a multi-class image dataset for computer aided gastrointestinal disease detection. In: Proceedings of the 8th ACM on Multimedia Systems Conference, Taipei, Taiwan, pp. 164–169, 2017.

  36. Wang, S., Chen, M., Li, Y., Shao, Y., Zhang, Y., Du, S., et al., Morphological analysis of dendrites and spines by hybridization of ridge detection with twin support vector machine. PeerJ. 4:e2207, 2016.

    Article  PubMed  PubMed Central  Google Scholar 

  37. Yu, L., Feng, L., Chen, C., Qiu, T., Li, L., and Wu, J., A Novel Multi-Feature Representation of Images for Heterogeneous IoTs. IEEE Access. 4:6204–6215, 2016.

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Digital Contents Research Institute, Sejong University, Seoul, Republic of Korea

    Jamil Ahmad, Khan Muhammad, Mi Young Lee & Sung Wook Baik

Authors
  1. Jamil Ahmad
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Khan Muhammad
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Mi Young Lee
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Sung Wook Baik
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Sung Wook Baik.

Ethics declarations

Funding

This work was supported by the National Research Foundation of Korea (NRF) grant funded by the Korea Government (MSIP) (No.2016R1A2B4011712).

Conflict of Interest

The authors declare that there is no conflict of interest.

Ethical Approval

This article does not contain any studies with human participants or animals performed by any of the authors.

Additional information

This article is part of the Topical Collection on Image & Signal Processing

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Ahmad, J., Muhammad, K., Lee, M.Y. et al. Endoscopic Image Classification and Retrieval using Clustered Convolutional Features. J Med Syst 41, 196 (2017). https://doi.org/10.1007/s10916-017-0836-y

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s10916-017-0836-y

Keywords

Search

Navigation