Skip to main content
SpringerLink
Log in

Medical Image Retrieval with Compact Binary Codes Generated in Frequency Domain Using Highly Reactive Convolutional Features

  • Image & Signal Processing
  • Published:
Journal of Medical Systems Aims and scope Submit manuscript

Abstract

Efficient retrieval of relevant medical cases using semantically similar medical images from large scale repositories can assist medical experts in timely decision making and diagnosis. However, the ever-increasing volume of images hinder performance of image retrieval systems. Recently, features from deep convolutional neural networks (CNN) have yielded state-of-the-art performance in image retrieval. Further, locality sensitive hashing based approaches have become popular for their ability to allow efficient retrieval in large scale datasets. In this paper, we present a highly efficient method to compress selective convolutional features into sequence of bits using Fast Fourier Transform (FFT). Firstly, highly reactive convolutional feature maps from a pre-trained CNN are identified for medical images based on their neuronal responses using optimal subset selection algorithm. Then, layer-wise global mean activations of the selected feature maps are transformed into compact binary codes using binarization of its Fourier spectrum. The acquired hash codes are highly discriminative and can be obtained efficiently from the original feature vectors without any training. The proposed framework has been evaluated on two large datasets of radiology and endoscopy images. Experimental evaluations reveal that the proposed method significantly outperforms other features extraction and hashing schemes in both effectiveness and efficiency.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16
Fig. 17

Similar content being viewed by others

References

  1. Aborokbah, M.M., Al-Mutairi, S., Sangaiah, A.K., and Samuel, O.W., Adaptive context aware decision computing paradigm for intensive health care delivery in smart cities—A case analysis. Sustain. Cities Soc., 2017. https://doi.org/10.1016/j.scs.2017.09.004.

  2. Samuel, O.W., Asogbon, G.M., Sangaiah, A.K., Fang, P., and Li, G., An integrated decision support system based on ANN and Fuzzy_AHP for heart failure risk prediction. Expert Syst. Appl. 68:163–172, 2017.

    Article  Google Scholar 

  3. Ahmad, J., Sajjad, M., Mehmood, I., and Baik, S.W., SiNC: Saliency-injected neural codes for representation and efficient retrieval of medical radiographs. PloS One. 12(8):e0181707, 2017.

    Article  PubMed  PubMed Central  Google Scholar 

  4. Ahmad, J., Sajjad, M., Mehmood, I., Rho, S., and Baik, S.W., Saliency-weighted graphs for efficient visual content description and their applications in real-time image retrieval systems. J. Real-Time Image Proces. 13(3):431–447, 2017. https://doi.org/10.1007/s11554-015-0536-0.

    Article  Google Scholar 

  5. Ahmad, J., Sajjad, M., Rho, S., and Baik, S.W., Multi-scale local structure patterns histogram for describing visual contents in social image retrieval systems. Multimed. Tools Appl. 75(20):12669–12692, 2016. https://doi.org/10.1007/s11042-016-3436-9.

    Article  Google Scholar 

  6. Jégou, H., Douze, M., and Schmid, C., Improving bag-of-features for large scale image search. Int. J. Comput. Vis. 87(3):316–336, 2010.

    Article  Google Scholar 

  7. Wang, J., Li, Y., Zhang, Y., Xie, H., and Wang, C. Boosted learning of visual word weighting factors for bag-of-features based medical image retrieval. In: Image and Graphics (ICIG), 2011 Sixth International Conference on, 2011. IEEE, pp 1035–1040

  8. Lowe, D.G., Distinctive image features from scale-invariant keypoints. Int. J. Comput. Vis. 60(2):91–110, 2004. https://doi.org/10.1023/B:VISI.0000029664.99615.94.

    Article  Google Scholar 

  9. Dalal, N., and Triggs, B., Histograms of oriented gradients for human detection. In: Computer Vision and Pattern Recognition, 2005. CVPR 2005. IEEE computer society conference on, 2005. IEEE, pp 886–893

  10. Jégou, H., Douze, M., Schmid, C., and Pérez, P., Aggregating local descriptors into a compact image representation. In: Computer Vision and Pattern Recognition (CVPR), 2010 I.E. conference on, 2010. IEEE, pp 3304–3311

  11. Douze, M., Ramisa, A., and Schmid, C., Combining attributes and fisher vectors for efficient image retrieval. In: in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2011. IEEE, pp 745–752

  12. Liu, L., Shen, C., Wang, L., van den Hengel, A., and Wang, C., Encoding high dimensional local features by sparse coding based fisher vectors. In: Advances in neural information processing systems, 2014. pp 1143–1151

  13. Oliva, A., and Torralba, A., Modeling the shape of the scene: A holistic representation of the spatial envelope. Int. J. Comput. Vis. 42(3):145–175, 2001.

    Article  Google Scholar 

  14. Wu, J., and Rehg, J.M., CENTRIST: A visual descriptor for scene categorization. IEEE Trans. Pattern Anal. Mach. Intell. 33(8):1489–1501, 2011.

    Article  PubMed  Google Scholar 

  15. Zhang, R., Shen, J., Wei, F., Li, X., and Sangaiah, A. K., Medical image classification based on multi-scale non-negative sparse coding. Artif. Intell. Med. 83:44–51, 2017. https://doi.org/10.1016/j.artmed.2017.05.006.

  16. He, K., Zhang, X., Ren, S., and Sun, J., Deep residual learning for image recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition, 2016. pp 770–778

  17. Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., and Rabinovich, A., Going deeper with convolutions. In: Proceedings of the IEEE conference on computer vision and pattern recognition, 2015. pp 1–9

  18. Girshick, R., Donahue, J., Darrell, T., and Malik, J., Rich feature hierarchies for accurate object detection and semantic segmentation. In: Proceedings of the IEEE conference on computer vision and pattern recognition, 2014. pp 580–587

  19. Long, J., Shelhamer, E., and Darrell, T., Fully convolutional networks for semantic segmentation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2015. pp 3431–3440

  20. Babenko, A., Slesarev, A., Chigorin, A., and Lempitsky, V., Neural codes for image retrieval. In: Computer Vision–European Conference on Computer Vision (ECCV). Springer, 2014. pp 584–599. doi:https://doi.org/10.1007/978-3-319-10590-1_38

  21. Babenko, A., and Lempitsky, V., Aggregating local deep features for image retrieval. In: Proceedings of the IEEE international conference on computer vision, 2015. pp 1269–1277

  22. Ahmad, J., Mehmood, I., and Baik, S.W., Efficient object-based surveillance image search using spatial pooling of convolutional features. J. Vis. Commun. Image Represent. 45:62–76, 2017.

    Article  Google Scholar 

  23. Ahmad, J., Mehmood, I., Rho, S., Chilamkurti, N., and Baik, S.W., Embedded deep vision in smart cameras for multi-view objects representation and retrieval. Comput. Electr Eng. 61C:297–311, 2017. https://doi.org/10.1016/j.compeleceng.2017.05.033.

    Article  Google Scholar 

  24. Ahmad, J., Muhammad, K., and Baik, S.W., Data augmentation-assisted deep learning of hand-drawn partially colored sketches for visual search. PloS One. 12(8):e0183838, 2017.

    Article  PubMed  PubMed Central  Google Scholar 

  25. Krizhevsky, A., Sutskever, I., and Hinton, G. E., Imagenet classification with deep convolutional neural networks. In: Advances in neural information processing systems. Curran associates, Inc., pp 1097–1105, 2012.

  26. Gong, Y., Wang, L., Guo, R., and Lazebnik, S., Multi-scale orderless pooling of deep convolutional activation features. In: Computer Vision–ECCV 2014. Springer, pp 392-407, 2014

  27. Kalantidis, Y., Mellina, C., and Osindero, S., Cross-dimensional weighting for aggregated deep convolutional features. In: European Conference on Computer Vision, 2016. Springer, pp 685–701

  28. Alzu’bi, A., Amira, A., and Ramzan, N., Content-based image retrieval with compact deep convolutional features. Neurocomputing 249:95-105, 2017. https://doi.org/10.1016/j.neucom.2017.03.072.

  29. Razavian, A. S., Azizpour, H., Sullivan, J., and Carlsson, S., CNN Features Off-the-Shelf: An Astounding Baseline for Recognition. In: 2014 I.E. Conference on Computer Vision and Pattern Recognition Workshops, 23–28 June 2014 2014. pp 512–519. https://doi.org/10.1109/CVPRW.2014.131

  30. Azizpour, H., Razavian, A., Sullivan, J., Maki, A., and Carlsson, S., From generic to specific deep representations for visual recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition workshops, 2015. pp 36–45.

  31. Mohedano, E., McGuinness, K., O'Connor, N. E., Salvador, A., Marqués, F., and Giró-i-Nieto, X., Bags of local convolutional features for scalable instance search. In: Proceedings of the 2016 ACM on International Conference on Multimedia Retrieval, 2016. ACM, pp 327–331

  32. Srinivas, M., Naidu, R.R., Sastry, C., and Mohan, C.K., Content based medical image retrieval using dictionary learning. Neurocomputing. 168:880–895, 2015. https://doi.org/10.1016/j.neucom.2015.05.036.

    Article  Google Scholar 

  33. Ahmad, J., Muhammad, K., Lee, M.Y., and Baik, S.W., Endoscopic image classification and retrieval using clustered convolutional features. J. Med. Syst. 41(12):196, 2017. https://doi.org/10.1007/s10916-017-0836-y.

    Article  PubMed  Google Scholar 

  34. Liao, X., Yin, J., Guo, S., Li, X., and Sangaiah, A.K., Medical JPEG image steganography based on preserving inter-block dependencies. Comput. Electr. Eng., 2017. https://doi.org/10.1016/j.compeleceng.2017.08.020.

  35. Charikar, M. S., Similarity estimation techniques from rounding algorithms. In: Proceedings of the thiry-fourth annual ACM symposium on Theory of computing, 2002. ACM, pp 380–388.

  36. Weiss, Y., Torralba, A., and Fergus, R., Spectral hashing. In: Advances in neural information processing systems, 2009. pp 1753–1760.

  37. Heo, J-P., Lee, Y., He, J., Chang, S-F., and Yoon, S-E., Spherical hashing. In: Computer Vision and Pattern Recognition (CVPR), 2012 I.E. conference on, 2012. IEEE, pp 2957–2964.

  38. Kulis, B., and Grauman, K., Kernelized locality-sensitive hashing. IEEE Trans. Pattern Anal. Mach. Intell. 34(6):1092–1104, 2012. https://doi.org/10.1109/TPAMI.2011.219.

    Article  PubMed  Google Scholar 

  39. Jin, Z., Li, C., Lin, Y., and Cai, D., Density sensitive hashing. IEEE trans.cybern. 44(8):1362–1371, 2014.

    Article  PubMed  Google Scholar 

  40. Gong, Y., and Lazebnik, S., Iterative quantization: A procrustean approach to learning binary codes. In: Computer Vision and Pattern Recognition (CVPR), 2011 I.E. Conference on, 2011. IEEE, pp 817–824.

  41. Yu, F., Kumar, S., Gong, Y., and Chang, S.- F., Circulant binary embedding. In: International conference on machine learning, 2014. pp 946–954.

  42. Zhang, T., Du, C., and Wang, J., Composite Quantization for Approximate Nearest Neighbor Search. In: ICML, 2014. vol 2. pp 838–846

  43. Erin Liong, V., Lu, J., Wang, G., Moulin, P., and Zhou, J., Deep hashing for compact binary codes learning. In: Proceedings of the IEEE conference on computer vision and pattern recognition, 2015. pp 2475–2483

  44. Lai, H., Pan, Y., Liu, Y., and Yan, S., Simultaneous feature learning and hash coding with deep neural networks. In: Proceedings of the IEEE conference on computer vision and pattern recognition, 2015. pp 3270–3278.

  45. Zhao, F., Huang, Y., Wang, L., and Tan, T., Deep semantic ranking based hashing for multi-label image retrieval. In: Proceedings of the IEEE conference on computer vision and pattern recognition, 2015. pp 1556–1564.

  46. Deng, J., Dong, W., Socher, R., Li, L-J., Li, K., and Fei-Fei, L., Imagenet: A large-scale hierarchical image database. In: Computer Vision and Pattern Recognition, 2009. CVPR 2009. IEEE Conference on, 2009. IEEE, pp 248–255

  47. Simonyan K, Zisserman A (2014) Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:14091556

  48. Welter, P., Deserno, T.M., Fischer, B., Günther, R.W., and Spreckelsen, C., Towards case-based medical learning in radiological decision making using content-based image retrieval. BMC Med. Inform. Decis. Mak. 11(1):1, 2011. https://doi.org/10.1186/1472-6947-11-68.

    Article  Google Scholar 

  49. Pogorelov, K., Randel, K. R., Griwodz, C., Eskeland, S. L., de Lange, T., Johansen, D., Spampinato, C., Dang-Nguyen, D-T., Lux, M., and Schmidt, P. T., Kvasir: A Multi-Class Image Dataset for Computer Aided Gastrointestinal Disease Detection. In: Proceedings of the 8th ACM on Multimedia Systems Conference, 2017. ACM, pp 164–169.

Download references

Funding

This work was supported by the National Research Foundation of Korea (NRF) grant funded by the Korea Government (MSIP) (No.2016R1A2B4011712).

Author information

Authors and Affiliations

  1. Digital Contents Research Institute, Sejong University, Seoul, Republic of Korea

    Jamil Ahmad, Khan Muhammad & Sung Wook Baik

Authors
  1. Jamil Ahmad
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Khan Muhammad
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Sung Wook Baik
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Sung Wook Baik.

Ethics declarations

Conflict of Interest

The authors declare that there is no conflict of interest.

Ethical Approval

This article does not contain any studies with human participants or animals performed by any of the authors.

Additional information

This article is part of the Topical Collection on Image & Signal Processing

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Ahmad, J., Muhammad, K. & Baik, S.W. Medical Image Retrieval with Compact Binary Codes Generated in Frequency Domain Using Highly Reactive Convolutional Features. J Med Syst 42, 24 (2018). https://doi.org/10.1007/s10916-017-0875-4

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s10916-017-0875-4

Keywords

Search

Navigation