Skip to main content
SpringerLink
Log in

A deep neural network model for content-based medical image retrieval with multi-view classification

  • Original article
  • Published:
The Visual Computer Aims and scope Submit manuscript

Abstract

In medical applications, retrieving similar images from repositories is most essential for supporting diagnostic imaging-based clinical analysis and decision support systems. However, this is a challenging task, due to the multi-modal and multi-dimensional nature of medical images. In practical scenarios, the availability of large and balanced datasets that can be used for developing intelligent systems for efficient medical image management is quite limited. Traditional models often fail to capture the latent characteristics of images and have achieved limited accuracy when applied to medical images. For addressing these issues, a deep neural network-based approach for view classification and content-based image retrieval is proposed and its application for efficient medical image retrieval is demonstrated. We also designed an approach for body part orientation view classification labels, intending to reduce the variance that occurs in different types of scans. The learned features are used first to predict class labels and later used to model the feature space for similarity computation for the retrieval task. The outcome of this approach is measured in terms of error score. When benchmarked against 12 state-of-the-art works, the model achieved the lowest error score of 132.45, with 9.62–63.14% improvement over other works, thus highlighting its suitability for real-world applications.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8

Similar content being viewed by others

Notes

  1. Online. http://libra.imib.rwth-aachen.de/irma/index en.php.

  2. https://www.imageclef.org/.

References

  1. Lehmann, T.M., Guld, M.O., Thies, C., Fischer, B., Keysers, D., Kohnen, M., Schubert, H., Wein, B.B.: Content-based image retrieval in medical applications for picture archiving and communication systems. In: Medical Imaging 2003: PACS and Integrated Medical Information Systems: Design and Evaluation. Volume 5033., International Society for Optics and Photonics, pp. 109–118 (2003)

  2. Shyu, C.R., Brodley, C.E., Kak, A.C., Kosaka, A., Aisen, A.M., Broderick, L.S.: Assert: a physician-in-the-loop content-based retrieval system for HRCT image databases. Comput. Vis. Image Underst. 75(1–2), 111–132 (1999)

    Article  Google Scholar 

  3. Wang, J.Z.: Pathfinder: multiresolution region-based searching of pathology images using IRM. In: Proceedings of the AMIA Symposium, American Medical Informatics Association 883 (2000)

  4. Tang, L.H., Hanka, R., Ip, H.H., Lam, R.: Extraction of semantic features of histological images for content-based retrieval of images. In: Medical Imaging 1999: PACS Design and Evaluation, Vol. 3662, pp. 360–369 (1999)

  5. Antani, S.K., Long, L.R., Thoma, G.R.: A biomedical information system for combined content-based retrieval of spine X-ray images, associated text information. In: ICVGIP (2002)

  6. Zare, M.R., Mueen, A., Seng, W.C., Awedh, M.H.: Combined feature extraction on medical x-ray images. In: 2011 Third International Conference on Computational Intelligence, Communication Systems and Networks (CICSyN), pp. 264–268. IEEE (2011)

  7. Pourghassem, H., Daneshvar, S.: A framework for medical image retrieval using merging-based classification with dependency probability-based relevance feedback. Turk. J. Electr. Eng. Comput. Sci. 21, 882–896 (2013)

    Google Scholar 

  8. Iakovidis, D.K., Pelekis, N., Kotsifakos, E.E., Kopanakis, I., Karanikas, H., Theodoridis, Y.: A pattern similarity scheme for medical image retrieval. IEEE Trans. Inf. Technol. Biomed. 13(4), 442–450 (2009)

    Article  Google Scholar 

  9. Aggarwal, P., Sardana, H.K., Vig, R.: Content-based medical image retrieval using patient’s semantics with proven pathology for lung cancer diagnosis, In: Fifth International Conference on Advances in Recent Technologies in Communication and Computing (ARTCom 2013). https://doi.org/10.1049/cp.2013.2204

  10. Zare, M.R., Mueen, A., Seng, W.C.: Automatic classification of medical X-ray images using a bag of visual words. IET Comput. Vis. 7(2), 105–114 (2013)

    Article  Google Scholar 

  11. Karthik, K., Kamath, S.S.: A hybrid feature modeling approach for content-based medical image retrieval. In: 2018 IEEE 13th International Conference on Industrial and Information Systems (ICIIS), pp. 7–12. IEEE (2018)

  12. Srinivas, M., Naidu, R.R., Sastry, C.S., Mohan, C.K.: Content based medical image retrieval using dictionary learning. Neurocomputing 168, 880–895 (2015)

    Article  Google Scholar 

  13. Getto, R., Kuijper, A., von Landesberger, T.: Extended surface distance for local evaluation of 3D medical image segmentations. Vis. Comput. 31(6–8), 989–999 (2015)

    Article  Google Scholar 

  14. Trapp, M., Schulze, F., Bühler, K., Liu, T., Dickson, B.J.: 3D object retrieval in an atlas of neuronal structures. Vis. Comput. 29(12), 1363–1373 (2013)

    Article  Google Scholar 

  15. Soundalgekar, P., Kulkarni, M., Nagaraju, D., Kamath, S.: Medical image retrieval using manifold ranking with relevance feedback. In: 2018 IEEE 12th International Conference on Semantic Computing (ICSC), pp. 369–373. IEEE (2018)

  16. Vikram, M., Anantharaman, A., Suhas, B.S.: An approach for multimodal medical image retrieval using latent Dirichlet allocation. In: Proceedings of the ACM India Joint International Conference on Data Science and Management of Data, pp. 44–51. (2019)

  17. Ahmad, J., Muhammad, K., Baik, S.W.: Medical image retrieval with compact binary codes generated in frequency domain using highly reactive convolutional features. J. Med. Syst. 42(2), 24 (2018)

    Article  Google Scholar 

  18. Liu, X., Tizhoosh, H.R., Kofman, J.: Generating binary tags for fast medical image retrieval based on convolutional nets and radon transform. In: 2016 International Joint Conference on Neural Networks (IJCNN), pp. 2872–2878. IEEE (2016)

  19. Zhu, S., Tizhoosh, H.R.: Radon features and barcodes for medical image retrieval via SVM. arXiv:1604.04675 (2016)

  20. Qayyum, A., Anwar, S.M., Awais, M., Majid, M.: Medical image retrieval using deep convolutional neural network. Neurocomputing 266, 8–20 (2017)

    Article  Google Scholar 

  21. Xi, P., Guan, H., Shu, C., Borgeat, L., Goubran, R.: An integrated approach for medical abnormality detection using deep patch convolutional neural networks. Vis Comput (2019). https://doi.org/10.1007/s00371-019-01775-7

    Article  Google Scholar 

  22. Madani, A., Moradi, M., Karargyris, A., Syeda-Mahmood, T.: Semi-supervised learning with generative adversarial networks for chest X-ray classification with ability of data domain adaptation. In: IEEE 15th International Symposium on Biomedical Imaging (ISBI 2018), vol. 2018, pp. 1038–1042. IEEE (2018)

  23. Nardelli, P., Jimenez-Carretero, D., Bermejo-Pelaez, D., Washko, G.R., Rahaghi, F.N., Ledesma-Carbayo, M.J., Estépar, R.S.J.: Pulmonary artery–vein classification in CT images using deep learning. IEEE Trans. Med. Imaging 37(11), 2428–2440 (2018)

    Article  Google Scholar 

  24. Camlica, Z., Tizhoosh, H.R., Khalvati, F.: Autoencoding the retrieval relevance of medical images. In: 2015 International Conference on Image Processing Theory, Tools and Applications (IPTA), pp. 550–555. IEEE (2015)

  25. Khatami, A., Babaie, M., Tizhoosh, H.R., Khosravi, A., Nguyen, T., Nahavandi, S.: A sequential search-space shrinking using CNN transfer learning and a radon projection pool for medical image retrieval. Expert Syst. Appl. 100, 224–233 (2018)

    Article  Google Scholar 

  26. Khatami, A., Babaie, M., Khosravi, A., Tizhoosh, H.R., Nahavandi, S.: Parallel deep solutions for image retrieval from imbalanced medical imaging archives. Appl. Soft Comput. 63, 197–205 (2018)

    Article  Google Scholar 

  27. Avni, U., Goldberger, J., Greenspan, H.: Addressing the imageclef 2009 challenge using a patch-based visual words representation. In: CLEF (Working Notes) (2009)

  28. Müller, H., Kalpathy-Cramer, J., Eggel, I., Bedrick, S., Radhouani, S., Bakke, B., Kahn, C.E., Hersh, W.: Overview of the clef 2009 medical image retrieval track. In: Workshop of the Cross-Language Evaluation Forum for European Languages, pp. 72–84. Springer (2009)

  29. Tommasi, T., Caputo, B., Welter, P., G”uld, M.O., Deserno, T.M.: Overview of the CLEF 2009 medical image annotation track. In: Proceedings Workshop of the Cross-Language Evaluation Forum for European Languages, pp 85–93. Springer (2009)

  30. Sze-To, A., Tizhoosh, H.R., Wong, A.K.: Binary codes for tagging X-ray images via deep de-noising autoencoders. In: 2016 International Joint Conference on Neural Networks (IJCNN), pp. 2864–2871. IEEE (2016)

  31. Sharma, S., Umar, I., Ospina, L., Wong, D., Tizhoosh, H.R.: Stacked autoencoders for medical image search. In: International Symposium on Visual Computing, pp. 45–54. Springer (2016)

  32. Anthimopoulos, M., Christodoulidis, S., Ebner, L., Christe, A., Mougiakakou, S.: Lung pattern classification for interstitial lung diseases using a deep convolutional neural network. IEEE Trans. Med. Imaging 35(5), 1207–1216 (2016)

    Article  Google Scholar 

  33. Tommasi, T., Caputo, B., Welter, P., Güld, M.O., Deserno, T.M.: Overview of the clef 2009 medical image annotation track. In: Workshop of the Cross-Language Evaluation Forum for European Languages, pp. 85–93. Springer (2009)

  34. Xue, Z., You, D., Candemir, S., Jaeger, S., Antani, S., Long, L.R., Thoma, G.R.: Chest X-ray image view classification. In: IEEE 28th International Symposium on Computer-Based Medical Systems, vol. 2015, pp. 66–71. IEEE (2015)

  35. Lehmann, T.M., Schubert, H., Keysers, D., Kohnen, M., Wein, B.B.: The IRMA code for unique classification of medical images. In: Medical Imaging 2003: PACS and Integrated Medical Information Systems: Design and Evaluation. Volume 5033., International Society for Optics and Photonics, pp. 440–452 (2003)

  36. Tizhoosh, H.R.: Barcode annotations for medical image retrieval: a preliminary investigation. In: 2015 IEEE International Conference on Image Processing (ICIP), pp. 818–822. IEEE (2015)

Download references

Acknowledgements

The authors gratefully acknowledge the Science and Engineering Research Board, Department of Science and Technology, Government of India, for its financial support through Early Career Research Grant (ECR/2017/001056) to second author. We also thank Dr. T.M. Deserno, Department of Medical Informatics, RWTH Aachen, Germany, for providing the dataset used in our experiments.

Author information

Authors and Affiliations

  1. Healthcare Analytics and Language Engineering (HALE) Lab, Department of Information Technology, National Institute of Technology Karnataka, Surathkal, Mangaluru, India

    K. Karthik & S. Sowmya Kamath

Authors
  1. K. Karthik
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. S. Sowmya Kamath
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to K. Karthik.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Karthik, K., Kamath, S.S. A deep neural network model for content-based medical image retrieval with multi-view classification. Vis Comput 37, 1837–1850 (2021). https://doi.org/10.1007/s00371-020-01941-2

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00371-020-01941-2

Keywords

Search

Navigation