Skip to main content
SpringerLink
Log in

Efficient Deep Feature Based Semantic Image Retrieval

  • Published:
Neural Processing Letters Aims and scope Submit manuscript

Abstract

Explosive growth of multimedia content leads to massive amount of images which are uploaded every day in the cyber world, medical imaging repository, and other areas. Retrieval of image of interest from internet or huge repository of image data set is still challenging and an open problem. Thus, content based image retrieval (CBIR) systems are developed. Success of CBIR system mainly depends on the image features which used for indexing and similarity measurement. CBIR system developed in deep learning framework has recently demonstrated promising results. In this paper, we present a modified-VGG16 (M-VGG16), deep convolution neural network (DCNN), for image feature extraction. These features are used for image indexing and retrieval in the CBIR system. In M-VGG16, we added \(1\times 1\) and \(3\times 3\) convolution kernels into input layer, followed by depth concatenation of the output of both convolution kernels. We apply principal component analysis (PCA) on the features obtained from M-VGG16 for getting robust features with reduced dimension, this leads to the compact image indexing and better retrieval performance. The performance M-VGG16 and M-VGG16 with PCA (M-VGG16 + PCA) is evaluated on benchmark datasets Corel-1k, Corel-10k, Coil-10k, and KADID-10k. Our proposed M-VGG16 + PCA model shows better results in terms of performance metric: precision, recall, and F1-score, when compared to state-of-art model AlexNet , VGG16, and other model proposed using DCNN and hand-crafted feature together. Moreover, in M-VGG16 + PCA the dimension of feature vector is 88% lesser than the M-VGG16 model.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16
Fig. 17
Fig. 18
Fig. 19

Similar content being viewed by others

References

  1. Alzu’bi A, Amira A, Ramzan N (2015) Semantic content-based image retrieval: a comprehensive study. J Vis Commun Image Represent 32:20–54

    Article  Google Scholar 

  2. Gurrin C, Smeaton AF, Doherty AR (2014) Lifelogging: personal big data. Found Trends Inf Retrieval 8(1):1–125

    Article  Google Scholar 

  3. Smeulders AW, Worring M, Santini S, Gupta A, Jain R (2000) Content-based image retrieval at the end of the early years. IEEE Trans Pattern Anal Mach Intell 22(12):1349–1380

    Article  Google Scholar 

  4. Gudivada VN, Raghavan VV (1995) Design and evaluation of algorithms for image retrieval by spatial similarity. ACM Trans Inf Syst 13(2):115–144

    Article  Google Scholar 

  5. Gu Y, Panda B, Haque K.A (2001) Design and analysis of data structures for querying image databases. In: Proceedings of the 2001 ACM symposium on applied computing, pp 236–241

  6. Niblack CW, Barber R, Equitz W, Flickner M.D, Glasman E.H, Petkovic D, Yanker P, Faloutsos C, Taubin G (1993) Qbic project: querying images by content, using color, texture, and shape. In: Storage and retrieval for image and video databases, vol 1908, pp 173–187

  7. Smith JR, Chang S-F (1997) Visualseek: a fully automated content-based image query system. In: Proceedings of the fourth ACM international conference on multimedia, pp 87–98

  8. Wang JZ, Li J, Wiederhold G (2001) Simplicity: semantics-sensitive integrated matching for picture libraries. IEEE Trans Pattern Anal Mach Intell 23(9):947–963

    Article  Google Scholar 

  9. Sivic J, Zisserman A (2003) Video google: a text retrieval approach to object matching in videos. In: IEEE international conference on computer vision, vol 3, pp 1470–1470

  10. Zhu L, Jin H, Zheng R, Feng X (2014) Weighting scheme for image retrieval based on bag-of-visual-words. IET Image Process 8(9):509–518

    Article  Google Scholar 

  11. Elsayad I, Martinet J, Urruty T, Djeraba C (2010) A new spatial weighting scheme for bag-of-visual-words. In: 2010 International workshop on content based multimedia indexing (CBMI), pp 1–6

  12. Huang J, Kumar S.R, Mitra M, Zhu W.-J, Zabih R (1997) Image indexing using color correlograms. In: Proceedings of IEEE computer society conference on computer vision and pattern recognition, pp 762–768

  13. Singha M, Hemachandran K (2012) Content based image retrieval using color and texture. Signal Image Process 3(1):39

    Google Scholar 

  14. Duanmu X (2010) Image retrieval using color moment invariant. In: 2010 Seventh international conference on information technology: new generations. IEEE, pp 200–203

  15. Ojala T, Pietikainen M, Maenpaa T (2002) Multiresolution gray-scale and rotation invariant texture classification with local binary patterns. IEEE Trans Pattern Anal Mach Intell 24(7):971–987

    Article  MATH  Google Scholar 

  16. Heikkilä M, Pietikäinen M, Schmid C (2006) Description of interest regions with center-symmetric local binary patterns. In: Computer vision, graphics and image processing, pp 58–69

  17. Robert MH, Shanmugam K (1973) It shak dinstein “texture features for image classification”. IEEE Trans Syst Man Cybernet SMC-3:610–621

  18. Gómez W, Pereira WCA, Infantosi AFC (2012) Analysis of co-occurrence texture statistics as a function of gray-level quantization for classifying breast ultrasound. IEEE Trans Med Imaging 31(10):1889–1899

    Article  Google Scholar 

  19. Bronstein AM, Bronstein MM, Guibas LJ, Ovsjanikov M (2011) Shape google: geometric words and expressions for invariant shape retrieval. ACM Trans Graph 30(1):1–20

    Article  Google Scholar 

  20. Wang X-Y, Yu Y-J, Yang H-Y (2011) An effective image retrieval scheme using color, texture and shape features. Comput Stand Interfaces 33(1):59–68

    Article  Google Scholar 

  21. Zhou XS, Huang TS (2000) Cbir: from low-level features to high-level semantics. In: Image and video communications and processing 2000, vol 3974, pp 426–431

  22. Liu W, Wang Z, Liu X, Zeng N, Liu Y, Alsaadi FE (2017) A survey of deep neural network architectures and their applications. Neurocomputing 234:11–26

    Article  Google Scholar 

  23. Guo Y, Liu Y, Oerlemans A, Lao S, Wu S, Lew MS (2016) Deep learning for visual understanding: a review. Neurocomputing 187:27–48

    Article  Google Scholar 

  24. Hinton G, Deng L, Yu D, Dahl GE, Mohamed A-R, Jaitly N, Senior A, Vanhoucke V, Nguyen P, Sainath TN et al (2012) Deep neural networks for acoustic modeling in speech recognition: the shared views of four research groups. IEEE Signal Process Mag 29(6):82–97

    Article  Google Scholar 

  25. Babenko A, Slesarev A, Chigorin A, Lempitsky V (2014) Neural codes for image retrieval. In: European conference on computer vision, pp 584–599

  26. Wan J, Wang D, Hoi S.C.H, Wu P, Zhu J, Zhang Y, Li J (2014) Deep learning for content-based image retrieval: a comprehensive study. In: Proceedings of the 22nd ACM international conference on multimedia, pp 157–166

  27. Gong Y, Wang L, Guo R, Lazebnik S (2014) Multi-scale orderless pooling of deep convolutional activation features. In: European conference on computer vision, pp 392–407

  28. Flickner M, Sawhney H, Niblack W, Ashley J, Huang Q, Dom B, Gorkani M, Hafner J, Lee D, Petkovic D et al (1995) Query by image and video content: the qbic system. Computer 28(9):23–32

  29. Pentland A, Picard RW, Sclaroff S (1996) Photobook: content-based manipulation of image databases. Int J Comput Vis 18(3):233–254

    Article  Google Scholar 

  30. Giveki D, Soltanshahi MA, Montazer GA (2017) A new image feature descriptor for content based image retrieval using scale invariant feature transform and local derivative pattern. Optik 131:242–254

    Article  Google Scholar 

  31. Naghashi V (2018) Co-occurrence of adjacent sparse local ternary patterns: a feature descriptor for texture and face image retrieval. Optik 157:877–889

    Article  Google Scholar 

  32. Tolias G, Sicre R, Jégou H (2015) Particular object retrieval with integral max-pooling of cnn activations. arXiv preprint arXiv:1511.05879

  33. Tzelepi M, Tefas A (2016) Relevance feedback in deep convolutional neural networks for content based image retrieval. In: Proceedings of the 9th hellenic conference on artificial intelligence, pp 1–7

  34. Saritha RR, Paul V, Kumar PG (2019) Content based image retrieval using deep learning process. Clust Comput 22(2):4187–4200

    Article  Google Scholar 

  35. Maji S, Bose S (2021) Cbir using features derived by deep learning. ACM/IMS Trans Data Sci 2(3):1–24

    Article  Google Scholar 

  36. Mustafic F, Prazina I, Ljubovic V (2019) A new method for improving content-based image retrieval using deep learning. In: 2019 XXVII international conference on information, communication and automation technologies (ICAT), pp 1–4

  37. Liu P, Guo J-M, Wu C-Y, Cai D (2017) Fusion of deep learning and compressed domain features for content-based image retrieval. IEEE Trans Image Process 26(12):5706–5717

    Article  MathSciNet  MATH  Google Scholar 

  38. Cai Y, Li Y, Qiu C, Ma J, Gao X (2019) Medical image retrieval based on convolutional neural network and supervised hashing. IEEE Access 7:51877–51885

    Article  Google Scholar 

  39. Özaydın U, Georgiou T, Lew M (2019) A comparison of CNN and classic features for image retrieval. In: 2019 International conference on content-based multimedia indexing (CBMI). IEEE, pp 1–4

  40. Tzelepi M, Tefas A (2018) Deep convolutional learning for content based image retrieval. Neurocomputing 275:2467–2478

    Article  Google Scholar 

  41. Shakarami A, Tarrah H (2020) An efficient image descriptor for image classification and cbir. Optik 214:164833

    Article  Google Scholar 

  42. Krizhevsky A, Sutskever I, Hinton GE (2012) Imagenet classification with deep convolutional neural networks. Adv Neural Inf Process Syst 25:1097–1105

    Google Scholar 

  43. Simonyan K, Zisserman A (2014) Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556

  44. Deng J, Dong W, Socher R, Li L-J, Li K, Fei-Fei L (2009) Imagenet: a large-scale hierarchical image database. In: 2009 IEEE conference on computer vision and pattern recognition, pp 248–255

  45. Russakovsky O, Deng J, Su H, Krause J, Satheesh S, Ma S, Huang Z, Karpathy A, Khosla A, Bernstein M, Berg AC, Fei-Fei L (2015) ImageNet large scale visual recognition challenge. Int J Comput Vis 115(3):211–252

    Article  MathSciNet  Google Scholar 

  46. Martinez AM, Kak AC (2001) Pca versus lda. IEEE Trans Pattern Anal Mach Intell 23(2):228–233

    Article  Google Scholar 

  47. Cichocki A, Amari S (2002) Adaptive blind signal and image processing: learning algorithms and applications. Wiley

  48. Long J, Shelhamer E, Darrell T (2015) Fully convolutional networks for semantic segmentation. In: 2015 IEEE conference on computer vision and pattern recognition (CVPR), pp 3431–3440

  49. Zhang Z (2016) Derivation of backpropagation in convolutional neural network (CNN). University of Tennessee, Knoxville

    Google Scholar 

  50. Wang J (2021) Modeling objects, concepts, aesthetics and emotions in big visual data

  51. Pavithra L, Sharmila TS (2018) An efficient framework for image retrieval using color, texture and edge features. Comput Electr Eng 70:580–593

    Article  Google Scholar 

  52. Kanaparthi SK, Raju U, Shanmukhi P, Aneesha GK, Rahman MEU (2020) Image retrieval by integrating global correlation of color and intensity histograms with local texture features. Multimed Tools Appl 79(47):34875–34911

    Article  Google Scholar 

  53. Verma M, Raman B, Murala S (2015) Local extrema co-occurrence pattern for color and texture image retrieval. Neurocomputing 165:255–269

    Article  Google Scholar 

  54. Singh VP, Srivastava R (2018) Improved image retrieval using fast colour-texture features with varying weighted similarity measure and random forests. Multimed Tools Appl 77(11):14435–14460

    Article  Google Scholar 

  55. Xie G, Huang Z, Guo B, Zheng Y, Yan Y (2020) Image retrieval based on the combination of region and orientation correlation descriptors. J Sens 6:66

    Google Scholar 

  56. Bhunia AK, Bhattacharyya A, Banerjee P, Roy PP, Murala S (2020) A novel feature descriptor for image retrieval by combining modified color histogram and diagonally symmetric co-occurrence texture pattern. Pattern Anal Appl 23(2):703–723

    Article  Google Scholar 

  57. Alsmadi MK (2017) An efficient similarity measure for content based image retrieval using memetic algorithm. Egypt J Basic Appl Sci 4(2):112–122

    Google Scholar 

  58. Ashraf R, Ahmed M, Ahmad U, Habib MA, Jabbar S, Naseer K (2020) Mdcbir-mf: multimedia data for content-based image retrieval by using multiple features. Multimed Tools Appl 79(13):8553–8579

    Article  Google Scholar 

  59. Bu H-H, Kim N-C, Kim S-H (2021) Content-based image retrieval using a combination of texture and color features. Hum Centric Comput Inf Sci 11:66

    Google Scholar 

  60. Ghodratnama S, Moghaddam HA (2021) Content-based image retrieval using feature weighting and c-means clustering in a multi-label classification framework. Pattern Anal Appl 24(1):1–10

    Article  Google Scholar 

  61. Kayhan N, Fekri-Ershad S (2021) Content based image retrieval based on weighted fusion of texture and color features derived from modified local binary patterns and local neighborhood difference patterns. Multimed Tools Appl 66:1–28

    Google Scholar 

  62. Zeiler Matthew D, Rob F (2013) Visualizing and understanding convolutional networks. CoRR. arXiv:abs/1311.2901

  63. Chu K, Liu G-H (2020) Image retrieval based on a multi-integration features model. Math Probl Eng 6:66

    Google Scholar 

  64. Nene SA, Nayar SK, Murase H et al (1996) Columbia object image library (coil-100)

  65. Joseph A, Rex ES, Christopher S, Jose J (2021) Content-based image retrieval using hybrid k-means moth flame optimization algorithm. Arab J Geosci 14(8):1–14

    Article  Google Scholar 

  66. Lin H, Hosu V, Saupe D (2019) Kadid-10k: A large-scale artificially distorted IQA database. In: 2019 Tenth international conference on quality of multimedia experience (QoMEX), pp 1–3

  67. Lin H, Hosu V, Saupe D (2020) Deepfl-iqa: weak supervision for deep IQA feature learning. arXiv preprint arXiv:2001.08113

Download references

Acknowledgements

This work is supported by Science and Engineering Research Board (SERB) of the Department of Science and Technology (DST), Government of India, under Mathematical Research Impact Centric Support (MATRICS) scheme. The Grant No. MTR/2018/001103.

Author information

Authors and Affiliations

  1. Department of Computer Science, B.H.U, Varanasi, UP, 221005, India

    Suneel Kumar, Manoj Kumar Singh & Manoj Mishra

Authors
  1. Suneel Kumar
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Manoj Kumar Singh
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Manoj Mishra
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Manoj Kumar Singh.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Kumar, S., Singh, M.K. & Mishra, M. Efficient Deep Feature Based Semantic Image Retrieval. Neural Process Lett 55, 2225–2248 (2023). https://doi.org/10.1007/s11063-022-11079-y

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11063-022-11079-y

Keywords

Search

Navigation