Skip to main content
SpringerLink
Log in

A novel image retrieval algorithm based on transfer learning and fusion features

  • Published:
World Wide Web Aims and scope Submit manuscript

Abstract

With proliferation of social media, image has become ubiquitous giving rise to the demand and importance of image semantic analysis and retrieval to access information quickly on social media. However, even with humongous information available, there are certain categories of images which are important for certain applications but are very scarce. Convolutional neural network is an effective method to extract high-level semantic features for image database retrieval. To overcome the problem of over-fitting when the number of training samples in dataset is limited, this paper proposes an image database retrieval algorithm based on the framework of transfer learning and feature fusion. Based on the fine-tuning of the pre-trained Convolutional Neural Network (CNN), the proposed algorithm first extracts the semantic features of the images. Principal Component Analysis (PCA) is then applied for dimension reduction and to reduce the computational complexity. Last, the semantic feature extracted from the CNN is fused with traditional low-level visual feature to improve the retrieval accuracy further. Experimental results demonstrated the effectiveness of the proposed method for image database retrieval.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Figure 1
Figure 2
Figure 3
Figure 4
Figure 5

Similar content being viewed by others

References

  1. Babenko, A., Slesarev, A., Chigorin, A., et al.: Neural codes for image retrieval [C]. European Conference on Computer Vision. 584–599 (2014)

  2. Bakar, S.A., Hitam, M.S.: Content-based image retrieval using SIFT for binary and greyscale images[C]. 2013 IEEE Int. Conf. Signal Image Process. Appl. 83–88, (2013)

  3. Bengio, Y.: Deep learning of representations for unsupervised and transfer learning[C]. Proceeding of ICML Workshop on Unsupervised and Transfer Learning. (2012)

  4. Cao, L., Wang, F.: Robust latent semantic exploration for image retrieval in social media[J]. Neurocomputing. 169, 2 December, 180–184 (2015)

    Article  Google Scholar 

  5. Carvajal, J.A., Romero, D.G.: Fine-tuning based deep convolutional networks for lepidopterous genus recognition [J]. Progress in Pattern Recognition, Image Analysis, Computer Vision, and Applications(CIARP), pp. 467–475, (2016)

  6. Chatfield, K., Simonyan, K., Vedaldi, A., Zisserman, A.: Return of the devil in the details: delving deep into convolutional nets[C]. In: British Machine Vision Conference. (2014)

  7. Chitrakar, P., Zhang, C., Warner, G., Liao, X.: Social media image retrieval using distilled convolutional neural network for suspicious e-crime and terrorist account detection[C]. 2016 IEEE International Symposium on Multimedia (ISM), pp. In: 493–498 (2016)

    Google Scholar 

  8. Ciresan, D., Meier, U., Schmidhuber, J.: Multi-column deep neural networks for image classification[C]. IEEE Conf. Comput. Vis. Pattern Recognit. 3642–3649 (2012)

  9. Deng, J., Dong, W., Socher, R., Li, L. J., Li, K., Li, F. F.: ImageNet: a large-scale hierarchical image database[C]. Proceedings of the 2009 IEEE Conference on Computer Vision and Pattern Recognition. Miami, 248–255, (2009)

  10. Donahue, J., Jia, Y., Vinyals, O., et al.: DeCAF: a deep convolutional activation feature for generic visual recognition[J]. Proceeding of the 31st International Conference on Maching Learning, PMLR. 32(1), 647–655 (2014)

    Google Scholar 

  11. Hao, J., Dong, J.: What is the best practice for CNNs applied to visual instance retrieval?[C]. International Conference on Learning Representations (ICLR). (2016)

  12. He, K.M., Zhang, X.Y., Ren, S.Q., Sun, J.: Spatial pyramid pooling in deep convolutional networks for visual recognition[J]. IEEE Trans. Pattern Anal. Mach. Intell. 37, 1904–1916 (2015)

    Article  Google Scholar 

  13. Hinton, G.E., Salakhutdinov, R.R.: Reducing the dimensionality of data with neural networks[J]. Science. 313(5786), 504–507 (2006)

    Article  MathSciNet  MATH  Google Scholar 

  14. Kalantidis, Y., Mellina, C., Osindero, S.: Cross-dimensional weighting for aggregated deep convolutional features[J]. European Conference on Computer Vision (ECCV). (2016)

  15. Krizhevsky, A., Sutskever, I., Hinton, G.E.: Imagenet classification with deep convolutional neural networks[C]. Proceeding of the 25th International Conference on Neural Information Processing Systems. 1097–1105 (2012)

  16. Kusamura, Y., Kozawa, Y.: GPU acceleration of content-based image retrieval based on SIFT descriptors[J]. International Conference on Network-Based Information Systems. (2016)

  17. LeCun, Y., Jackel, L., Bottou, L., Cortes, C., Denker, J.S., Drucker, H., et al.: Learning algorithms for classification: a comparison on handwritten digit recognition. Neural networks: The statistical mechanics perspective[M]. 261–276 (1995)

  18. Liu, G.-H., Yang, J.-Y., et al.: Content-based image retrieval using computational visual attention model[J]. Pattern Recogn. 48, 2554–2566 (2015)

    Article  MathSciNet  Google Scholar 

  19. Liu, Y., Hu, D., Fan, J., Wang, F., Zhang, D.: Multi-feature fusion for crime scene investigation image database retrieval. [C]. International Conference on Digital Image Computing: Techniques and Applications (DICTA), Sydney, Australia, Nov.30-Dec. 2, (2017)

  20. Lu, X., Duan, X., Mao, X., Li, Y., Zhang, X.: Feature extraction and fusion using deep convolutional neural networks for face detection[J]. Math. Probl. Eng. 1376726 (2017, 2017)

  21. Oliva, A., Torralba, A.: Modeling the shape of the scence: a holistic representation of the spatial envelope[J]. Int. J. Comput. Vis. 42, 145–175 (2001)

    Article  MATH  Google Scholar 

  22. Pan, S.J., Yang, Q.: A survey on transfer learning[J]. IEEE Trans. Knowl. Data Eng. 22(10), 1345–1359 (2010)

    Article  Google Scholar 

  23. Parida, P., Bhoi, N.: 2-D Gabor filter based transition region extraction and morphological operation for image segmentation[J]. Comput. Electr. Eng. 62, 119–134 (2017)

    Article  Google Scholar 

  24. Shankar, T., Yamuna, G., Suman, G.: Segmentation of natural colour image based on colour-texture features[C]. 2013 International Conference on Communication and Signal Processing, pp. 455–459, (2013)

  25. Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., Rabinovich, A.: Going deeper with convolutions[C]. Proc. IEEE Conf. Comput. Vis. Pattern Recognit. 1–9 (2015)

  26. Taigman, Y., Yang, M., Ranzato, M., Wolf, L.: Deepface: closing the gap to human-level performance in face verification[C]. Proc. IEEE Conf. Comput. Vis. Pattern Recognit. 1701–1708 (2014)

  27. Tolias, G.: Particular object retrieval with integral max-pooling of CNN activations [C]. International Conference on Learning Representations (ICLR). (2016)

  28. Tolias, G., Sicre, R., Jégou, H.: Particular object retrieval with integral max-pooling of CNN activations[C]. In: Proceeding of the 4th International Conference on Learning Representations (ICLR) (2016)

    Google Scholar 

  29. Vedaldi, A., Lenc, K: Matconvnet: Convolutional neural networks for matlab[C]. Proceedings of the 23rd Annual ACM Conference on Multimedia Conference, pp: 689–692, (2015)

  30. Zhang, J., Yoo, C.-W., Ha, S.-W.: ROI based natural image retrieval using color and texture feature[C]. Fourth International Conference on Fuzzy Systems and Knowledge Discovery. 4, 740–744 (2007)

    Google Scholar 

  31. Zheng, L., Wang, S., Liu, Z., Qi, T.: Packing and padding: coupled multi-index for accurate image retrieval [C]. IEEE Conf. Comput. Vis. Pattern Recognit. (2014)

  32. Zheng, L., Wang, S., Zhou, W., and Qi, T.: Bayes merging of multiple vocabularies for scalable image retrieval [C]. IEEE Conf. Comput. Vis. Pattern Recognit. (2014)

  33. Zheng, L., Wang, S., Tian, Q.: "Coupled binary embedding for large-scale image retrieval"[J]. IEEE Trans. Image Process. 23(8), 3368–3380 (June, 2014)

    Article  MathSciNet  MATH  Google Scholar 

Download references

Acknowledgements

This work was supported by Science and Technology Project Fund (No.2016GABJC51) under Ministry of Public Security of China, and National Natural Science Fund of China (No.61671377).

Author information

Authors and Affiliations

  1. Center for Image and Information Processing, Xi’an University of Posts and Telecommunications, Xi’an, 710121, China

    Ying Liu, Yanan Peng & Kengpang Lim

  2. Department of Computer Engineering, Santa Clara University, California, 95053, USA

    Nam Ling

Authors
  1. Ying Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Yanan Peng
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Kengpang Lim
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Nam Ling
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Ying Liu.

Additional information

This article belongs to the Topical Collection: Special Issue on Social Media and Interactive Technologies

Guest Editors: Timothy K. Shih, Lin Hui, Somchoke Ruengittinun, and Qing Li

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Liu, Y., Peng, Y., Lim, K. et al. A novel image retrieval algorithm based on transfer learning and fusion features. World Wide Web 22, 1313–1324 (2019). https://doi.org/10.1007/s11280-018-0585-y

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11280-018-0585-y

Keywords

Search

Navigation