Skip to main content
SpringerLink
Log in

Hierarchical BoW with segmental sparse coding for large scale image classification and retrieval

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

The bag-of-words (BoW) has been widely regarded as the most successful algorithms for content-based image related tasks, such as large scale image retrieval, classification, and object categorization. Large visual words acquired by BoW quantization through large vocabulary or codebooks have been receiving much attention in the past years. However, not only construction of large vocabulary but also the quantization process impose a heavy burden in terms of time and memory complexities. In order to tackle this issue, we propose an efficient hierarchical BoW (HBoW) to achieve large visual words through quantization by a compact vocabulary instead of large vocabulary. Our vocabulary is very compact since it is only composed of two small dictionaries which is learned through segmental sparse decomposition of local features. To generate the BoW with large size, we first divide the local features into two half parts, and use the two small dictionaries to compute their sparse codes. Then, we map the two indices of the maximum elements of the two sparse codes to a large set of visual words based upon the fact that data with similar properties will share the same base weighted with the largest sparse coefficient. To further make similar patches have higher probability of select the same dictionary base to get similar BoW vectors, we propose a novel collaborative dictionary learning method by imposing the similarity regularization factor together with the row sparsity regularization across data instances during group sparse coding. Additionally, based on index combination of top-2 large sparse codes of local descriptors, we propose a soft BoW assignment method so that our proposed HBoW can tolerate different word selection for similar patches. By employing the inverted file structure built through our HBoW, K-nearest neighbors (KNN) can be efficiently retrieved. After incorporation of our fast KNN search into the SVM-KNN classification method, our HBoW can be used for efficient image classification and logo recognition. Experiments on serval well-known datasets show that our approach is effective for large scale image classification and retrieval.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11

Similar content being viewed by others

References

  1. Avrithis Y, Kalantidis Y (2012) Approximate gaussian mixtures for large scale vocabularies. In: Proc of ECCV

  2. Chua T-S, Tang S, Trichet R, Tan HK, Song Y (2009) Moviebase: A movie database for event detection and behavioral analysis. In: Proceedings of ACM multimedia 2009 workshop on Web-Scale multimedia corpus

  3. Deng J, Berg AC, Li K, Fei-Fei L (2010) What does classifying more than 10,000 image categories tell us? . In: Proc of the 11th European conference on computer vision: Part V, ECCV’10. Springer, Berlin, pp 71–84

  4. Deng J, Dong W, Socher R, Li L-J, Li K, Fei-fei L (2009) Imagenet: A large-scale hierarchical image database. In: Proc of conference on computer vision and pattern recognition (CVPR). http://image-net.org/

  5. Girod B, Chandrasekhar V, Chen DM, Cheung NM, Grzeszczuk R, Reznik Y, Takacs G, Tsai SS, Vedantham R. (2011) Mobile visual search. IEEE Signal Processing Magazine, Special Issue on Media Search in Mobile Devices 28 (4):61–76

    Article  Google Scholar 

  6. Hastie T, Iain J, Efron B, Tibshirani R (2004) Least angle regression. Ann Stat 32(2):407–499

    Article  MathSciNet  MATH  Google Scholar 

  7. Jegou H, Douze M, Schmid C (2010) Improving bag-of-features for large scale image search. Int J Comput Vis 87(3):316–336

    Article  Google Scholar 

  8. Jegou H, Douze M, Schmid C (2011) Product quantization for nearest neighbor search. IEEE Trans Pattern Anal Mach Intell 33:117–128

    Article  Google Scholar 

  9. Jiang Y-G, Yang J, Ngo C-W, Hauptmann AG (2010) Representations of keypoint-based semantic concept detection A comprehensive study. IEEE Trans Multimedia 12(1):42–53

    Article  Google Scholar 

  10. Lee DD, Seung HS (1999) Learning the parts of objects by non-negative matrix factorization. Nature 401(6755):788–791

    Article  MATH  Google Scholar 

  11. Li D, Yang L, Hua XS, Zhang HJ (2010) Large-scale robust visual codebook construction. In: ACM Multimedia ’10

  12. Li P, Lu X, Wang Q (2015) From dictionary of visual words to subspaces Locality-constrained affine subspace coding. In: Proc of conference on computer vision and pattern recognition (CVPR), pp 2348–2357

  13. Li Y, Tang S, Lin M, Zhang Y, Li J, Yan S (2018) Implicit negative sub-categorization and sink diversion for object detection. IEEE Trans Image Process 27(4):1561–1574

    Article  MathSciNet  Google Scholar 

  14. Liu J, Tang S, Li Y (2017) Collaborative dictionary learning and soft assignment for sparse coding of image features. In: MultiMedia Modeling - 23rd International Conference, MMM 2017, Reykjavik, Iceland, January 4-6, 2017, Proceedings, Part I, pp 443–451

  15. Mairal J, Bach F, Ponce J, Sapiro G (2010) Online learning for matrix factorization and sparse coding. J Mach Learn Res 11:19–60

    MathSciNet  MATH  Google Scholar 

  16. Mantziou E, Papadopoulos S, Kompatsiaris Y (2013) Scalable training with approximate incremental laplacian eigenmaps and pca. In: Proceedings of the 21st ACM international conference on Multimedia, ACM Multimedia, pp 381–384

  17. Mikulik A, Perdoch M, Chum O, Matas J (2013) Learning vocabularies over a fine quantization. Int J Comput Vis 103(1):163–175

    Article  MathSciNet  Google Scholar 

  18. Muja M, Lowe DG (2014) Scalable nearest neighbor algorithms for high dimensional data. IEEE Trans Pattern Anal Mach Intell 36(11):2227–2240

    Article  Google Scholar 

  19. Nie L, Yan S, Wang M, Hong R, Chua T-S (2012) Harvesting visual concepts for image search with complex queries. In: Proc of ACM multimedia 2012 conference

  20. Nister D, Stewenius H (2006) Scalable recognition with a vocabulary tree. In: Proc of conference on computer vision and pattern recognition (CVPR), pp 2161–2168

  21. Petitcolas FAP (2000) Watermarking schemes evaluation. IEEE Signal Process 17(5):117–128

    Article  Google Scholar 

  22. Philbin J, Chum O, Isard M, Sivic J, Zisserman A (2007) Object retrieval with large vocabularies and fast spatial matching. In: Proc of conference on computer vision and pattern recognition (CVPR), pp 1–8

  23. Philbin J , Chum O, Isard M, Sivic J, Zisserman A (2008) Lost in quantization: improving particular object retrieval in large scale image databases. In: Proc of conference on computer vision and pattern recognition (CVPR)

  24. Sivic J, Zisserman A (2003) Video google: a text retrieval approach to object matching in videos. In: Proc of ICCV, pp 1470–1477

  25. Strelow D, Bengio S, Pereira F, Singer Y (2009) Group sparse coding. In: Neural information processing systems - NIPS

  26. Stricker MA, Orengo M (1995) Similarity of color images. In: SPIE conference on storage and retrieval for image and video databases III, vol 2420, pp 381–392

  27. Tang S, Chen H, Lv K, Zhang YD (2015) Large visual words for large scale image classification. In: 2015 IEEE international conference on image processing (ICIP), pp 1170–1174

  28. Tang S, Li J-T, Li M, Xie C, Liu Y-Z, Tao K, Xu S-X (2008) TRECVID 2008 High-Level feature extraction By MCG-ICT-CAS. In: Proceedings of TRECVID 2008 Workshop

  29. Tang S, Li Y, Deng L, Zhang Y-D (2017) Object localization based on proposal fusion. IEEE Trans Multimedia 19(9):2105–2116

    Article  Google Scholar 

  30. Tang S, Zhang YD, Chen H (2015) Scalable logo recognition based on compact sparse dictionary for mobile devices. In: 2015 IEEE 17th international workshop on multimedia signal processing (MMSP), pp 1–6

  31. Tang S, Zhang YD, Xua Z-X, Li H, Zheng Y-T, Li J-T (2015) An efficient concept detection system via sparse ensemble learning. Neurocomputing 69:124–133

    Article  Google Scholar 

  32. Tang S, Zheng Y-T, Wang Y, Chua T-S (2012) Sparse ensemble learning for concept detection. IEEE Trans Multimedia 14(1):43–54

    Article  Google Scholar 

  33. Wang M, Hua X-S, Hong R, Tang J, Qi G-J, Song Y (2010) Unified video annotation via multi-graph learning. IEEE Trans Circuits Syst Video Technol 19 (5):733–746

    Article  Google Scholar 

  34. Wu C (2007) SiftGPU: a GPU implementation of scale invariant feature transform (SIFT) http://cs.unc.edu/ccwu/siftgpu

  35. Xie H, Ke G, Zhang Y, Tang S, Li J, Liu Y (2011) Efficient feature detection and effective post-verification for large scale near-duplicate image search. IEEE Trans Multimedia 13(6):1319–1332

    Article  Google Scholar 

  36. Yang J, Yu K, Gong Y, Huang T (2009) Linear spatial pyramid matching using sparse coding for image classification. In: Proc of Conference on Computer Vision and Pattern Recognition (CVPR)

  37. Zhang H, Berg AC, Maire M, Malik J (2006) Svm-knn: Discriminative nearest neighbor classification for visual category recognition. In: Proc of conference on computer vision and pattern recognition (CVPR), pp 2126–2136

  38. Zhang YD, Wang Y, Tang S, Hoi SCH, Li JT (2014) Fsph: fitted spectral hashing for efficient similarity search. Comput Vis Image Underst 124:3–11

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Beijing Advanced Innovation Center for Imaging Technology, Capital Normal University, Beijing, 100048, People’s Republic of China

    Jianshe Zhou &  Narentuya

  2. Institute of Computing Technology, Chinese Academy of Sciences, Beijing, 100190, People’s Republic of China

    Sheng Tang

  3. College of Information and Engineering, Capital Normal University, Beijing, 100048, People’s Republic of China

    Jie Liu

Authors
  1. Jianshe Zhou
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Narentuya
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Sheng Tang
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Jie Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Jie Liu.

Additional information

The preliminary version of this paper was partly published in the IEEE International Conference on Image Processing (ICIP) 2015, and partly in the 17th IEEE International Workshop on Multimedia Signal Processing (MMSP 2015), and partly in the 19th International Conference on Multimedia Modeling (MMM 2017).

This work was supported by National Nature Science Foundation of China (61371194, 61672361), Beijing Natural Science Foundation (4152012), Beijing Advanced Innovation Center for Imaging Technology (BAICIT-2016009)

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Zhou, J., Narentuya, Tang, S. et al. Hierarchical BoW with segmental sparse coding for large scale image classification and retrieval. Multimed Tools Appl 77, 22319–22338 (2018). https://doi.org/10.1007/s11042-018-5955-z

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-018-5955-z

Keywords

Search

Navigation