Skip to main content
Springer Nature Link
Log in

Scale-space multi-view bag of words for scene categorization

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

As a widely-used method in the image categorization tasks, the Bag-of-Words (BoW) method still suffers from many limitations such as overlooking spatial information. In this paper, we propose four improvements to the BoW method to consider spatial and semantic information as well as information from multiple views. In particular, our contributions are: (a) encoding spatial information based on a combination of wavelet transform image scaling and a new image partitioning scheme, (b) proposing a spatial-information- and content-aware visual word dictionary generation approach, (c) developing a content-aware feature weighting approach to considers the significance of the features for different semantics, (d) proposing a novel weighting strategy to fuse color information when discriminative shape features are lacking. We call our method Scale-Space Multi-View Bag of Words (SSMV-BoW). We conducted extensive experiments to evaluate our SSMV-BoW and compare it to the state-of-the-art scene categorization methods. For our experiments, we use four publicly available and widely used scene categorization benchmark datasets. Results demonstrate that our SSMV-BoW outperforms the methods using both hand-crafted and deep learning features. In addition, ablation studies show that all four improvements contribute to the performance of our SSMV-BoW.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12

Similar content being viewed by others

References

  1. Aharon M, Elad M, Bruckstein A (2006) K-SVD: an algorithm for designing Overcomplete dictionaries for sparse representation. IEEE Trans Signal Process 54(11):4311–4322

    MATH  Google Scholar 

  2. Ahmed KT, Ummesafi S, Iqbal A (2019) Content based image retrieval using image features information fusion. Info Fusion 51:76–99

    Google Scholar 

  3. Arcos-García Á, Álvarez-García JA, Soria-Morillo LM (2018) Deep neural network for traffic sign recognition systems: an analysis of spatial transformers and stochastic optimisation methods. Neural Netw 99:158–165

    Google Scholar 

  4. Babaee M, Rigoll G, Bahmanyar R, Datcu M (2014) Locally linear salient coding for image classification. In Proceeding of the International Workshop on Content-Based Multimedia Indexing (CBMI)

  5. Bahmanyar R, Murillo Montes de Oca A, Datcu M (2015) The semantic gap: an exploration of user and computer perspectives in earth observation images. IEEE Geosci Remote Sens Lett 12(10):2046–2050

    Google Scholar 

  6. Bai S, Tang H (2018) Softly combining an ensemble of classifiers learned from a single convolutional neural network for scene categorization. Appl Soft Comput 67:183–196

    Google Scholar 

  7. Bampis L, Gasteratos A (2019) Revisiting the bag-of-visual-words model: a hierarchical localization architecture for mobile systems. Robot Auton Syst 113:104–119

    Google Scholar 

  8. Banerji, S., Sinha, A., & Liu, C. (2013, August). A new bag of words LBP (BoWL) descriptor for scene image classification.In International conference on computer analysis of images and patterns (pp. 490-497).Springer, Berlin, Heidelberg.

  9. Bolovinou A, Pratikakis I, Perantonis S (2013) Bag of spatio-visual words for context inference in scene classification. Pattern Recogn 46(3):1039–1053

    Google Scholar 

  10. Bosch, A., Zisserman, A., & Munoz, X. (2007, July). Representing shape with a spatial pyramid kernel.In Proceedings of the 6th ACM international conference on Image and video retrieval (pp. 401-408).ACM.

  11. Cakir F, Güdükbay U, Ulusoy Ö (2011) Nearest-neighbor based metric functions for indoor scene recognition. Comput Vis Image Underst 115(11):1483–1492

    Google Scholar 

  12. Chan TH, Jia K, Gao S, Lu J, Zeng Z, Ma Y (2015) PCANet: a simple deep learning baseline for image classification? IEEE Trans Image Process 24(12):5017–5032

    MathSciNet  MATH  Google Scholar 

  13. Cheng G, Li Z, Yao X, Guo L, Wei Z (2017) Remote sensing image scene classification using bag of convolutional features. IEEE Geosci Remote Sens Lett 14(10):1735–1739

    Google Scholar 

  14. Cheng C, Long X, Li Y (2019) VLAD encoding based on LLC for image classification. In Proceedings of the 2019 11th International Conference on Machine Learning and Computing (pp. 417-422). ACM

  15. Csurka G, Dance C, Fan L, Willamowski J, Bray C (2004) Visual categorization with bags of keypoints. In Workshop on statistical learning in computer vision, ECCV (Vol. 1, no. 1-22, pp. 1-2)

  16. Dalal N, Triggs B (2005) Histograms of oriented gradients for human detection. In Computer Vision and Pattern Recognition, 2005. CVPR 2005.IEEE Computer Society Conference on (Vol. 1, pp. 886-893)

  17. de Lima GV, Saito PT, Lopes FM, Bugatti PH (2019) Classification of texture based on bag-of-visual-words through complex networks. Expert Syst Appl 133:215–224

    Google Scholar 

  18. Dixit M, Chen S, Gao D, Rasiwasia N, Vasconcelos N (2015) Scene classification with semantic fisher vectors. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 2974-2983)

  19. Doersch C, Gupta A, Efros AA (2013) Mid-level visual element discovery as discriminative mode seeking. In Advances in neural information processing systems (pp. 494-502)

  20. Escalante HJ, Ponce-López V, Escalera S, Baró X, Morales-Reyes A, Martínez-Carranza J (2015) Evolving weighting schemes for the bag of visual words. Neural Comput & Applic:1–15

  21. Fan H, Zhou E (2016) Approaching human level facial landmark localization by deep learning. Image Vis Comput 47:27–35

    Google Scholar 

  22. Fan RE, Chang KW, Hsieh CJ, Wang XR, Lin CJ (2008) LIBLINEAR: a library for large linear classification. J Mach Learn Res 9:1871–1874

    MATH  Google Scholar 

  23. Farinella GM, Allegra D, Moltisanti M, Stanco F, Battiato S (2016) Retrieval and classification of food images. Comput Biol Med 77:23–39

    Google Scholar 

  24. Fornoni M, Caputo B (2012) Indoor scene recognition using task and saliency-driven feature pooling. In Proceedings of the British Machine Vision Conference (no.EPFL-CONF-192418)

  25. Foumani SNM, Nickabadi A (2019) A probabilistic topic model using deep visual word representation for simultaneous image classification and annotation. J Vis Commun Image Represent 59:195–203

    Google Scholar 

  26. Gao Z, Zhang H, Xu GP, Xue YB, Hauptmann AG (2015) Multi-view discriminative and structured dictionary learning with group sparsity for human action recognition. Signal Process 112:83–97

    Google Scholar 

  27. Giveki D, Montazer GA, Soltanshahi MA (2017) Atanassov's intuitionistic fuzzy histon for robust moving object detection. Int J Approx Reason 91:80–95

    MathSciNet  MATH  Google Scholar 

  28. Giveki D, Soltanshahi MA, Yousefvand M (2020) Proposing a new feature descriptor for moving object detection. Optik 209:164563

    Google Scholar 

  29. Harada, T., Ushiku, Y., Yamashita, Y., &Kuniyoshi, Y. (2011, June). Discriminative spatial pyramid. In Computer Vision and Pattern Recognition (CVPR), 2011 IEEE Conference on (pp. 1617-1624).

  30. Hernández-García R, Ramos-Cózar J, Guil N, García-Reyes E, Sahli H (2018) Improving bag-of-visual-words model using visual n-grams for human action classification. Expert Syst Appl 92:182–191

    Google Scholar 

  31. Huang X, Li SZ, Wang Y (2004) Shape localization based on statistical method using extended local binary pattern. In Multi-Agent Security and Survivability, 2004 IEEE First Symposium on (pp. 184-187).

  32. Huang Y, Huang K, Yu Y, Tan T (2011) Salient coding for image classification. In Computer Vision and Pattern Recognition (CVPR), 2011 IEEE Conference on (pp. 1753-1760)

  33. Khan FS, Van De Weijer J, Anwer RM, Bagdanov AD, Felsberg M, Laaksonen J (2018) Scale coding bag of deep features for human attribute and action recognition. Mach Vis Appl 29(1):55–71

    Google Scholar 

  34. Kim J, Liu C, Sha F, Grauman K (2013) Deformable spatial pyramid matching for fast dense correspondences.In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (pp. 2307-2314)

  35. Krizhevsky A, Sutskever I, Hinton GE (2012) Imagenet classification with deep convolutional neural networks. In Advances in neural information processing systems (pp. 1097-1105)

  36. Lazebnik S, Schmid C, Ponce J (2006) Beyond bags of features: spatial pyramid matching for recognizing natural scene categories. In Computer Vision and Pattern Recognition, 2006 IEEE Computer Society Conference on (Vol. 2, pp. 2169-2178)

  37. Li C, Hamza AB (2013) Intrinsic spatial pyramid matching for deformable 3d shape retrieval. Int J Multimed Info Retriev 2(4):261–271

    Google Scholar 

  38. Li Y, Yang M, Zhang ZM (2018) A survey of multi-view representation Learning. IEEE Transactions on Knowledge and Data Engineering

  39. Li J, Zhang B, Lu G, Zhang D (2019) Generative multi-view and multi-feature learning for classification. Info Fusion 45:215–226

    Google Scholar 

  40. Lowe DG (2004) Distinctive image features from scale-invariant keypoints. Int J Comput Vis 60(2):91–110

    Google Scholar 

  41. Lu C, Shi J, Jia J (2013) Online robust dictionary learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (pp. 415-422)

  42. Montazer GA, Giveki D (2015) An improved radial basis function neural network for object image retrieval. Neurocomputing 168:221–233

    Google Scholar 

  43. Montazer GA, Giveki D (2015) Content based image retrieval system using clustered scale invariant feature transforms. Optik 126(18):1695–1699

    Google Scholar 

  44. Montazer GA, Giveki D (2017) Scene classification using multi-resolution WAHOLB features and neural network classifier. Neural Process Lett 46(2):681–704

    Google Scholar 

  45. Montazer GA, Soltanshahi MA, Giveki D (2015) Extended bag of visual words for face detection. In International Work-Conference on Artificial Neural Networks (pp. 503-510). Springer, Cham

  46. Nakayama H, Harada T, Kuniyoshi Y (2010) Global gaussian approach for scene categorization using information geometry. In Computer Vision and Pattern Recognition (CVPR), 2010 IEEE Conference on (pp. 2336–2343)

  47. Nanni L, Lumini A, Brahnam S (2012) Survey on LBP based texture descriptors for image classification. Expert Syst Appl 39(3):3634–3641

    Google Scholar 

  48. Ojala T, Pietikäinen M, Mäenpää T (2002) Multiresolution gray-scale and rotation invariant texture classification with local binary patterns. Patt Anal Mach Intell IEEE Transact 24(7):971–987

    MATH  Google Scholar 

  49. Oliva A, Torralba A (2001) Modeling the shape of the scene: a holistic representation of the spatial envelope. Int J Comput Vis 42(3):145–175

    MATH  Google Scholar 

  50. Penatti OA, Silva FB, Valle E, Gouet-Brunet V, Torres RDS (2014) Visual word spatial arrangement for image retrieval and classification. Pattern Recogn 47(2):705–720

    Google Scholar 

  51. Quattoni A, Torralba A (2009) Recognizing indoor scenes. In Computer Vision and Pattern Recognition, 2009.CVPR 2009. IEEE Conference on (pp. 413-420)

  52. Rantoson R, Bartoli A (2018) A 3D deformable model-based framework for the retrieval of near-isometric flattenable objects using bag-of-visual-Words. Comput Vis Image Underst 167:89–108

    Google Scholar 

  53. Ravishankar S, Bresler Y (2015) Online sparsifyingtransform learning—part II: convergence analysis. IEEE J Select Topics Signal Process 9(4):637–646

    Google Scholar 

  54. Saikia, A. R., Bora, K., Mahanta, L. B., & Das, A. K. (2019). Comparative assessment of CNN architectures for classification of breast FNAC images. Tissue Cell, 57, 8-14.

  55. Shang R, Meng Y, Wang W, Shang F, Jiao L (2019) Local discriminative based sparse subspace learning for feature selection. Pattern Recogn 92:219–230

    Google Scholar 

  56. Silva FB, Werneck RDO, Goldenstein S, Tabbone S, Torres RDS (2018) Graph-based bag-of-words for classification. Pattern Recogn 74:266–285

    Google Scholar 

  57. Stanković RS, Falkowski BJ (2003) The Haar wavelet transform: its status and achievements. Comput Electr Eng 29(1):25–44

    MATH  Google Scholar 

  58. Sulam J, Ophir B, Zibulevsky M, Elad M (2016) Trainlets: dictionary learning in high dimensions. IEEE Trans Signal Process 64(12):3180–3193

    MathSciNet  MATH  Google Scholar 

  59. Sun S (2013) A survey of multi-view machine learning. Neural Comput & Applic 23(7–8):2031–2038

    Google Scholar 

  60. Sun Y, Xue B, Zhang M, Yen GG (2019) Evolving deep convolutional neural networks for image classification. IEEE Trans Evol Comput 24(2):394–407

    Google Scholar 

  61. Szummer M, Picard RW (1998) Indoor-outdoor image classification. In Content-Based Access of Image and Video Database, 1998. Proceedings., 1998 IEEE International Workshop on (pp. 42-51)

  62. Tian Y, Luo P, Wang X, Tang X (2015) Deep learning strong parts for pedestrian detection. In Proceedings of the IEEE International Conference on Computer Vision (pp. 1904-1912)

  63. Tirilly P, Claveau V, Gros P (2010) Distances and weighting schemes for bag of visual words image retrieval. In Proceedings of the international conference on multimedia information retrieval (pp. 323-332). ACM

  64. Upadhyay PK, Chandra S (2019) An improved bag of dense features for skin lesion recognition. J King Saud Univ-Comp Info Sci

  65. Van Gemert JC, Geusebroek JM, Veenman CJ, Smeulders AW (2008) Kernel codebooks for scene categorization. In European conference on computer vision (pp. 696–709). Springer Berlin Heidelberg

  66. Vogel J, Schiele B (2007) Semantic modeling of natural scenes for content-based image retrieval. Int J Comput Vis 72(2):133–157

    Google Scholar 

  67. Wang Y, Gong S (2007) Conditional random field for natural scene categorization. In BMVC (pp. 1-10).

  68. Wang R, Tao D (2016) Non-local auto-encoder with collaborative stabilization for image restoration. IEEE Trans Image Process 25(5):2117–2129

    MathSciNet  MATH  Google Scholar 

  69. Wang S, Wang Y, Zhu SC (2013) Hierarchical space tiling for scene modeling. In Computer Vision–ACCV 2012 (pp. 796–810). Springer Berlin Heidelberg

  70. Wright J, Ma Y, Mairal J, Sapiro G, Huang TS, Yan S (2010) Sparse representation for computer vision and pattern recognition. Proc IEEE 98(6):1031–1044

    Google Scholar 

  71. Wu J, Rehg JM (2011) CENTRIST: A visual descriptor for scene categorization. Patt Anal Mach Intell IEEE Transact 33(8):1489–1501

    Google Scholar 

  72. Wu R, Wang B, Wang W, Yu Y (2015) Harvesting discriminative meta objects with deep CNN features for scene classification. In Proceedings of the IEEE International Conference on Computer Vision (pp. 1287–1295)

  73. Wu J, Lin Z, Zha H (2019) Essential tensor learning for multi-view spectral Clustering. IEEE Transact Image Process

  74. Xiao Y, Wu J, Yuan J (2014) mCENTRIST: a multi-channel feature generation mechanism for scene categorization. IEEE Trans Image Process 23(2):823–836

    MathSciNet  MATH  Google Scholar 

  75. Xie J, Zhu M (2019) Investigation of acoustic and visual features for acoustic scene classification. Expert Syst Appl 126:20–29

    Google Scholar 

  76. Yang J, Yu K, Gong Y, Huang T (2009) Linear spatial pyramid matching using sparse coding for image classification. In Computer Vision and Pattern Recognition, 2009.CVPR 2009. IEEE Conference on (pp. 1794-1801)

  77. Yang YB, Zhu QH, Mao XJ, Pan LY (2015) Visual feature coding for image classification integrating dictionary structure. Pattern Recogn 48(10):3067–3075

    Google Scholar 

  78. Yin W, Xu D, Wang Z, Zhao Z, Chen C, Yao Y (2019) Perceptually learning multi-view sparse representation for scene categorization. J Vis Commun Image Represent 60:59–63

    Google Scholar 

  79. Yu J, Qin Z, Wan T, Zhang X (2013) Feature integration analysis of bag-of-features model for image retrieval. Neurocomputing 120:355–364

    Google Scholar 

  80. Yu WJ, Chen ZD, Luo X, Liu W, Xu XS (2019) DELTA: a deep dual-stream network for multi-label image classification. Pattern Recogn 91:322–331

    Google Scholar 

  81. Zhang W, Shan S, Gao W, Chen X, Zhang H (2005) Local gabor binary pattern histogram sequence (lgbphs): a novel non-statistical model for face representation and recognition. In Computer Vision, 2005.ICCV 2005.Tenth IEEE International Conference on (Vol. 1, pp. 786-791)

  82. Zhang S, Tian Q, Hua G, Huang Q, Gao W (2014) ObjectPatchNet: towards scalable and semantic image annotation and retrieval. Comput Vis Image Underst 118:16–29

    Google Scholar 

  83. Zhang Z, Xu Y, Yang J, Li X, Zhang D (2015) A survey of sparse representation: algorithms and applications. IEEE Access 3:490–530

    Google Scholar 

  84. Zhao J, Xie X, Xu X, Sun S (2017) Multi-view learning overview: recent progress and new challenges. Info Fusion 38:43–54

    Google Scholar 

  85. Zhou L, Zhou Z, Hu D (2013) Scene classification using a multi-resolution bag-of-features model. Pattern Recogn 46(1):424–433

    Google Scholar 

  86. Zhu SS, Yung NH (2014) Improve scene categorization via sub-scene recognition. Mach Vis Appl 25(6):1561–1572

    Google Scholar 

Download references

Acknowledgements

The author would like to thanks Dr. Reza Bahmanyar from Remote Sensing Technology Institute, German Aerospace Center (DLR), Wessling, Germany for helping me with proofreading the paper.

Author information

Authors and Affiliations

  1. Department of Computer Engineering, Malayer University, P. O. Box 65719-95863, Malayer, Iran

    Davar Giveki

Authors
  1. Davar Giveki
    View author publications

    You can also search for this author inPubMed Google Scholar

Corresponding author

Correspondence to Davar Giveki.

Ethics declarations

Conflict of interest

There is no conflict of interest.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Giveki, D. Scale-space multi-view bag of words for scene categorization. Multimed Tools Appl 80, 1223–1245 (2021). https://doi.org/10.1007/s11042-020-09759-9

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-020-09759-9

Keywords