Skip to main content
SpringerLink
Log in

Graph convolutional networks with attention for multi-label weather recognition

  • Original Article
  • Published:
Neural Computing and Applications Aims and scope Submit manuscript

Abstract

Weather recognition is a significant technique for many potential computer vision applications in our daily lives. Generally, most existing works treat weather recognition as a single-label classification task, which cannot describe the weather conditions comprehensively due to the complex co-occurrence dependencies between different weather conditions. In this paper, we propose a novel Graph Convolution Networks with Attention (GCN-A) model for multi-label weather recognition. To our best knowledge, this is the first attempt to introduce GCN into weather recognition. Specifically, we employ GCN to capture weather co-occurrence dependencies via a directed graph. The graph is built over weather labels, where each node (weather label) is represented by word embeddings of a weather label. Furthermore, we design a re-weighted mechanism to build weather correlation matrix for information propagation among different nodes in GCN. In addition, we develop a channel-wise attention module to extract informative semantic features of weather for effective model training. Compared with the state-of-the-art methods, experiment results on two widely used benchmark datasets demonstrate that our proposed GCN-A model achieves promising performance.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10

Similar content being viewed by others

References

  1. Lu C, Lin D, Jia J, Tang C (2014) Two-class weather classification, In: Proceedings of the Conference on Computer Vision and Pattern Recognition. IEEE Computer Society, pp. 3718–3725

  2. Zhao B, Hua L, Li X, Lu X, Wang Z (2019) Weather recognition via classification labels and weather-cue maps. Pattern Recognit 95:272–284

    Article  Google Scholar 

  3. Lu C, Lin D, Jia J, Tang C (2017) Two-class weather classification. IEEE Trans Pattern Anal Mach Intell 39(12):2510–2524

    Article  Google Scholar 

  4. Zhao B, Li X, Lu X, Wang Z (2018) A CNN-RNN architecture for multi-label weather recognition. Neurocomputing 322:47–57

    Article  Google Scholar 

  5. Sun Q, Liu H, Harada T (2017) Online growing neural gas for anomaly detection in changing surveillance scenes. Pattern Recognit 64:187–201

    Article  Google Scholar 

  6. Li X, Ye M, Liu Y, Zhang F, Liu D, Tang S (2017) Accurate object detection using memory-based models in surveillance scenes. Pattern Recognit 67:73–84

    Article  Google Scholar 

  7. De-la-Torre M, Granger E, Sabourin R, Gorodnichy DO (2015) Adaptive skew-sensitive ensembles for face recognition in video surveillance. Pattern Recognit 48(11):3385–3406

    Article  Google Scholar 

  8. Katsura H, Miura J, Hild M, Shirai Y (2005) A view-based outdoor navigation using object recognition robust to changes of weather and seasons. J Robot Soc Japan 23(1):75–83

    Article  Google Scholar 

  9. Loncomilla P, Ruiz-del-Solar J, Martínez LM (2016) Object recognition using local invariant features for robotic applications: a survey. Pattern Recognit 60:499–514

    Article  Google Scholar 

  10. Kurihata H, Takahashi T, Ide I, Mekada Y, Murase H, Tamatsu Y, Miyahara T (2005) Rainy weather recognition from in-vehicle camera images for driver assistance. In: Proceedings of the Intelligent Vehicles Symposium. IEEE, pp. 205–210

  11. Pavlic M, Rigoll G, Ilic S (2013) Classification of images in fog and fog-free scenes for use in vehicles, In: Proceedings of the Intelligent Vehicles Symposium. IEEE, pp. 481–486

  12. Roser M, Moosmann F (2008) Classification of weather situations on single color images, In: Proceedings of the Intelligent Vehicles Symposium. IEEE, pp. 798–803

  13. Hautiére N, Tarel J-P, Lavenant J, Aubert D (2006) Automatic fog detection and estimation of visibility distance through use of an onboard camera. Mach Vision Appl 17(1):8–20

    Article  Google Scholar 

  14. Zhang Z, Ma H, Fu H, Zhang C (2016) Scene-free multi-class weather classification on single images. Neurocomputing 207:365–373

    Article  Google Scholar 

  15. Zhang Z, Ma H (2015) Multi-class weather classification on single images, In: Proceedings of the International Conference on Image Processing. IEEE, pp. 4396–4400

  16. Lin D, Lu C, Huang H, Jia J (2017) Rscm: region selection and concurrency model for multi-class weather recognition. IEEE Trans Image Process 26(9):4154–4167

    Article  MathSciNet  Google Scholar 

  17. L. Yao, C. Mao, and Y. Luo, Graph convolutional networks for text classification, In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 33, 2019, pp. 7370–7377

  18. S. Guo, Y. Lin, N. Feng, C. Song, and H. Wan, Attention based spatial-temporal graph convolutional networks for traffic flow forecasting, In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 33, 2019, pp. 922–929

  19. Chen Z, Wei X, Wang P, Guo Y (2019) Multi-label image recognition with graph convolutional networks, In: Proceedings of the Conference on Computer Vision and Pattern Recognition. Computer Vision Foundation / IEEE, pp. 5177–5186

  20. Yan X, Luo Y, Zheng X (2009) Weather recognition based on images captured by vision system in vehicle, In: Proceedings of the International Symposium on Neural Networks. Springer, pp. 390–398

  21. Fathy M, Siyal MY (1995) An image detection technique based on morphological edge detection and background differencing for real-time traffic analysis. Pattern Recognit Lett 16(12):1321–1330

    Article  Google Scholar 

  22. Pavlic M, Belzner H, Rigoll G, Ilic S (2012) Image based fog detection in vehicles, In: Proceedings of the Intelligent Vehicles Symposium. IEEE, pp. 1132–1137

  23. Kurihata H, Takahashi T, Mekada Y, Ide I, Murase H, Tamatsu Y, Miyahara T (2006) Raindrop detection from in-vehicle video camera images for rainfall judgment, In: Proceedings of the First International Conference on Innovative Computing, Information and Control-Volume I (ICICIC’06), vol. 2. IEEE, pp. 544–547

  24. Bronte S, Bergasa LM, Alcantarilla PF (2009) Fog detection system based on computer vision techniques, In: Proceedings of the International IEEE Conference on Intelligent Transportation Systems. IEEE, pp. 1–6

  25. Li Q, Kong Y, Xia S-m (2014) A method of weather recognition based on outdoor images, In: Proceedings of the International Conference on Computer Vision Theory and Applications (VISAPP), vol. 2. IEEE, pp. 510–516

  26. Song H, Chen Y, Gao Y (2014) Weather condition recognition based on feature extraction and k-nn, In: Proceedings of the Foundations and Practical Applications of Cognitive Systems and Information Processing. Springer, pp. 199–210

  27. Krizhevsky A, Sutskever I, Hinton GE (2012) Imagenet classification with deep convolutional neural networks, In: Proceedings of the Advances in neural information processing systems, pp. 1097–1105

  28. Wang P, Chen P, Yuan Y, Liu D, Huang Z, Hou X, Cottrell GW (2018) Understanding convolution for semantic segmentation, In: Proceedings of the Winter Conference on Applications of Computer Vision. IEEE Computer Society, pp. 1451–1460

  29. Lin T, Dollár P, Girshick RB, He K, Hariharan B, Belongie SJ (2017) Feature pyramid networks for object detection, In: Proceedings of the Conference on Computer Vision and Pattern Recognition. IEEE Computer Society, pp. 936–944

  30. Simonyan K, Zisserman A (2015) Very deep convolutional networks for large-scale image recognition, In: Proceedings of the International Conference on Learning Representations, Y. Bengio and Y. LeCun, Eds.,

  31. He K, Zhang X, Ren S, Sun J (2016) Deep residual learning for image recognition, In: Proceedings of the Conference on Computer Vision and Pattern Recognition. IEEE Computer Society, pp. 770–778

  32. Elhoseiny M, Huang S, Elgammal AM (2015) Weather classification with deep convolutional neural networks, In: Proceedings of the International Conference on Image Processing. IEEE, pp. 3349–3353

  33. Shi Y, Li Y, Liu J, Liu X, Murphey YL (2018) Weather recognition based on edge deterioration and convolutional neural networks, In: Proceedings of the 24th International Conference on Pattern Recognition (ICPR). IEEE, pp. 2438–2443

  34. Guerra JCV, Khanam Z, Ehsan S, Stolkin R, McDonald-Maier K (2018) Weather classification: A new multi-class dataset, data augmentation approach and comprehensive evaluations of convolutional neural networks, In: Proceedings of the NASA/ESA Conference on Adaptive Hardware and Systems (AHS). IEEE, pp. 305–310

  35. Li X, Wang Z, Lu X (2017) A multi-task framework for weather recognition, In: Proceedings of the International Conference on Multimedia. ACM, pp. 1318–1326

  36. Ma J, Chow TW, Zhang H (2020) Semantic-gap-oriented feature selection and classifier construction in multilabel learning, In: IEEE Transactions on Cybernetics

  37. Zhang J, Luo Z, Li C, Zhou C, Li S (2019) Manifold regularized discriminative feature selection for multi-label learning. Pattern Recognit 95:136–150

    Article  Google Scholar 

  38. Ma J, Zhang H, Chow TW (2019) Multilabel classification with label-specific features and classifiers: A coarse-and fine-tuned framework, In: IEEE Transactions on Cybernetics

  39. Zhang X, Zhou X, Lin M, Sun J (2018) Shufflenet: An extremely efficient convolutional neural network for mobile devices, In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 6848–6856

  40. Chollet F (2017) Xception: Deep learning with depthwise separable convolutions, In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1251–1258

  41. Liu W, Liu X, Ma H, Cheng P (2017) Beyond human-level license plate super-resolution with progressive vehicle search and domain priori gan, In: Proceedings of the 25th ACM international conference on Multimedia, pp. 1618–1626

  42. He L, Wang Y, Liu W, Zhao H, Sun Z, Feng J (2019) Foreground-aware pyramid reconstruction for alignment-free occluded person re-identification, In: Proceedings of the IEEE International Conference on Computer Vision, pp. 8450–8459

  43. Wang J, Yang Y, Mao J, Huang Z, Huang C, Xu W (2016) Cnn-rnn: A unified framework for multi-label image classification, In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2285–2294

  44. Zhu F, Li H, Ouyang W, Yu N, Wang X (2017) Learning spatial regularization with image-level supervisions for multi-label image classification, In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 5513–5522

  45. Wang Z, Chen T, Li G, Xu R, Lin L (2017) Multi-label image recognition by recurrently discovering attentional regions, In: Proceedings of the IEEE international conference on computer vision, pp. 464–472

  46. Li Q, Qiao M, Bian W, Tao D (2016) Conditional graphical lasso for multi-label image classification, In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2977–2986

  47. Lee C-W, Fang W, Yeh C-K, Frank Wang Y-C (2018) Multi-label zero-shot learning with structured knowledge graphs, In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1576–1585

  48. Kipf TN, Welling M (2016) Semi-supervised classification with graph convolutional networks, arXiv preprint arXiv:1609.02907

  49. Ge W, Yang S, Yu Y (2018) Multi-evidence filtering and fusion for multi-label classification, object detection and semantic segmentation based on weakly supervised learning, In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1277–1286

  50. Deng J, Dong W, Socher R, Li L-J, Li K, Fei-Fei L (2009) Imagenet: A large-scale hierarchical image database, In: Proceedings of the IEEE conference on computer vision and pattern recognition. Ieee, pp. 248–255

  51. Hu J, Shen L, Sun G (2018) Squeeze-and-excitation networks, In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 7132–7141

  52. Nair V, Hinton GE (2010) Rectified linear units improve restricted boltzmann machines, In: Proceedings of the ICML,

  53. Laffont P-Y, Ren Z, Tao X, Qian C, Hays J (2014) Transient attributes for high-level understanding and editing of outdoor scenes. ACM Trans Graph (TOG) 33(4):1–11

    Article  Google Scholar 

  54. Pennington J, Socher R, Manning CD (2014) Glove: Global vectors for word representation, In: Proceedings of the conference on empirical methods in natural language processing (EMNLP), pp. 1532–1543

  55. Zhang M-L, Zhou Z-H (2007) Ml-knn: a lazy learning approach to multi-label learning. Pattern Recognit 40(7):2038–2048

    Article  Google Scholar 

  56. Benites F, Sapozhnikova E (2015) Haram: a hierarchical aram neural network for large-scale text classification, In: Proceedings of the IEEE international conference on data mining workshop (ICDMW). IEEE, pp. 847–854

Download references

Acknowledgements

This work is supported by the National Key R&D Program of China (2019YFC1408405), National Natural Science Foundation of China (No.61672475, 61872326).

Author information

Authors and Affiliations

  1. College of Information Science and Engineering, Ocean University of China, Qingdao, 266000, China

    Kezhen Xie, Zhiqiang Wei, Lei Huang, Qibing Qin & Wenfeng Zhang

  2. Pilot National Laboratory for Marine Science and Technology, Qingdao, China

    Zhiqiang Wei & Lei Huang

Authors
  1. Kezhen Xie
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Zhiqiang Wei
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Lei Huang
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Qibing Qin
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Wenfeng Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Lei Huang.

Ethics declarations

Conflict of Interest

The authors declared that they have no conflict of interest.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Xie, K., Wei, Z., Huang, L. et al. Graph convolutional networks with attention for multi-label weather recognition. Neural Comput & Applic 33, 11107–11123 (2021). https://doi.org/10.1007/s00521-020-05650-8

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00521-020-05650-8

Keywords

Search

Navigation