Skip to main content
SpringerLink
Log in

Real-time traffic sign detection based on multiscale attention and spatial information aggregator

  • Original Research Paper
  • Published:
Journal of Real-Time Image Processing Aims and scope Submit manuscript

Abstract

Traffic sign detection, as an important part of intelligent driving, can effectively guide drivers to regulate driving and reduce the occurrence of traffic accidents. Currently, the deep learning-based detection methods have achieved very good performance. However, existing network models do not adequately consider the importance of lower-layer features for traffic sign detection. The lack of information on the lower-layer features is a major obstacle to the accurate detection of traffic signs. To solve the above problems, we propose a novel and efficient traffic sign detection method. First, we remove a prediction branch of the YOLOv3 network model to reduce the redundancy of the network model parameters and improve the real-time performance of detection. After that, we propose a multiscale attention feature module. This module fuses the feature information from different layers and refines the features to enhance the Feature Pyramid Network. In addition, we introduce a spatial information aggregator. This enables the spatial information of the lower-layer feature maps to be fused into the higher-layer feature maps. The robustness of our proposed method is further demonstrated by experiments on GTSDB, CCTSDB2021 and TT100k datasets. Specifically, the average execution time on CCTSDB2021 demonstrates the excellent real-time performance of our method. The experimental results show that the method has better accuracy than the original YOLOv3 and YOLOv5 network models.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig.1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6

Similar content being viewed by others

References

  1. Zhang, J., Xie, Z., Sun, J., Zou, X., Wang, J.: A cascaded R-CNN with multiscale attention and imbalanced samples for traffic sign detection. IEEE Access 8, 29742–29754 (2020)

    Article  Google Scholar 

  2. Zhang, J., Wang, W., Lu, C., Wang, J., Sangaiah, A.K.: Lightweight deep network for traffic sign classification. Ann. Telecommun. 75(7), 369–379 (2020)

    Article  Google Scholar 

  3. Maldonado-Bascón, S., Lafuente-Arroyo, S., Gil-Jimenez, P., Gómez-Moreno, H., López-Ferreras, F.: Road-sign detection and recognition based on support vector machines. IEEE Trans. Intell. Transp. Syst. 8(2), 264–278 (2007)

    Article  MATH  Google Scholar 

  4. Jang, C., Kim, C., Kim, D., Lee, M., Sunwoo, M.: Multiple exposure images based traffic light recognition. In: IEEE Intelligent Vehicles Symposium Proceedings, pp. 1313–1318 (2014)

  5. De Charette, R., Nashashibi, F.: Real time visual traffic lights recognition based on spot light detection and adaptive traffic lights templates. In: IEEE Intelligent Vehicles Symposium, pp. 358–363 (2009)

  6. Cai, Z., Gu, M., Li, Y.: Real-time arrow traffic light recognition system for intelligent vehicle. In: Proceedings of the International Conference on Image Processing, Computer Vision, and Pattern Recognition (IPCV), pp.1 (2012)

  7. Zhang, J., Feng, W., Yuan, T., Wang, J., Sangaiah, A.K.: SCSTCF: Spatial-channel selection and temporal regularized correlation filters for visual tracking. Appl. Soft Comput. 118, 108485 (2022)

    Article  Google Scholar 

  8. Zhang, J., Sun, J., Wang, J., Li, Z., Chen, X.: An object tracking framework with recapture based on correlation filters and Siamese networks. Comput. Electr. Eng. 98, 107730 (2022)

    Article  Google Scholar 

  9. Zhang, J.M., Yuan, T.Y., He, Y.Q., Wang, J.: A background-aware correlation filter with adaptive saliency-aware regularization for visual tracking. Neural Comput. Appl. 34(8), 6359–6376 (2022)

    Article  Google Scholar 

  10. Girshick, R., Donahue, J., Darrell, T., Malik, J.: Rich feature hierarchies for accurate object detection and semantic segmentation. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 580–587 (2014)

  11. Girshick, R.: Fast r-cnn. In: Proceedings of the IEEE inter-national conference on computer vision, pp. 1440–1448 (2015)

  12. Ren, S., He, K., Girshick, R., Sun, J.: Faster r-cnn: Towards real-time object detection with region proposal networks. Advances in neural information processing systems, 28 (2015)

  13. Redmon, J., Divvala, S., Girshick, R., Farhadi, A.: You only look once: Unified, real-time object detection. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 779–788 (2016)

  14. Redmon, J., Farhadi, A.: YOLO9000: better, faster, stronger. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 7263–7271 (2017)

  15. Redmon, J., Farhadi, A.: Yolov3: An incremental improvement. arXiv preprint. arXiv:1804.02767, 2018

  16. Liu, W., Anguelov, D., Erhan, D., Szegedy, C., Reed, S., Fu, C. Y., Berg, A. C.: Ssd: Single shot multibox detector. In: European conference on computer vision, pp. 21–37 (2016)

  17. The code address, https://github.com/ultralytics/yolov3

  18. Dai, Y., Gieseke, F., Oehmcke, S., Wu, Y., Barnard, K.: Attentional feature fusion. In: Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 3560–3569 (2021)

  19. Lin, T. Y., Dollár, P., Girshick, R., He, K., Hariharan, B., Belongie, S.: Feature pyramid networks for object detection. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2117–2125 (2017)

  20. Huang, S., Lu, Z., Cheng, R., He, C.: FaPN: Feature-aligned pyramid network for dense image prediction. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 864–873 (2021)

  21. Yu, F., Zhang, Z., Shen, H., Huang, Y., Cai, S., Du, S.: FPGA implementation and image encryption application of a new PRNG based on a memristive Hopfield neural network with a special activation gradient. Chin. Phys. B 31(2), 020505 (2022)

    Article  Google Scholar 

  22. Yang, T., Long, X., Sangaiah, A.K., Zheng, Z., Tong, C.: Deep detection network for real-life traffic sign in vehicular networks. Comput. Netw. 136, 95–104 (2018)

    Article  Google Scholar 

  23. Lu, Y., Lu, J., Zhang, S., Hall, P.: Traffic signal detection and classification in street views using an attention model. Comput. Vis. Media 4(3), 253–266 (2018)

    Article  Google Scholar 

  24. Li, J., Liang, X., Wei, Y., Xu, T., Feng, J.,Yan, S.: Perceptual generative adversarial networks for small object detection. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1222–1230 (2017)

  25. Tian, Y., Gelernter, J., Wang, X., Li, J., Yu, Y.: Traffic sign detection using a multi-scale recurrent attention network. IEEE Trans. Intell. Transp. Syst. 20(12), 4466–4475 (2019)

    Article  Google Scholar 

  26. Luo, H., Yang, Y., Tong, B., Wu, F., Fan, B.: Traffic sign recognition using a multi-task convolutional neural network. IEEE Trans. Intell. Transp. Syst. 19(4), 1100–1111 (2017)

    Article  Google Scholar 

  27. Song, S., Que, Z., Hou, J., Du, S., Song, Y.: An efficient convolutional neural network for small traffic sign detection. J. Syst. Architect. 97, 269–277 (2019)

    Article  Google Scholar 

  28. Hu, J., Shen, L., Sun, G.: Squeeze-and-excitation networks. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 7132–7141 (2018)

  29. Fu, J., Liu, J., Tian, H., Li, Y., Bao, Y., Fang, Z., Lu, H.: Dual attention network for scene segmentation. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 3146–3154 (2019)

  30. Bahdanau, D., Cho, K., Bengio, Y.: Neural machine translation by jointly learning to align and translate. In: The 3rd International Conference on Learning Representations (ICLR), http://arxiv.org/abs/1409.0473v6. ICLR (2015)

  31. Mnih, V., Heess, N., Graves, A.: Recurrent models of visual attention. Advances in neural information processing systems, 27 (2014)

  32. Wang, F., Jiang, M., Qian, C., Yang, S., Li, C., Zhang, H., Tang, X.: Residual attention network for image classification. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3156–3164 (2017)

  33. Long, J., Shelhamer, E., Darrell, T.: Fully convolutional networks for semantic segmentation. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3431–3440 (2015)

  34. Fan, H., Ling, H.: Siamese cascaded region proposal networks for real-time visual tracking. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7952–7961 (2019)

  35. Fan, H., Ling, H.: CRACT: Cascaded Regression-Align-Classification for Robust Visual Tracking. In: 2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 7013–7020 (2021)

  36. Tong, K., Wu, Y., Zhou, F.: Recent advances in small object detection based on deep learning: a review. Image Vis. Comput. 97, 103910 (2020)

    Article  Google Scholar 

  37. Liu, Y., Liu, H.Y., Fan, J.L., Gong, Y.C., Li, Y.H., Wang, F.P., Lu, J.: A survey of research and application of small object detection based on deep learning. Acta Electonica Sin. 48(3), 590 (2020)

    Google Scholar 

  38. Houben, S., Stallkamp, J., Salmen, J., Schlipsing, M., Igel, C.: Detection of traffic signs in real-world images: The German Traffic Sign Detection Benchmark. In: The 2013 international joint conference on neural networks (IJCNN), pp. 1–8 (2013)

  39. Zhang, J.M., Zou, X., Kuang, L.-D., Wang, J., Sherratt, R.S., Yu, X.F.: CCTSDB 2021: a more comprehensive traffic sign detection benchmark. HCIS 12, 23 (2022)

    Google Scholar 

  40. Zhu, Z., Liang, D., Zhang, S., Huang, X., Li, B., Hu, S.: Traffic-sign detection and classification in the wild. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2110–2118 (2016)

  41. Arcos-García, Á., Alvarez-Garcia, J.A., Soria-Morillo, L.M.: Evaluation of deep neural networks for traffic sign detection systems. Neurocomputing 316, 332–344 (2018)

    Article  Google Scholar 

  42. Liu, Y., Peng, J., Xue, J.H., Chen, Y., Fu, Z.H.: TSingNet: Scale-aware and context-rich feature learning for traffic sign detection and recognition in the wild. Neurocomputing 447, 10–22 (2021)

    Article  Google Scholar 

Download references

Acknowledgements

This work was supported by the Open Fund of Key Laboratory of Safety Control of Bridge Engineering, Ministry of Education (Changsha University of Science and Technology) under Grant 21KB06, the Science Fund for Creative Research Groups of Hunan Province under Grant 2020JJ1006, the Natural Science Foundation of Hunan Province under Grant 2021JJ30456, and the National Natural Science Foundation of China under Grant 61972056.

Author information

Authors and Affiliations

  1. Key Laboratory of Safety Control of Bridge Engineering, Ministry of Education, Changsha University of Science and Technology, Changsha, 410114, China

    Jianming Zhang & Zi Ye

  2. School of Computer and Communication Engineering, Changsha University of Science and Technology, Changsha, 410114, China

    Jianming Zhang, Zi Ye, Jin Wang & Jin Zhang

  3. Jinhua Advanced Research Institute, Jinhua, 321013, China

    Xiaokang Jin

Authors
  1. Jianming Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Zi Ye
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Xiaokang Jin
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Jin Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Jin Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Jianming Zhang.

Ethics declarations

Conflict of interest

The authors declare that they have no conflict of interest.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Zhang, J., Ye, Z., Jin, X. et al. Real-time traffic sign detection based on multiscale attention and spatial information aggregator. J Real-Time Image Proc 19, 1155–1167 (2022). https://doi.org/10.1007/s11554-022-01252-w

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11554-022-01252-w

Keywords

Search

Navigation