Skip to main content
SpringerLink
Log in

Transfer learning based hybrid 2D-3D CNN for traffic sign recognition and semantic road detection applied in advanced driver assistance systems

  • Published:
Applied Intelligence Aims and scope Submit manuscript

Abstract

Annually, deep learning algorithms have proven their effectiveness in many vision-based applications, such as autonomous driving, traffic, and congestion monitoring, and so on. In computer vision, accurate traffic sign recognition and semantic road detection are vital challenges for increased safety, which are becoming a major research topic for intelligent transport systems community. In this paper, a deep learning-based driving assistance system has been proposed. To this end, we present hybrid 2D-3D CNN models based on the transfer learning paradigm to achieve better performance on benchmark real-world datasets. The primary goal of transfer learning is to improve the learning process in the target domain while transferring relevant knowledge from the source domain. We combine a pre-trained deep 2D CNN and a shallow 3D CNN to significantly reduce complexity and speed-up the training algorithm. The first model, called Hybrid-TSR, is intended to effectively address the task of traffic sign recognition. Hybrid-SRD is the second architecture that allows the semantic detection of road space through a combination of up-sampling and deconvolutional operations. The experimental results show that the proposed methods have considerable relevance in terms of efficiency and accuracy.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10

Similar content being viewed by others

References

  1. Arcos-García A, Álvarez-García J A, Soria-Morillo LM (2018) Deep neural network for traffic sign recognition systems: an analysis of spatial transformers and stochastic optimisation methods. Neural Netw 99:158–165

    Article  Google Scholar 

  2. Arnab A, et al. (2018) Conditional random fields meet deep neural networks for semantic segmentation. IEEE Signal Process Mag 35(1):37–52

    Article  Google Scholar 

  3. Ayachi R, Afif M, Said Y, Atri M (2019) Traffic signs detection for real-world application of an advanced driving assisting system using deep learning. Neural Process Lett 51:837–851

    Article  Google Scholar 

  4. Badrinarayanan V, Kendall A, Cipolla R (2015) SegNet: a deep convolutional encoder-decoder architecture for image segmentation. arXiv:1511.00561

  5. Bottou L (2010) Large-scale machine learning with stochastic gradient descent. In: Proceedings of COMPSTAT’2010, pp 177–186

  6. Chen Z, Chen Z (2017) RBNet: a deep neural network for unified road and road boundary detection. In: ICONIP, pp 677–687

  7. Cireşan D, Meier U, Masci J, Schmidhuber J (2012) Multi-column deep neural network for traffic sign classification. Neural Netw 32:333–338

    Article  Google Scholar 

  8. Gao P, Yuan R, Wang F, Xiao L, Fujita H, Zhang Y (2020) Siamese attentional keypoint network for high performance visual tracking. Knowl Based Syst 193:105448. https://doi.org/10.1016/j.knosys.2019.105448

    Article  Google Scholar 

  9. Gao P, Zhang Q, Wang F, Xiao L, Fujita H, Zhang Y (2020) Learning reinforced attentional representation for end-to-end visual tracking. Inf Sci 517:52–67. https://doi.org/10.1016/j.ins.2019.12.084

    Article  Google Scholar 

  10. Ge L, Liang H, Yuan J, Thalmann D (2017) 3D Convolutional neural networks for efficient and robust hand pose estimation from single depth images. In: CVPR, pp 5679–5688

  11. Gecer B, Azzopardi G, Petkov N (2017) Color-blob-based COSFIRE filters for object recognition. Image Vis Comput 57:165–174

    Article  Google Scholar 

  12. Greenhalgh J, Mirmehdi M (2012) Real-time detection and recognition of road traffic signs. IEEE Trans Intell Transp Syst 13(4):1498–1506

    Article  Google Scholar 

  13. He K, Zhang X, Ren S, Sun J (2015) Deep residual learning for image recognition. arXiv:1512.03385

  14. Hechri A, Hmida R, Mtibaa A (2015) Robust road lanes and traffic signs recognition for driver assistance system. Int J Comput Sci Eng 10(1/2):202–209

    Google Scholar 

  15. Hmida R, Ben Abdelali A, Mtibaa A (2018) Hardware implementation and validation of a traffic road sign detection and identification system. J Real-Time Image Proc 15(1):13–30

    Article  Google Scholar 

  16. Howard AG, et al. (2017) MobileNets: efficient convolutional neural networks for mobile vision applications. arXiv:1704.04861

  17. Hu K, Shen L, Albanie S, Sun G, Wu E (2017) Squeeze-and-excitation networks. arXiv:1709.01507

  18. Ji S, Xu W, Yang M, Yu K (2013) 3D convolutional neural networks for human action recognition. IEEE Trans Pattern Anal Mach Intell 35(1):221–231. https://doi.org/10.1109/TPAMI.2012.59

    Article  Google Scholar 

  19. John N, Anusha B, Kutty K (2015) A reliable method for detecting road regions from a single image based on color distribution and vanishing point location. Procedia Comput Sci 58:2–9

    Article  Google Scholar 

  20. Kingma DP, Ba J (2014) Adam: a method for stochastic optimization. arXiv:1412.6980

  21. LeCun Y, Bengio Y, Hinton G (2015) Deep learning. Nature 521(7553):436–444

    Article  Google Scholar 

  22. Lin C, Li L, Luo W, Wang KCP, Guo J (2019) Transfer learning based traffic sign recognition using inception-v3 model. Engineering 47(3):242–250

    Google Scholar 

  23. Liu Z, Li D, Ge SS, Tian F (2020) Small traffic sign detection from large image. Appl Intell 50:1–13

    Article  Google Scholar 

  24. Long J, Shelhamer E, Darrell T (2015) Fully convolutional networks for semantic segmentation. In: CVPR, pp 3431–3440

  25. Luo X, Zhu J, Yu Q (2019) Efficient convNets for fast traffic sign recognition. IET Intell Trans Syst 13(6):1011–1015. https://doi.org/10.1049/iet-its.2018.5489

    Article  Google Scholar 

  26. Nam JH, Yang SH, Hu W, Kim BG (2015) A robust real-time road detection algorithm using color and edge information. In: ISVC, pp 532–541

  27. Noh H, Hong S, Han B (2015) Learning deconvolution network for semantic segmentation. arXiv:1505.04366

  28. Oliveira G, et al. (2018) Efficient and robust deep networks for semantic segmentation. Int J Robot Res 37:472–491

    Article  Google Scholar 

  29. Onisr, 2019 French road safety observatory. ONISR. https://doi.org/http://www.onisr.securite-routiere.interieur.gouv.fr/contenus/en/road-safety-policy

  30. Pan SJ, Yang Q (2010) A survey on transfer learning. IEEE Trans Knowl Data Eng 22 (10):1345–1359

    Article  Google Scholar 

  31. Peng X, Li Y, Wei X, Luo J, Murphey YL (2017) Traffic sign recognition with transfer learning. In: SSCI, pp 1–7

  32. Ronneberger O, Fischer P, Brox T (2015) U-net: convolutional networks for biomedical image segmentation. arXiv:1505.04597

  33. Saadna Y, Behloul A, Mezzoudj S (2019) Speed limit sign detection and recognition system using SVM and MNIST datasets. Neural Comput Appl 31:5005–5015. https://doi.org/10.1007/s00521-018-03994-w

    Article  Google Scholar 

  34. Sandler M, Howard A, Zhu M, Zhmoginov A, Chen LC (2018) MobileNetV2: inverted residuals and linear bottlenecks. arXiv:1801.04381

  35. Sermanet P, LeCun Y (2011) Traffic sign recognition with multi-scale convolutional networks. In: IJCNN, pp 2809–2813

  36. Shustanov A, Yakimov P (2017) CNN design for real-time traffic sign recognition. Procedia Eng 201:718–725

    Article  Google Scholar 

  37. Simonyan K, Zisserman A (2014) Very deep convolutional networks for large-scale image recognition. arXiv:1409.1556

  38. Stallkamp J, Schlipsing M, Salmen J, Igel C (2011) The German traffic sign recognition benchmark: a multi-class classification competition. In: IJCNN, pp 1453–1460

  39. Stallkamp J, Schlipsing M, Salmen J, Igel C (2012) Man vs. computer: Benchmarking machine learning algorithms for traffic sign recognition. Neural Netw 32:323–332

    Article  Google Scholar 

  40. Szegedy C, et al. (2014) Going deeper with convolutions. arXiv:1409.4842

  41. Tabernik D, Skočaj D (2019) Deep learning for large-scale traffic-sign detection and recognition. arXiv:1904.00649

  42. Tan M, Le QV (2019) EfficientNet: rethinking model scaling for convolutional neural networks. arXiv:1905.11946

  43. Teichmann M, Weber M, Zöllner M, Cipolla R, Urtasun R (2018) MultiNet: real-time joint semantic reasoning for autonomous driving. In: IV, pp 1013–1020

  44. Tran D et al (2015) Learning spatiotemporal features with 3D convolutional networks. In: ICCV, pp 4489–4497

  45. Wen L, Jo K (2017) Traffic sign recognition and classification with modified residual networks. In: SII, pp 835–840

  46. Who (2019) Global status report on road safety 2018. WHO. http://www.who.int/violence_injury_prevention/road_safety_status/2018/en. Accessed 19 Aug 2019

  47. Wong A, Shafiee MJ, St. Jules M (2018) MicronNet: a highly compact deep convolutional neural network architecture for real-time embedded traffic sign classification. IEEE Access 6:59803–59810. https://doi.org/10.1109/ACCESS.2018.2873948

    Article  Google Scholar 

  48. Xiao L, et al. (2018) Hybrid conditional random field based camera-LIDAR fusion for road detection. Inf Sci 432:543–558

    Article  MathSciNet  Google Scholar 

  49. Yao G, Lei T, Zhong J, Jiang P (2019) Learning multi-temporal-scale deep information for action recognition. Appl Intell 49(6):2017–2029. https://doi.org/10.1007/s10489-018-1347-3

    Article  Google Scholar 

  50. Yu L, Jin M, Zhou K (2019) Multi-channel biomimetic visual transformation for object feature extraction and recognition of complex scenes. Appl Intell 50:792–811. https://doi.org/10.1007/s10489-019-01550-0

    Article  Google Scholar 

  51. Zaklouta F, Stanciulescu B, Hamdoun O (2011) Traffic sign classification using K-d trees and random forests. In: IJCNN, pp 2151–2155

  52. Zang D, et al. (2018) Deep learning–based traffic sign recognition for unmanned autonomous vehicles. Proc Inst Mech Eng Part I: J Syst Control Eng 232(5):497–505

    Google Scholar 

  53. Zhang S, Zhang Z, Sun L, Qin W (2019) One for all: a mutual enhancement method for object detection and semantic segmentation. Appl Sci 10(1):13

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Electrical Department, National Engineering School of Monastir (ENIM), Laboratory of Electronics and Micro-electronics (LR99ES30), Faculty of Sciences of Monastir (FSM), University of Monastir, Monastir, Tunisia

    Khaled Bayoudh & Abdellatif Mtibaa

  2. Electrical Department, National Engineering School of Monastir (ENIM), Laboratory of Control, Electrical Systems and Environment (LASEE), National Engineering School of Monastir, University of Monastir, Monastir, Tunisia

    Fayçal Hamdaoui

Authors
  1. Khaled Bayoudh
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Fayçal Hamdaoui
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Abdellatif Mtibaa
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Khaled Bayoudh.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Bayoudh, K., Hamdaoui, F. & Mtibaa, A. Transfer learning based hybrid 2D-3D CNN for traffic sign recognition and semantic road detection applied in advanced driver assistance systems. Appl Intell 51, 124–142 (2021). https://doi.org/10.1007/s10489-020-01801-5

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10489-020-01801-5

Keywords

Search

Navigation