Skip to main content

Advertisement

SpringerLink
Log in

Image-based automatic traffic lights detection system for autonomous cars: a review

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

From the early stages of autonomous vehicle’s development, traffic light detection/perception system have been an important area of research for making collision safe self-driving vehicles. Here Automatic Traffic Light Detection System (ALTDS) helps in accurate detection of Traffic Lights for Autonomous vehicles and Driver assistance systems (DAS). These vision-based system captures images using a camera mounted on a car and no other sensors. As traffic light is a small object in a real-time traffic scenario, so high-quality images are the main success factor of ATLDS. This paper elucidates the ideas and challenges that needs to be worked upon for better traffic light detection system used in self-driving cars. In this paper, we present a state-of-art review of various techniques used in traffic light detection. Different ATLDS techniques such as preprocessing, segmentation, feature extraction, classification and post-processing are categorized based on the features used at each stage, a comparison of the pros and cons of each technique is also provided. The hardware/software limitations, laws related to self-driving vehicles along-with simulation environments are also provided. The conclusion and future scope are given at the end.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3

Similar content being viewed by others

Data availability

Not Applicable.

References

  1. Almagambetov A, Velipasalar S, Baitassova A (2015) Mobile standards-based traffic light detection in assistive devices for individuals with color-vision deficiency. IEEE Trans Intell Transp Syst 16:1305–1320. https://doi.org/10.1109/TITS.2014.2361139

    Article  Google Scholar 

  2. Aneesh AN, Shine L, Pradeep R, Sajith V (2019) Real-time traffic light detection and recognition based on deep RetinaNet for self driving cars. 2019 2nd Int Conf Intell Comput Instrum control Technol ICICICT 2019 1554–1557. https://doi.org/10.1109/ICICICT46008.2019.8993293

  3. Any G, Ayesha C (2019) 2019 IEEE intelligent vehicles symposium, IV 2019. IEEE Intell. Veh. Symp. Proc. 2019-June

  4. Arena F, Pau G, Ralescu A, et al (2022) An Innovative Framework for Dynamic Traffic Lights Management Based on the Combined Use of Fuzzy Logic and Several Network Architectures J Adv Transp 2022 https://doi.org/10.1155/2022/1383349

  5. Badue C (2019) Self-driving cars: a survey

  6. Barnes D, Maddern W, Posner I (2015) Exploiting 3D semantic scene priors for online traffic light interpretation. In: 2015 IEEE intelligent vehicles symposium (IV). IEEE, pp 573–578. https://doi.org/10.1109/IVS.2015.7225746

  7. Behrendt K, Novak L, Botros R (2017) A deep learning approach to traffic lights: Detection, tracking, and classification. In: 2017 IEEE International Conference on Robotics and Automation (ICRA). IEEE, pp 1370–1377. https://doi.org/10.1109/ICRA.2017.7989163

  8. Belkouri D, Laing R, Gray D (2022) Through the eyes of Autonomous Vehicles–using laser scanning technology to engage the public via the analysis of journeys seen from a different perspective. Transp Res Procedia 60:496–503. https://doi.org/10.1016/j.trpro.2021.12.064

  9. Benjelloun F, El Manaa I, Sabri MA, Yahyaouy A, Aarab A (2020) The comparison between two methods of object detection: Fast Yolo model and Delaunay Triangulation. In: 2020 International Conference on Intelligent Systems and Computer Vision (ISCV). IEEE, pp 1–6. https://doi.org/10.1109/ISCV49265.2020.9204197

  10. Bhalla A, Nikhila MS, Singh P (2020) Simulation of self-driving car using deep learning. Proc 3rd Int Conf Intell sustain Syst ICISS 2020 519–525. https://doi.org/10.1109/ICISS49785.2020.9315968

  11. Chen SC (2019) Multimedia for autonomous driving. IEEE MultiMedia 26(3):5–8. https://doi.org/10.1109/MMUL.2019.2935397

  12. Chen Z, Huang X (2016) Accurate and reliable detection of traffic lights using multiclass learning and multiobject tracking. IEEE Intell Transp Syst Mag 8(4):28–42. https://doi.org/10.1109/MITS.2016.2605381

  13. Daily M, Laboratories HRL, Medasani S Self-Driving Cars

  14. De Charette R, Nashashibi F (2009) Real time visual traffic lights recognition based on spot light detection and adaptive traffic lights templates. In: 2009 IEEE Intelligent Vehicles Symposium. IEEE, pp 358–363

  15. De Charette R, Nashashibi F (2009) Traffic light recognition using image processing compared to learning processes. In: 2009 IEEE/RSJ International Conference on Intelligent Robots and Systems. IEEE, pp 333–338. https://doi.org/10.1109/IROS.2009.5353941

  16. Dean MD, Kockelman K (2022) Our self-driving future will be shaped by policies of today. Nat Electron 5(1):2–4. https://doi.org/10.1038/s41928-021-00708-4

  17. Diaz-Cabrera M, Cerri P, Medici P (2015) Robust real-time traffic light detection and distance estimation using a single camera. Expert Syst Appl 42(8):3911–3923. https://doi.org/10.1016/j.eswa.2014.12.037

  18. Diaz-Cabrera M, Cerri P, Sanchez-Medina J (2012) Suspended traffic lights detection and distance estimation using color features. In: 2012 15th international IEEE conference on intelligent transportation systems. IEEE, pp 1315–1320. https://doi.org/10.1109/ITSC.2012.6338765

  19. Ditty MA, Hicok G, Sweedler J, Farabet C, Yousuf MA, Chan TY, ... Hettena D (2019) U.S. Patent Application No. 16/186,473

  20. DriveU DriveU. https://www.uni-ulm.de/in/iui-drive-u/projekte/driveu-traffic-light-dataset/

  21. Fairfield N (2011) Traffic light mapping and detection. 5421–5426

  22. Fan Q, Brown L, Smith J (2016) A closer look at Faster R-CNN for vehicle detection. In: 2016 IEEE intelligent vehicles symposium (IV). IEEE, pp 124–129. https://doi.org/10.1109/IVS.2016.7535375

  23. Haltakov V, Mayr J, Unger C, Ilic S (2015) Semantic segmentation based traffic light detection at day and at night. In: German conference on pattern recognition. Springer, Cham, pp 446–457. https://doi.org/10.1007/978-3-319-24947-6

  24. Gao F, Wang C (2020) Hybrid strategy for traffic light detection by combining classical and self-learning detectors. IET Intell Transp Syst 14:735–741. https://doi.org/10.1049/iet-its.2019.0782

    Article  Google Scholar 

  25. Gautam S, Kumar A (2022) Automatic traffic light detection for self-driving cars using transfer learning. In: Lecture Notes in Networks and Systems

  26. Vallespi-Gonzalez C (2017) U.S. Patent No. 9,672,446. Washington, DC: U.S. Patent and Trademark Office

  27. Greenblatt NA (2016) Self-driving cars and the law. IEEE Spectr 53:46–51. https://doi.org/10.1109/MSPEC.2016.7419800

    Article  Google Scholar 

  28. Gupta A, Anpalagan A, Guan L, Khwaja AS (2021) Deep learning for object detection and scene perception in self-driving cars: Survey, challenges, and open issues. Array 10:100057. https://doi.org/10.1016/j.array.2021.100057

  29. Hassan N, Ming KW,Wah CK (2020) A comparative study on hsv-based and deep learning-based object detection algorithms for pedestrian traffic light signal recognition. In: 2020 3rd International Conference on Intelligent Autonomous Systems (ICoIAS). IEEE, pp 71–76.https://doi.org/10.1109/ICoIAS49312.2020.9081854

  30. Hirabayashi M, Sujiwo A, Monrroy A, Kato S, Edahiro M (2019) Traffic light recognition using high-definition map features. Rob Auton Syst 111:62–72. https://doi.org/10.1016/j.robot.2018.10.004

  31. Huh JH, Seo YS (2019) Understanding edge computing: Engineering evolution with artificial intelligence. IEEE Access 7:164229–164245. https://doi.org/10.1109/ACCESS.2019.2945338

  32. Iftikhar M, Riaz O, Ali T, Mumtaz S, Sharif W, Arshad H (2022) Traffic light detection: a cost effective approach

  33. Jensen MB, Philipsen MP,Møgelmose A, Moeslund TB, Trivedi MM (2016) Vision for looking at traffic lights: Issues, survey, and perspectives. IEEE Trans Intell Transp Syst 17(7):1800–1815

  34. Ji X, Cheng X (2021) An adaptive multisensor image fusion method based on monogenic features. IEEE Sensors J 21:15598–15606. https://doi.org/10.1109/JSEN.2020.3009242

    Article  Google Scholar 

  35. Jie Y, Xiaomin C, Pengfei G, Zhonglong X (2013) A new traffic light detection and recognition algorithm for electronic travel aid. In: 2013 Fourth International Conference on Intelligent Control and Information Processing (ICICIP). IEEE, pp 644–648. https://doi.org/10.1109/ICICIP.2013.6568153

  36. Jin C, Ge Z, Zhu Z, He L (2018) Novel scheme for image-based traffic-light detection. Int Comput Signals Syst Conf 2018:640–643

  37. John V, Yoneda K, Qi B, Liu Z, Mita S (2014) Traffic light recognition in varying illumination using deep learning and saliency map. In: 17th International IEEE Conference on Intelligent Transportation Systems (ITSC). IEEE, pp 2286–2291. https://doi.org/10.1109/ITSC.2014.6958056

  38. Kim S, Han DS (2017) Real time traffic light detection algorithm based on color map and multilayer HOG-SVM. J Broadcast Eng 22(1):62–69.https://doi.org/10.5909/jbe.2017.22.1.62

  39. Kim HK., Park JH, Jung HY (2011) Effective traffic lights recognition method for real time driving assistance systemin the daytime. Int J Electr Comput Eng 5(11):1429–1432

  40. Kim HK, Shin YN, Kuk SG, Park JH, Jung HY (2013) Night-time traffic light detection based on svm with geometric moment features. Int J Comput Inform Eng 7(4):472–475

  41. Kim HK, Park JH, Jung HY (2018) An efficient color space for deep-learning based traffic light recognition. J Adv Transp 2018. https://doi.org/10.1155/2018/2365414

  42. Kim HK, Yoo KY, Park JH, Jung HY (2019) Traffic light recognition based on binary semantic segmentation network. Sensors 19(7):1700. https://doi.org/10.3390/s19071700

  43. Kim Y, Huh JH, ChungM (2021) Traffic inference system using correlation analysis with various predicted big data. Electronics 10(3):354. https://doi.org/10.3390/electronics10030354

  44. Kim J, Cho H, Hwangbo M, et al Deep Traffic Light Detection for Self-driving Cars from a Large-scale Dataset

  45. Ichikawa K, Kindo T (2016) U.S. Patent No. 9,304,513. Washington, DC: U.S. Patent and Trademark Office

  46. Kulkarni R, Dhavalikar S, Bangar S (2018) Traffic light detection and recognition for self driving cars using deep learning. In: 2018 Fourth International Conference on Computing Communication Control and Automation (ICCUBEA). IEEE, pp 1–4. https://doi.org/10.1109/ICCUBEA.2018.8697819

  47. Lee GG, Park BK (2017) Traffic light recognition using deep neural networks. 2017 IEEE Int Conf Consum Electron ICCE 2017:277–278. https://doi.org/10.1109/ICCE.2017.7889317

    Article  Google Scholar 

  48. Levinson J, Askeland J, Dolson J, Thrun S Traffic Light Mapping, Localization, and State Detection for Autonomous Vehicles

  49. Li J, Bao H, Han X, Pan F, Pan W, Zhang F, Wang D (2017) Real-time self-driving car navigation and obstacle avoidance using mobile 3D laser scanner and GNSS. Multimed Tools Appl 76:23017–23039. https://doi.org/10.1007/s11042-016-4211-7

    Article  Google Scholar 

  50. LISA. http://cvrr.ucsd.edu/LISA/datasets.html

  51. LiuW,Wang Z, Zhou B, Yang S, Gong Z (2021) Real-time signal light detection based on yolov5 for railway. In: IOP Conference Series: Earth and Environmental Science 769(4):042069. IOP Publishing

  52. Lu Y, Lu J, Zhang S, Hall P (2018) Traffic signal detection and classification in street views using an attention model. Comput Vis Media 4:253–266. https://doi.org/10.1007/s41095-018-0116-x

    Article  Google Scholar 

  53. Malley RO, Jones E, Glavin M (2010) Rear-Lamp Vehicle Detection and Tracking in Low-Exposure Color Video for Night Conditions 11:453–462, Rear-Lamp Vehicle Detection and Tracking in Low-Exposure Color Video for Night Conditions

  54. Michael M, Schlipsing M (2015) Extending traffic light recognition: efficient classification of phase and pictogram. Proc Int Jt Conf neural networks 2015-Septe: https://doi.org/10.1109/IJCNN.2015.7280499

  55. Muller J, Dietmayer K (2018) Detecting traffic lights by single shot detection. In: 2018 21st International Conference on Intelligent Transportation Systems (ITSC). IEEE, pp 266–273. https://doi.org/10.1109/ITSC.2018.8569683

  56. Ni J, Chen Y, Chen Y, Zhu J, Ali D, Cao W (2020) A survey on theories and applications for self-driving cars based on deep learning methods. Appl Sci 10(8):2749. https://doi.org/10.3390/APP10082749

  57. Nienh D, Drescher M, Marius JZ (2010) Visual state estimation of traffic lights using hidden Markov models. 0–5. https://doi.org/10.1109/ITSC.2010.5625241

  58. Omachi M, Omachi S (2010) Detection of traffic light using structural information. In: IEEE 10th international conference on signal processing proceedings. IEEE, pp 809–812

  59. Ouyang Z, Niu J, Liu Y, Guizani M (2020) Deep CNN-based real-time traffic light detector for self-driving vehicles. IEEE Trans Mob Comput 19:300–313. https://doi.org/10.1109/TMC.2019.2892451

    Article  Google Scholar 

  60. Ozcelik Z, Tastimur C, Karakose M, Akin E (2017) A vision based traffic light detection and recognition approach for intelligent vehicles. 2nd Int Conf Comput Sci Eng UBMK 2017 424–429. https://doi.org/10.1109/UBMK.2017.8093430

  61. Pascal Voc. https://gluon-cv.mxnet.io/build/examples_datasets/pascal_voc.html.

  62. Philipsen MP, Jensen MB, Mogelmose A, Moeslund TB, Trivedi MM (2015) Traffic light detection: A learning algorithm and evaluations on challenging dataset. In: 2015 IEEE 18th International Conference on Intelligent Transportation Systems. IEEE, pp 2341–2345. https://doi.org/10.1109/ITSC.2015.378

  63. Rao SS, Desai SR (n.d.) 2nd International Conference on IoT Based Control Networks and Intelligent Systems (ICICNIS 2021) Machine Learning based Traffic Light Detection and IR Sensor based Proximity Sensing for Autonomous Cars

  64. Rubio JC, Serrat J, López AM, Ponsa D (2012) Multiple-Target Tracking for Intelligent Headlights Control 13:594–605

  65. Saha D, De S (2022 Practical Self-Driving Cars: Survey of the State-of-the-Art. https://doi.org/10.20944/preprints202202.0123.v1

  66. Saini S, Nikhil S, Konda KR, et al (2017) An efficient vision-based traffic light detection and state recognition for autonomous vehicles. IEEE Intell Veh Symp proc 606–611. https://doi.org/10.1109/IVS.2017.7995785

  67. Salarian M, Manavella A (2015) A vision based system for traffic lights recognition. https://doi.org/10.1109/IntelliSys.2015.7361224

  68. Salti S, Petrelli A, Tombari F, Fioraio N, Di Stefano L (2013) A traffic sign detection pipeline based on interest region extraction. In: The 2013 International Joint Conference on Neural Networks (IJCNN). IEEE, pp 1–7

  69. Shen Y, Ozguner U, Redmill K, Liu J (2009) A robust video based traffic light detection algorithm for intelligent vehicles. IEEE Intell Veh Symp proc 521–526. https://doi.org/10.1109/IVS.2009.5164332

  70. Shi Z, Zou Z, Zhang C (2016) Real-time traffic light detection with adaptive background suppression filter. IEEE Trans Intell Transp Syst 17:690–700. https://doi.org/10.1109/TITS.2015.2481459

    Article  Google Scholar 

  71. Singh P, DiwakarM, Cheng X, Shankar A (2021) A new wavelet-based multi-focus image fusion technique using method noise and anisotropic diffusion for real-time surveillance application. J Real-Time Image Process 18(4):1051–1068.https://doi.org/10.1007/s11554-021-01125-8

  72. Siogkas G, Skodras E, Dermatas E (2012) Traffic lights detection in adverse conditions using color, symmetry and spatiotemporal information. In: VISAPP (1), pp 620–627. https://doi.org/10.5220/0003855806200627

  73. Sooksatra S, Kondo T (2014) Red traffic light detection using fast radial symmetry transform. In: 2014 11th international conference on electrical engineering/electronics, computer, telecommunications and information technology (ECTI-CON). IEEE, pp 1–6. https://doi.org/10.1109/ECTICon.2014.6839767

  74. Surasak T, Takahiro I, Cheng CH et al (2018) Histogram of oriented gradients for human detection in video. Proc 2018 5th Int Conf bus Ind res smart Technol next Gener information. Eng Bus Soc Sci ICBIR 2018:172–176. https://doi.org/10.1109/ICBIR.2018.8391187

    Article  Google Scholar 

  75. Thao LQ, Cuong DD, Anh NT, et al (2022) Automatic traffic red-light violation detection using AI. Ing des Syst d’Information 27:75–80. https://doi.org/10.18280/isi.270109

  76. Tran THP, Pham CC, Nguyen TP, Duong TT, Jeon JW (2016) Real-time traffic light detection using color density. In: 2016 IEEE International Conference on Consumer Electronics-Asia (ICCE-Asia). IEEE, pp 1–4. https://doi.org/10.1109/ICCE-Asia.2016.7804791

  77. Trehard G, Pollard E, Bradai B, Nashashibi F (2014) Tracking both pose and status of a traffic light via an interacting multiple model filter. In: 17th international conference on information fusion (FUSION). IEEE, pp 1–7

  78. Trehard G, Pollard E, Bradai B, Nashashibi F (2014) Tracking both pose and status of a traffic light via an interacting multiple model filter. In: 17th international conference on information fusion (FUSION). IEEE, pp 1–7

  79. Dataset Link : TT100K. https://cg.cs.tsinghua.edu.cn/traffic-sign

  80. Vitas D, Tomic M, Burul M (2020) Traffic light detection in autonomous driving systems. IEEE Consum Electron Mag 9(4):90–96. https://doi.org/10.1109/MCE.2020.2969156

  81. Wang JG, Zhou LB (2019) Traffic light recognition with high dynamic range imaging and deep learning. IEEE Trans Intell Transp Syst 20:1341–1352. https://doi.org/10.1109/TITS.2018.2849505

    Article  Google Scholar 

  82. Wang C, Jin T, Yang M, Wang B (2011) Robust and real-time traffic lights recognition in complex urban environments. Int J Comput Intell Syst 4(6):1383–1390. https://doi.org/10.1080/18756891.2011.9727889

  83. Wang K, Tang X, Zhao S, Zhou Y (2021) Simultaneous detection and tracking using deep learning and integrated channel feature for ambint traffic light recognition. J Ambient Intell Humaniz Comput 13:271–281. https://doi.org/10.1007/s12652-021-02900-y

    Article  Google Scholar 

  84. Widyantoro DH, Saputra KI (2016) Traffic lights detection and recognition based on color segmentation and circle hough transform. In: 2015 International Conference on Data and Software Engineering (ICoDSE). IEEE, pp 237–240. https://doi.org/10.1109/ICODSE.2015.7437004

  85. Wonghabut P, Kumphong J, Ung-Arunyawee R, Leelapatra W, Satiennam T (2018) Traffic light color identification for automatic traffic light violation detection system. In: 2018 International Conference on Engineering, Applied Sciences, and Technology (ICEAST). IEEE, pp 1–4. https://doi.org/10.1109/ICEAST.2018.8434400

  86. Yabuuchi K, Hirano M, Senoo T, Kishi N, Ishikawa M (2020) Real-time traffic light detection with frequency patterns using a high-speed camera. Sensors (Switzerland) 20:1–19. https://doi.org/10.3390/s20144035

    Article  Google Scholar 

  87. Yoneda K, Kuramoto A, Suganuma N, Asaka T, Aldibaja M., Yanase R. (2020) Robust traffic light and arrow detection using digital map with spatial prior information for automated driving. Sensors 20(4):1181.https://doi.org/10.3390/s20041181

  88. Zhang Y, Xue J, Zhang G, Zhang Y, Zheng N (2014) A multi-feature fusion based traffic light recognition algorithm for intelligent vehicles. In: Proceedings of the 33rd Chinese control conference. IEEE, pp 4924–4929. https://doi.org/10.1109/ChiCC.2014.6895775

  89. Zhou Y, Chen Z, Huang X (2016) A system-on-chip FPGA design for real-time traffic signal recognition system. In: 2016 IEEE International Symposium on Circuits and Systems (ISCAS). IEEE, pp 1778–1781. https://doi.org/10.1109/ISCAS.2016.7538913

  90. Zhou X, Yuan J, Liu H (2017) Real-time traffic light recognition based on c-hog features. Comput Inform 36(4):793–814.https://doi.org/10.4149/cai_2017_4_793

Download references

Code availability

Not Applicable.

Funding

Not Applicable.

Author information

Authors and Affiliations

  1. Panjab University, Chandigarh, India

    Sarita Gautam & Anuj Kumar

Authors
  1. Sarita Gautam
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Anuj Kumar
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Sarita Gautam.

Ethics declarations

Conflict of interests

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Gautam, S., Kumar, A. Image-based automatic traffic lights detection system for autonomous cars: a review. Multimed Tools Appl 82, 26135–26182 (2023). https://doi.org/10.1007/s11042-023-14340-1

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-023-14340-1

Keywords

Search

Navigation