Skip to main content

Advertisement

Springer Nature Link
Log in

Applications of pre-trained CNN models and data fusion techniques in Unity3D for connected vehicles

  • Published:
Applied Intelligence Aims and scope Submit manuscript

Abstract

Intelligent Transportation Systems (ITS) aim to enhance road safety and Internet of Things (IoT)-related solutions are crucial in achieving this objective. By leveraging Vehicle-to-Vehicle (V2V) and Vehicle-to-Infrastructure (V2I) technologies, drivers can access valuable information about their surroundings. This research utilized the Unity 3D game engine to simulate various traffic scenarios, exploring a stochastic environment with two data sources: camera and road sign labels. We developed a full-duplex communication system to enable the communication between Python and Unity. This allows the vehicle to capture images in Unity and classify them using Convolutional Neural Network (CNN) models coded in Python. To improve road sign detection accuracy, we applied multi-sensor Data Fusion (DF) techniques to fuse the information received from the sources. We applied DF methods such as the Kalman filter, Dempster-Shafer theory, and Fuzzy Integral Operators to combine the two sources of information. Furthermore, our proposed CNN model incorporates an Ordered Weighted Averaging (OWA) layer to fuse information from three pre-trained CNN models. Our results show that the proposed model integrating the OWA layer achieved an accuracy of 98.81%, outperforming six state-of-the-art models. We compared the Extended Kalman Filter (EKF) and Unscented Kalman Filter (UKF). In our work, EKF exhibited a lower execution time (0.02 seconds), yielding less accurate results. UKF, however, provided a more accurate estimate while being more computationally complex. Furthermore, the Dempster-Shafer model showed approximately 30% better accuracy compared to the Fuzzy Integral Operator. Using this methodology on autonomous vehicles in our virtual environment led to making more accurate decisions, even in a variety of weather conditions and accident scenarios. The findings of this research contribute to the development of more efficient and safer vehicles.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16
Fig. 17
Fig. 18
Fig. 19
Fig. 20
Fig. 21
Fig. 22
Fig. 23
Fig. 24
Fig. 25
Fig. 26
Fig. 27
Fig. 28
Fig. 29
Fig. 30
Fig. 31
Fig. 32
Fig. 33
Fig. 34
Fig. 35
Fig. 36

Similar content being viewed by others

Explore related subjects

Discover the latest articles and news from researchers in related subjects, suggested using machine learning.

Data Availability

The data underpinning the conclusions of this study is accessible from the corresponding author, [Behzad Moshiri], upon reasonable request.

Notes

  1. https://drive.google.com/file/d/16J187jj1glVNeV64m6hLMHXvP66jACX2/view?usp=sharing

References

  1. Meng T, Jing X, Yan Z, Pedrycz W (2020) A survey on machine learning for data fusion. Inf Fusion 57:115–129

    Article  MATH  Google Scholar 

  2. Thakur A, Malekian R, Bogatinoska D (2017) Internet of things based solutions for road safety and traffic management in intelligent transportation systems. ICT Innovations 2017:47–56

    Google Scholar 

  3. Parrado N, Donoso Y. (2015) Congestion based mechanism for route discovery in a V2I–V2V system applying smart devices and IoT. Sensors 15(4):7768–7806

  4. Kuzmic, J, Rudolph, G., 2020. Unity 3D Simulator of Autonomous Motorway Traffic Applied to Emergency Corridor Building. Proceedings of the 5th International Conference on Internet of Things, Big Data and Security,

  5. Islam M, Anderson D, Pinar A, Havens T, Scott G, Keller J (2020) Enabling Explainable Fusion in Deep Learning With Fuzzy Integral Neural Networks. IEEE Trans Fuzzy Syst 28(7):1291–1300

    Article  MATH  Google Scholar 

  6. Bhatti F, Shah MA, Maple C, Islam SU (2019) A novel internet of things-enabled accident detection and reporting system for smart city environments. Sensors 19(9):2017

  7. Gadekar PA (2018) Smart application for post-accident management using IoT. Int J Adv Res Comput Sci 9(2):684–687

    Article  MATH  Google Scholar 

  8. Khaliq KA, Chughtai O, Shahwani A, Qayyum A, Pannek J (2019) Road accidents detection, data collection and data analysis using V2X communication and Edge/Cloud computing. Electronics 8(8):896

    Article  Google Scholar 

  9. Filev D, Yager RR (1998) On the issue of obtaining OWA operator weights. Fuzzy Sets Syst 94(2):157–169

    Article  MathSciNet  MATH  Google Scholar 

  10. Garcia-Roger D, Martin-Sacristan D, Roger S, Monserrat J, Kousaridas A, Spapis P, Ayaz S, Zhou C (2018) 5G multi-antenna V2V channel modeling with a 3D game engine. In: 2018 IEEE wireless communications and networking conference workshops (WCNCW),

  11. Mahmood Z (ed) (2020) Connected vehicles in the internet of things: Concepts, technologies and frameworks for the IoV. Springer International Publishing, Cham

    MATH  Google Scholar 

  12. Cai L, Pan J, Zhao L, Shen X (2017) Networked electric vehicles for green intelligent transportation. IEEE Common Stand Mag 1(2):77–83

    Article  MATH  Google Scholar 

  13. Lyu F, Li M, Shen X (2020) Vehicular networking for road safety. Springer International Publishing, Cham

    Book  MATH  Google Scholar 

  14. Vaidya B, Mouftah HT (2020) IoT applications and services for connected and autonomous electric vehicles. Arab J Sci Eng 45(4):2559–2569

    Article  MATH  Google Scholar 

  15. Elmore P, Anderson DT, Petry F (2020) Evaluation of Heterogeneous Uncertain Information Fusion. J Ambient Intell Comput 11:799–811

    Article  MATH  Google Scholar 

  16. Anderson DT, Deardorff M, Havens T, Kakula S, Wilkin T, Islam M, Pinar A, Buck A (2020) Fuzzy Integral = Contextural Linear Order Statistic

  17. Cannaday A, Davis C, Scott G, Ruprecht B, Anderson DT (2020) Broad area search and detection of surface-to-air missile sites using spatial fusion of component object detections from deep neural networks. Accepted by IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing

  18. Islam M, Murray B, Buck A, Anderson DT, Scott G, Popescu M, Keller J (2020) Extending the Morphological Hit-or-Miss Transform to Deep Neural Networks. IEEE Transactions on Neural Networks and Learning Systems

  19. Islam M, Anderson DT, Petry F, Elmore P (2019) An efficient evolutionary algorithm to optimize the Choquet integral, in International journal of intelligent systems, vol 34 (3)

  20. Scott G, Hagan K, Marcum R, Hurt J, Anderson DT, Davis C (2018) Enhanced Fusion of Deep Neural Networks for Classification of Benchmark High-Resolution Image Datasets. IEEE Geosci Remote Sens Soc Lett 15(9):1451–1455

    Article  Google Scholar 

  21. Durst P, Goodin C, Bethel C, Anderson DT, Carruth D, Lim H (2018) A perception-based fuzzy route planning algorithm for autonomous unmanned ground vehicles. Unmanned Syst 6(4):251–266

    Article  Google Scholar 

  22. Ahmed HU, Huang Y, Lu P (2021) A review of car-following models and modeling tools for human and autonomous-ready driving behaviors in micro-simulation. Smart Cities 4(1):314–335

    Article  MATH  Google Scholar 

  23. Kordestani M, Dehghani M, Moshiri B, Saif M (2020) A new fusion estimation method for multi-rate multi-sensor systems with missing measurements. IEEE Access 8:47522–47532

    Article  Google Scholar 

  24. Welch G, Bishop G (1995) An introduction to the Kalman filter. Technical Report TR95-041. University of North Carolina at Chapel Hill, Department of Computer Science

  25. Sakai A, Ingram D, Dinius J, Chawla K, Raffin A, Paques A (2018) PythonRobotics: a Python code collection of robotics algorithms. arXiv:1808.10703

  26. Olaverri-Monreal C, Errea-Moreno J, Díaz-Álvarez A, Biurrun-Quel C, Serrano-Arriezu L, Kuba M (2018) Connection of the SUMO Microscopic Traffic Simulator and the Unity 3D Game Engine to Evaluate V2X Communication-Based Systems. Sensors 18(12):4399

    Article  Google Scholar 

  27. Xu Z (2004) Uncertain linguistic aggregation operators based approach to multiple attribute group decision making under uncertain linguistic environment. Inf Sci 168:171–184

    Article  MATH  Google Scholar 

  28. Yager RR (1988) On ordered weighted averaging aggregation operators in multi-criteria decision making. IEEE Trans Syst Man Cybern 18:183–190

    Article  MATH  Google Scholar 

  29. Xu Z (2005) An overview of methods for determining OWA weights. Int J Intell Syst 20(8):843–865

    Article  MATH  Google Scholar 

  30. Wan E, Van Der Merwe R (2000) n.d. The unscented Kalman filter for nonlinear estimation. In: Proceedings of the IEEE 2000 adaptive systems for signal processing. Communications, and Control Symposium

  31. Dominguez-Catena I, Paternain D, Galar M (2021) A study of OWA operators learned in convolutional neural networks. Appl Sci 11(16):7195

    Article  MATH  Google Scholar 

  32. Forcén J, Pagola M, Barrenechea E, Bustince H (2020) Learning ordered pooling weights in image classification. Neurocomputing. 411:45–53

    Article  MATH  Google Scholar 

  33. Basiri J, Taghiyareh F, Moshiri B (2010) A hybrid approach to predict churn. In: 2010 IEEE asia-pacific services computing conference

  34. Moradi M, Delavar M, Moshiri B, Khamespanah F (2014) A novel approach to support majority voting in spatial group MCDM using density induced OWA operator forseismic vulnerability assessment. The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, XL-2/W3, pp 209–214

  35. Ghaderi M, Yazdani N, Moshiri B, Tayefeh Mahmoudi M (2010) A new approach for text feature selection based on OWA operator. In: 2010 5th International symposium on telecommunications

  36. Fayyad J, Jaradat MA, Gruyer D, Najjaran H (2020) Deep learning sensor fusion for autonomous vehicle perception and localization: A review. Sensors 20(15):4220

    Article  MATH  Google Scholar 

  37. Kolar P, Benavidez P, Jamshidi M (2020) Survey of datafusion techniques for laser and vision based sensor integration for autonomous navigation. Sensors 20(8):2180

    Article  MATH  Google Scholar 

  38. Yeong DJ, Velasco-Hernandez G, Barry J, Walsh J (2021) Sensor and sensor fusion technology in autonomous vehicles: A review. Sensors 21(6):2140

    Article  MATH  Google Scholar 

  39. Pan H, Sun W, Sun Q, Gao H (2021) Deep learning based data fusion for sensor fault diagnosis and tolerance in autonomous vehicles. Chinese J Mech Eng 34(1):1–11

    Article  MATH  Google Scholar 

  40. Alam F, Mehmood R, Katib I, Albogami NN, Albeshri A (2017) Data fusion and IoT for smart ubiquitous environments: A survey. Ieee Access 5:9533–9554

    Article  Google Scholar 

  41. Miglani A, Kumar N (2019) Deep learning models for traffic flow prediction in autonomous vehicles: A review, solutions, and challenges. Veh Commun 20:100184

    MATH  Google Scholar 

  42. Vargas J, Alsweiss S, Toker O, Razdan R, Santos J (2021) An overview of autonomous vehicles sensors and their vulnerability to weather conditions. Sensors 21(16):5397

    Article  Google Scholar 

  43. Hu JW, Zheng BY, Wang C, Zhao CH, Hou XL, Pan Q, Xu Z (2020) A survey on multi-sensor fusion based obstacle detection for intelligent ground vehicles in off-road environments. Front Inf Technol Electr Eng 21(5):675–692

    Article  MATH  Google Scholar 

  44. Kuutti S, Bowden R, Jin Y, Barber P, Fallah S (2020) A survey of deep learning applications to autonomous vehicle control. IEEE Trans Intell Trans Syst 22(2):712–733

    Article  Google Scholar 

  45. Himeur Y, Rimal B, Tiwary A, Amira A (2022) Using artificial intelligence and data fusion for environmental monitoring: A review and future perspectives. Information Fusion

  46. Rezaie J, Moshiri B, Araabi BN, Asadian A (2007) GPS/INS integration using nonlinear blending filters. In SICE Annual Conference 2007 (pp 1674-1680) IEEE

  47. Majumder A (2021) Setting up ML agents Toolkit. In: Deep reinforcement learning in unity. Berkeley, CA:Apress, pp 155–207

  48. The Ho QN, Do TT, Minh PS, Nguyen V-T, Nguyen VTT (2023) Turning chatter detection using a multi-input convolutional neural network via image and sound signal. Machines 11:644

    Article  MATH  Google Scholar 

  49. Oyedeji OA, Khan S, Erkoyuncu JA (2024) Application of CNN for multiple phase corrosion identification and region detection. Appl Soft Comput 112008

  50. Wu R, Qin K, Fang Y, Xu Y, Zhang H, Li W, Luo X, Han Z, Liu S, Li Q (2024) Application of CNN in the diagnosis for the invasion depth of gastrointestinal cancer: a systematic review and meta-analysis. Journal of Gastrointestinal Surgery

  51. Yadav SP, Jindal M, Rani P, De Albuquerque VHC, Nascimento CDS, Kumar M (2023) An improved deep learning-based optimal object detection system from images. Multimedia Tools and Applications

  52. Parsaeimehr E, Fartash M, Torkestani JA (2023) Improving Feature Extraction Using a Hybrid of CNN and LSTM for Entity Identification. Neural Process Lett 55(5):5979–5994

    Article  MATH  Google Scholar 

  53. Xu H, Tian Y, Ren H, Liu X (2024) A lightweight channel and time attention enhanced 1D CNN model for environmental sound classification. Expert Syst Appl 249:123768

    Article  Google Scholar 

  54. Szarata A, Ostaszewski P, Mirzahossein H (2023) Simulating the impact of autonomous vehicles (AVs) on intersections traffic conditions using TRANSYT and PTV Vissim. Innovative Infrastructure Solutions 8(6)

  55. Sadid H, Antoniou C (2024) A simulation-based impact assessment of autonomous vehicles in urban networks. IET Intell Trans Syst 18(9):1677–1696

    Article  MATH  Google Scholar 

  56. Sural S, Sahu N, Rajkumar RR (2024) ContextualFusion: Context-based multi-sensor fusion for 3D object detection in adverse operating conditions. In 2024 IEEE intelligent vehicles symposium (IV) (pp 1534-1541) IEEE

  57. Verstraete T, Muhammad N (2024) Pedestrian collision avoidance in autonomous vehicles: a review. Computers 13(3):78

    Article  MATH  Google Scholar 

  58. Wang H, Liu C, Cai Y, Chen L, Li Y (2024) YOLOv8-QSD: an improved small object detection algorithm for autonomous vehicles based on YOLOv8. IEEE Transactions on Instrumentation and Measurement, 1

  59. Berta R, Lazzaroni L, Capello A, Cossu M, Forneris L, Pighetti A, Bellotti F (2024) Development of deep-learning-based autonomous agents for low-speed maneuvering in unity. J Intell Connected Veh 7(3):229–244

    Article  Google Scholar 

  60. Qi W, Qin W, Yun Z (2024) Closed-loop state of charge estimation of Li-ion batteries based on deep learning and robust adaptive Kalman filter. Energy 307:132805. https://doi.org/10.1016/j.energy.2024.132805

    Article  Google Scholar 

  61. Fidon L, Aertsen M, Kofler F, Bink A, David AL, Deprest T, Emam D, Guffens F, Jakab A, Kasprian G, Kienast P, Melbourne A, Menze B, Mufti N, Pogledic I, Prayer D, Stuempflen M, Van Elslander E, Ourselin, S.,... Vercauteren T (2024) A Dempster-Shafer Approach to Trustworthy AI With Application to Fetal Brain MRI Segmentation. IEEE Transactions on Pattern Analysis and Machine Intelligence 46(5):3784–3795

  62. Yildiz G, Ulu A, Dızdaroğlu B, Yildiz D (2023) Hybrid image improving and CNN (HIICNN) stacking ensemble method for traffic sign recognition. IEEE Access 11:69536–69552

    Article  MATH  Google Scholar 

  63. Gulzar Y (2023) Fruit image classification model based on MobileNetV2 with deep transfer learning technique. Sustainability 15(3):1906

    Article  MATH  Google Scholar 

  64. Maldonado S, Vairetti C, Jara K, Carrasco M, López J (2023) OWAdapt: An adaptive loss function for deep learning using OWA operators. Knowl-Based Syst 280:111022. https://doi.org/10.1016/j.knosys.2023.111022

    Article  MATH  Google Scholar 

  65. Li S, Yoon H-S (2023) Sensor fusion-based vehicle detection and tracking using a single camera and radar at a traffic intersection. Sensors 23:4888

    Article  MATH  Google Scholar 

  66. Wei P, Zeng Y, Ouyang W, Zhou J (2023) Multi-sensor environmental perception and adaptive cruise control of intelligent vehicles using kalman filter. IEEE Transactions on Intelligent Transportation Systems, pp 1–10

  67. Zhang H, Yang Z, Xiong H, Zhu T, Long Z, Wu W (2023) Transformer Aided Adaptive Extended Kalman Filter for Autonomous Vehicle Mass Estimation. Processes 11:887

    Article  MATH  Google Scholar 

  68. Park G (2024) Optimal vehicle position estimation using adaptive unscented Kalman filter based on sensor fusion. Mechatronics 99:103144

    Article  MATH  Google Scholar 

  69. Choi JD, Kim MY (2022) A sensor fusion system with thermal infrared camera and LiDAR for autonomous vehicles and deep learning based object detection. ICT Express 9(2):222–227

    Article  MATH  Google Scholar 

  70. Hasanujjaman M, Chowdhury MZ, Jang YM (2023) Sensor Fusion in Autonomous Vehicle with Traffic Surveillance Camera System: Detection, Localization, and AI Networking. Sensors 23(6):3335

    Article  MATH  Google Scholar 

  71. Tang Q, Liang J, Zhu F (2023) A comparative review on multi-modal sensors fusion based on deep learning. Signal Process 213:109165

    Article  MATH  Google Scholar 

  72. Zhou X, Yang Q, Liu Q, Liang W, Wang K, Liu Z, Ma J, Jin Q (2023) Spatial-Temporal Federated Transfer Learning with multi-sensor data fusion for cooperative positioning. Inf Fusion 105:102182

    Article  MATH  Google Scholar 

  73. Farid A, Hussain F, Khan K, Shahzad M, Khan U, Mahmood Z (2023) A Fast and Accurate Real-Time Vehicle Detection Method Using Deep Learning for Unconstrained Environments. Applied Sciences 13(5):3059. https://doi.org/10.3390/app13053059

    Article  MATH  Google Scholar 

  74. Zhang H, Yang G, Yu H, Zheng Z (2023) Kalman Filter-Based CNN-BiLSTM-ATT Model for Traffic Flow Prediction. Computers, Materials and Continua/Computers, Materials and Continua (Print) 76(1):1047–1063

  75. Cavdar T, Ebrahimpour N, Kakız MT, and Günay, FB (2022) Decision-making for the anomalies in IIoTs based on 1D convolutional neural networks and Dempster–Shafer theory (DS-1DCNN) The Journal of Supercomputing 79(2):1683–1704

  76. Hassani S, Dackermann U, Mousavi M, Li J (2023) A systematic review of data fusion techniques for optimized structural health monitoring. Inf Fusion 103:102136

    Article  Google Scholar 

  77. Mohammed A, Kora R (2023) A comprehensive review on ensemble deep learning: Opportunities and challenges. J King Saud University - Comput Inf Sci 35(2):757–774

    MATH  Google Scholar 

  78. Kurdthongmee W, Suwannarat K, Wattanapanich C (2023) A framework to estimate the key point within an object based on a deep learning object detection. HighTech Innov J 4(1):106–121

    Article  Google Scholar 

  79. Sayeed MS, Abdulrahim H, Razak SFA, Bukar UA, Yogarayan S (2023) IoT Raspberry Pi based smart parking system with weighted K-Nearest Neighbours Approach. Civil Eng J 9(8):1991–2011

    Article  Google Scholar 

  80. Razak SFA, Fang KY, Kamis NH, Amin AHM, Yogarayan S (2023) Simulation of Vehicular Bots-Based DDoS Attacks in Connected Vehicles Networks. HighTech Innov J 4(4):854–869

  81. German Traffic Sign benchmarks (n.d.) https://benchmark.ini.rub.de/

  82. Road sign detection (2020) Kaggle. https://www.kaggle.com/datasets/andrewmvd/road-sign-detection

  83. Hosseini SH, Ghaderi F, Moshiri B, Norouzi M (2023) Road sign classification using transfer learning and pre-trained CNN models. In: Communications in computer and information science (pp. 39–52)

Download references

Author information

Authors and Affiliations

  1. School of Electrical Engineering, Iran University of Science and Technology, Tehran, Iran

    Mojtaba Norouzi

  2. School of Electrical and Computer Engineering, Tarbiat Modares University, Tehran, Iran

    Seyed Hossein Hosseini

  3. Physics Department, College of Science, Northeastern University, Boston, MA, USA

    Mohammad Khoshnevisan

  4. Faculty of Engineering and Natural Sciences, Department of Computer Engineering, Antalya Bilim University, Antalya, Turkey

    Mohammad Khoshnevisan

  5. School of Electrical and Computer Engineering, College of Engineering, University of Tehran, Tehran, Iran

    Behzad Moshiri

  6. Department of Electrical and Computer Engineering, University of Waterloo, Waterloo, ON, Canada

    Behzad Moshiri

Authors
  1. Mojtaba Norouzi
    View author publications

    You can also search for this author inPubMed Google Scholar

  2. Seyed Hossein Hosseini
    View author publications

    You can also search for this author inPubMed Google Scholar

  3. Mohammad Khoshnevisan
    View author publications

    You can also search for this author inPubMed Google Scholar

  4. Behzad Moshiri
    View author publications

    You can also search for this author inPubMed Google Scholar

Contributions

Mojtaba Norouzi was responsible for the conceptualization, methodology, investigation, data curation, and writing and editing of the manuscript. He also provided the necessary resources for the research. Seyed Hossein Hosseini contributed to the investigation and data curation. Mohammad Khoshnevisan assisted in the methodology and participated in the review and editing of the manuscript. Behzad Moshiri contributed to the conceptualization, investigation, and formal analysis of the study.

Corresponding author

Correspondence to Behzad Moshiri.

Ethics declarations

Declaration of AI-assisted technologies in the writing process

During the preparation stages of our work, the authors used ChatGPT to improve the readability and reduce spelling errors. Despite using this tool, the authors fully reviewed and edited the content and take full responsibility for the originality content of the publication.

Conflict of Interest

The authors declare that there are no competing financial interests or personal relationships that could have inappropriately biased the work reported in this paper.

Ethical and Informed Consent for Data Used

The data utilized in this study were either generated or obtained from publicly accessible databases that do not include any personally identifiable information. Consequently, informed consent was not required. All data handling procedures are aligned with ethical research guidelines.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Norouzi, M., Hosseini, S.H., Khoshnevisan, M. et al. Applications of pre-trained CNN models and data fusion techniques in Unity3D for connected vehicles. Appl Intell 55, 390 (2025). https://doi.org/10.1007/s10489-024-06213-3

Download citation

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s10489-024-06213-3

Keywords