Abstract
This study presents a comprehensive framework for vehicle fault diagnosis using engine sound signals, leveraging deep learning models and a multi-view approach. Traditional methods for vehicle fault diagnosis often rely on the expertise of mechanics or diagnostic tools, which can be costly, time-consuming, and may not always provide accurate results. To address these limitations, we propose CarFaultNet, a multi-view model that processes both scalograms and spectrograms simultaneously to capture complementary information from these time-frequency representations. Our approach incorporates transfer learning with pretrained convolutional neural networks, including AlexNet, GoogLeNet, ShuffleNet, SqueezeNet, and MobileNet v2, as well as CarFaultNet, which combines two MobileNet networks. The results demonstrate that CarFaultNet outperforms traditional machine learning methods and single-view deep learning models, achieving a precision of 95.32%, recall of 94.83%, F1-score of 94.99%, and accuracy of 95.00%. Class activation mapping visualizations provide valuable insights into the model’s decision-making process, highlighting the regions of the input images that are most influential for the classification of different vehicle faults. By leveraging a large, diverse dataset encompassing various vehicle models and real-world operating conditions, our approach addresses the drawbacks of previous studies and demonstrates the potential of deep learning for practical and effective vehicle fault diagnosis.
![](http://media.springernature.com/m312/springer-static/image/art%3A10.1007%2Fs11760-024-03746-5/MediaObjects/11760_2024_3746_Fig1_HTML.png)
![](http://media.springernature.com/m312/springer-static/image/art%3A10.1007%2Fs11760-024-03746-5/MediaObjects/11760_2024_3746_Fig2_HTML.png)
![](http://media.springernature.com/m312/springer-static/image/art%3A10.1007%2Fs11760-024-03746-5/MediaObjects/11760_2024_3746_Fig3_HTML.png)
![](http://media.springernature.com/m312/springer-static/image/art%3A10.1007%2Fs11760-024-03746-5/MediaObjects/11760_2024_3746_Fig4_HTML.png)
![](http://media.springernature.com/m312/springer-static/image/art%3A10.1007%2Fs11760-024-03746-5/MediaObjects/11760_2024_3746_Fig5_HTML.png)
![](http://media.springernature.com/m312/springer-static/image/art%3A10.1007%2Fs11760-024-03746-5/MediaObjects/11760_2024_3746_Fig6_HTML.png)
![](http://media.springernature.com/m312/springer-static/image/art%3A10.1007%2Fs11760-024-03746-5/MediaObjects/11760_2024_3746_Fig7_HTML.png)
![](http://media.springernature.com/m312/springer-static/image/art%3A10.1007%2Fs11760-024-03746-5/MediaObjects/11760_2024_3746_Fig8_HTML.png)
![](http://media.springernature.com/m312/springer-static/image/art%3A10.1007%2Fs11760-024-03746-5/MediaObjects/11760_2024_3746_Fig9_HTML.png)
![](http://media.springernature.com/m312/springer-static/image/art%3A10.1007%2Fs11760-024-03746-5/MediaObjects/11760_2024_3746_Fig10_HTML.png)
![](http://media.springernature.com/m312/springer-static/image/art%3A10.1007%2Fs11760-024-03746-5/MediaObjects/11760_2024_3746_Fig11_HTML.png)
Similar content being viewed by others
Explore related subjects
Discover the latest articles, news and stories from top researchers in related subjects.Data availability
No datasets were generated or analysed during the current study.
References
Edmonds, E., American Automobile Association: One-in-Three U.S. Drivers Cannot Pay for an Unexpected Car Repair Bill, [Online]. (2017). Available: https://newsroom.aaa.com/2017/04/one-three-u-s-drivers-cannot-pay-unexpected-car-repair-bill. [Accessed 28 04 2024]
Karaman, E., Rende, H., Akşahin, M.F.: Recognition of vehicles from their engine sound. Mühendis Ve Makina. 60(695), 148–164 (2019)
Xu, L., Wang, T., Xie, J., Yang, J., Gao, G.: A Mechanism-Based Automatic Fault Diagnosis Method for Gearboxes, Sensors, vol. 22, no. 23, p. 9150, (2022)
Lei, Y., Jia, F., Lin, J., Xing, S., Ding, S.X.: An Intelligent Fault diagnosis method using unsupervised feature learning towards mechanical Big Data. IEEE Trans. Industr. Electron. 63(5), 3137–3147 (2016)
Liu, Y., Zhang, J., Ma, L.: A fault diagnosis approach for diesel engines based on self-adaptive WVD, improved FCBF and PECOC-RVM. Neurocomputing. 117, 600–611 (2016)
Wang, Y., Ma, Q., Zhu, Q., Liu, X., Zhao, L.: An intelligent approach for engine fault diagnosis based on Hilbert–Huang transform and support vector machine. Appl. Acoust. 75, 1–9 (2014)
Feng, Z., Zhang, D., Zuo, M.: Planetary Gearbox Fault diagnosis via joint amplitude and frequency demodulation analysis based on variational mode decomposition. Appl. Sci., 7(8), 775 (2017)
López-Torres, C., Riba, J.-R., Garcia, A., Romeral, L.: Detection of eccentricity faults in five-phase Ferrite-PM assisted synchronous reluctance machines. Appl. Sci. 7(6), 565 (2017)
Qu, Y., He, M., Deutsch, J., He, D.: Detection of Pitting in Gears using a deep sparse autoencoder. Appl. Sci. 7(5), 515 (2017)
Gao, C., Xue, W., Ren, Y., Zhou, Y.: Numerical control machine tool fault diagnosis using hybrid stationary subspace analysis and least squares support vector machine with a single sensor. Appl. Sci. 7(4), 346 (2017)
Lupea, I., Lupea, M., Coroian, A.: Helical Gearbox Defect Detection with Machine Learning Using Regular Mesh Components and Sidebands, Sensors, vol. 24, no. 11, p. 3337, (2024)
Moshrefi, A., Tawfik, H.H., Elsayed, M.Y., Nabki, F.: Industrial Fault Detection Employing Meta Ensemble Model Based on Contact Sensor Ultrasonic Signal, Sensors, vol. 24, no. 7, p. 2297, (2024)
Qu, N., Wei, W., Hu, C.: Series Arc Fault Detection Based on Multimodal Feature Fusion, Sensors, vol. 23, no. 17, p. 7646, (2023)
de las Morenas, J., Moya-Fernández, F., López-Gómez, J.A.: The Edge Application of Machine Learning Techniques for Fault Diagnosis in Electrical Machines, Sensors, vol. 23, no. 5, p. 2649, (2023)
Yang, X., Yang, J., Jin, Y., Liu, Z.: A New Method for Bearing Fault Diagnosis across Machines Based on Envelope Spectrum and Conditional Metric Learning, Sensors, vol. 24, no. 9, p. 2674, (2024)
Madain, M., Al-Mosaiden, A., Al-khassaweneh, M.: Fault diagnosis in vehicle engines using sound recognition techniques, in IEEE International Conference on Electro/Information Technology, Normal, IL, USA, 2010. (2010)
Mofleh, A., Shmroukh, A., Ghazaly, N.: Fault Detection and classification of Spark Ignition Engine based on acoustic signals and Artificial neural network. Int. J. Mech. Prod. Eng. Res. Dev. 10(3), 5571–5578 (2020)
Navea, R., Sybingco, E.: Design and Implementation of an Acoustic-Based Car Engine Fault Diagnostic System in the Android Platform, in International Research Conference in Higher Education, Manila, Philippines, (2013)
Siegel, J., Kumar, S., Ehrenberg, I., Sarma, S.: Engine Misfire Detection with Pervasive Mobile Audio, in Joint European Conference on Machine Learning and Knowledge Discovery in Databases, Turin, Italy, (2016)
Wu, J.-D., Liu, C.-H.: Investigation of engine fault diagnosis using discrete wavelet transform and neural network. Expert Syst. Appl. 35(3), 1200–1213 (2008)
Figlus, T., Liščák, Š., Wilk, A., Łazarz, B.: Condition monitoring of engine timing system by using wavelet packet decomposition of a acoustic signal. J. Mech. Sci. Technol. 28(5), 1663–1671 (2014)
Yılmaz, G., Mete, N.F., Umugabekazi, U., Aydemir, Ç.: Dalgacık Dönüşümü ve Özbağlanım Model Parametreleri Öznitelikleri Ile Otomobil Motor Seslerinden Arıza Tespiti. J. Investigations Eng. Technol. 3(2), 48–54 (2020)
Akbalık, F., Yıldız, A., Ertuğrul, Ö.F., Zan, H.: Engine fault detection by sound analysis and machine learning. Appl. Sci. 14(15), 6532 (2024)
Shrestha, A., Mahmood, A.: Review of Deep Learning algorithms and architectures. IEEE Access. 7, 53040–53065 (2019)
Wiatowski, T., Bolcskei, H.: A Mathematical Theory of Deep Convolutional neural networks for feature extraction. IEEE Trans. Inf. Theory. 64(3), 1845–1866 (2018)
Metin, S.Z., Uyulan, Ç., Farhad, S., Ergüzel, T.T., Türk, Ö., Metin, B., Çerezci, Ö., Tarhan, N.: Deep learning-based artificial Intelligence can differentiate treatment-resistant and responsive depression cases with high accuracy. Clin. EEG Neurosci. 0(0), (2024)
Cao, C., Liu, F., Tan, H., Song, D., Shu, W., Li, W., Zhou, Y., Bo, X., Xie, Z.: Deep learning and its applications in Biomedicine. Genom. Proteom. Bioinform. 16(1), 17–32 (2018)
Long, M., Cao, Y., Cao, Z., Wang, J. and M., Jordan: Transferable representation learning with Deep Adaptation Networks. IEEE Trans. Pattern Anal. Mach. Intell. 41(12), 3071–3085 (2019)
Ding, Z., Nasrabadi, N.M., Fu, Y.: Semi-supervised deep domain adaptation via coupled neural networks. IEEE Trans. Image Process. 27(11), 5214–5224 (2018)
Mirzaei, A., Pourahmadi, V., Soltani, M., Sheikhzadeh, H.: Deep feature selection using a teacher-student network. Neurocomputing. 383, 396–408 (2020)
Mian, Z., Deng, X., Dong, X., Tian, Y., Cao, T., Chen, K., Jaber, T.A.: A literature review of fault diagnosis based on ensemble learning. Eng. Appl. Artif. Intell. 127, 107357 (2024)
Liu, D., Wang, M., Chen, M.: Feature ensemble net: A Deep Framework for detecting incipient faults in dynamical processes. IEEE Trans. Industr. Inf. 18(12), 8618–8628 (2022)
Ruan, D., Chen, X., Gühmann, C., Yan, J.: Improvement of Generative Adversarial Network and Its Application in Bearing Fault Diagnosis: A Review, Lubricants, vol. 11, no. 2, p. 74, (2023)
Chen, M., Shao, H., Dou, H., Li, W., Liu, B.: Data Augmentation and Intelligent Fault diagnosis of Planetary Gearbox using ILoFGAN under extremely limited samples. IEEE Trans. Reliab. 72(3), 1029–1037 (2023)
Shen, S., Sadoughi, M., Chen, X., Hong, M., Hu, C.: A deep learning method for online capacity estimation of lithium-ion batteries. J. Energy Storage. 25, 100817 (2019)
Wu, Y., Li, W.: Online Capacity Estimation for Lithium-Ion batteries based on semi-supervised convolutional neural network. World Electr. Veh. J. 12(4), 256 (2021)
Li, X., Yu, S., Lei, Y., Li, N., Yang, B.: Dynamic vision-based Machinery Fault diagnosis with Cross-modality Feature Alignment. IEEE/CAA J. Automatica Sinica. 11(10), 2068–2081 (2024)
Li, X., Zhang, W., Li, X., Hao, H.: Partial domain adaptation in remaining useful life Prediction with Incomplete Target Data. IEEE/ASME Trans. Mechatron. 29(3), 1903–1913 (2024)
Qin, C., Jin, Y., Tao, J., Xiao, D., Yu, H., Liu, C., Shi, G., Lei, J., Liu, C.: DTCNNMI: A deep twin convolutional neural networks with multi-domain inputs for strongly noisy diesel engine misfire detection. Measurement. 180, 109548 (2021)
Terwilliger, A.M., Siegel, J.E.: Improving Misfire Fault Diagnosis with Cascading Architectures via Acoustic Vehicle Characterization, Sensors, vol. 22, no. 20, p. 7736, (2022)
Firmino, J.L., Neto, J.M., Oliveira, A.G., Silva, J.C., Mishina, K.V., Rodrigues, M.C.: Misfire detection of an internal combustion engine based on vibration and acoustic analysis. J. Brazilian Soc. Mech. Sci. Eng. 43(7), 336 (2021)
Müller, M.: Fourier Analysis of Signals. In: Fundamentals of Music Processing, pp. 39–114. Springer International Publishing, Cham (2015)
Yildiz, A., Zan, H., Said, S.: Classification and analysis of epileptic EEG recordings using convolutional neural network and class activation mapping. Biomed. Signal Process. Control. 68, 102720 (2021)
Daubechies, I.: The wavelet transform, time-frequency localization and signal analysis. IEEE Trans. Inf. Theory. 36(5), 961–1005 (1990)
Shao, F., Shen, Z.: How can artificial neural networks approximate the brain? Front. Psychol., 13, (2023)
Chen, L., Li, S., Bai, Q., Yang, J., Jiang, S., Miao, Y.: Review of image classification algorithms based on convolutional neural networks. Remote Sens. 13(22), 4712 (2021)
Yu, W., Yang, K., Yao, H., Sun, X., Xu, P.: Exploiting the complementary strengths of multi-layer CNN features for image retrieval. Neurocomputing. 237, 235–241 (2017)
Yamashita, R., Nishio, M., Do, R.K.G., Togashi, K.: Convolutional neural networks: An overview and application in radiology. Insights into Imaging. 9(4), 611–629 (2018)
Hung, J.C., Chang, J.-W.: Multi-level transfer learning for improving the performance of deep neural networks: Theory and practice from the tasks of facial emotion recognition and named entity recognition. Appl. Soft Comput. 109, 107491 (2021)
Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., Berg, A.C., Fei-Fei, L.: ImageNet large scale visual recognition challenge. Int. J. Comput. Vision. 115(3), 211–252 (2015)
He, X., Chen, Y.: Transferring CNN ensemble for hyperspectral image classification. IEEE Geosci. Remote Sens. Lett. 18(5), 876–880 (2021)
Ding, Y., Ding, P., Zhao, X., Cao, Y., Jia, M.: Transfer learning for remaining useful life prediction across operating conditions based on Multisource Domain Adaptation. IEEE/ASME Trans. Mechatron. 27(5), 4143–4152 (2022)
Kimura, N., Yoshinaga, I., Sekijima, K., Azechi, I., Baba, D.: Convolutional Neural Network Coupled with a Transfer-Learning Approach for Time-Series Flood Predictions, Water, vol. 12, no. 1, p. 96, (2019)
Pan, S.J., Yang, Q.: A survey on transfer learning. IEEE Trans. Knowl. Data Eng. 22(10), 1345–1359 (2010)
Shao, S., McAleer, S., Yan, R., Baldi, P.: Highly Accurate Machine Fault diagnosis using deep transfer learning. IEEE Trans. Industr. Inf. 15(4), 2446–2455 (2019)
Ancel, J., Mahmoud, W., Denis, M.: Gunshot detection from audio excerpts of urban sounds using transfers learning, The Journal of the Acoustical Society of America, vol. 153, no. 3_supplement, p. A45, (2023)
Krizhevsky, A., Sutskever, I., Hinton, G.E.: Imagenet classification with deep convolutional neural networks, in Advances in Neural Information Processing Systems 25 (NIPS Lake Tahoe, NV, United States, 2012. (2012)
Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., Rabinovich, A.: Going deeper with convolutions, in IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Boston, MA, USA, 2015. (2015)
Zhang, X., Zhou, X., Lin, M., Sun, J.: ShuffleNet: An Extremely Efficient Convolutional Neural Network for Mobile Devices, in IEEE/CVF Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA, 2018. (2018)
Iandola, F.N., Han, S., Moskewicz, M.W., Ashraf, K., Dally, W.J., Keutzer, K.: SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and < 0.5 MB model size, arXiv preprint arXiv:1602.07360, (2016)
Dong, K., Zhou, C., Ruan, Y., Li, Y., MobileNetV2 Model for Image Classification, in: 2nd International Conference on Information Technology and Computer Application (ITCA), Guangzhou, China, 2020. (2020)
Kingma, D.P., Ba, J.: Adam: A Method for Stochastic Optimization, arXiv preprint arXiv:1412.6980, (2014)
Kemalkar, A.K., Bairagi, V.K.: Engine fault diagnosis using sound analysis, in International Conference on Automatic Control and Dynamic Optimization Techniques (ICACDOT), Pune, India, 2016. (2016)
Acknowledgements
The numerical calculations reported in this paper were fully performed at TUBITAK ULAKBIM, High Performance and Grid Computing Center (TRUBA resources).
Funding
This research received no external funding. The numerical calculations reported in this paper were fully performed at TUBITAK ULAKBIM, High Performance and Grid Computing Center (TRUBA resources).
Author information
Authors and Affiliations
Contributions
Methodology, F.A. and H.Z.; Software, H.Z.; Data curation, F.A.; Writing—review & editing, A.Y. and Ö.F.E.; Supervision, A.Y. and Ö.F.E. All authors have read and agreed to the published version of the manuscript.
Corresponding author
Ethics declarations
Institutional review board
Not applicable.
Informed consent
Not applicable.
Competing interests
The authors declare no competing interests.
Additional information
Publisher’s note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.
About this article
Cite this article
Akbalik, F., Yildiz, A., Ertuğrul, Ö.F. et al. Enhancing vehicle fault diagnosis through multi-view sound analysis: integrating scalograms and spectrograms in a deep learning framework. SIViP 19, 182 (2025). https://doi.org/10.1007/s11760-024-03746-5
Received:
Revised:
Accepted:
Published:
DOI: https://doi.org/10.1007/s11760-024-03746-5