Skip to main content

Advertisement

Log in

Enhancing vehicle fault diagnosis through multi-view sound analysis: integrating scalograms and spectrograms in a deep learning framework

  • Original Paper
  • Published:
Signal, Image and Video Processing Aims and scope Submit manuscript

Abstract

This study presents a comprehensive framework for vehicle fault diagnosis using engine sound signals, leveraging deep learning models and a multi-view approach. Traditional methods for vehicle fault diagnosis often rely on the expertise of mechanics or diagnostic tools, which can be costly, time-consuming, and may not always provide accurate results. To address these limitations, we propose CarFaultNet, a multi-view model that processes both scalograms and spectrograms simultaneously to capture complementary information from these time-frequency representations. Our approach incorporates transfer learning with pretrained convolutional neural networks, including AlexNet, GoogLeNet, ShuffleNet, SqueezeNet, and MobileNet v2, as well as CarFaultNet, which combines two MobileNet networks. The results demonstrate that CarFaultNet outperforms traditional machine learning methods and single-view deep learning models, achieving a precision of 95.32%, recall of 94.83%, F1-score of 94.99%, and accuracy of 95.00%. Class activation mapping visualizations provide valuable insights into the model’s decision-making process, highlighting the regions of the input images that are most influential for the classification of different vehicle faults. By leveraging a large, diverse dataset encompassing various vehicle models and real-world operating conditions, our approach addresses the drawbacks of previous studies and demonstrates the potential of deep learning for practical and effective vehicle fault diagnosis.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11

Similar content being viewed by others

Explore related subjects

Discover the latest articles, news and stories from top researchers in related subjects.

Data availability

No datasets were generated or analysed during the current study.

References

  1. Edmonds, E., American Automobile Association: One-in-Three U.S. Drivers Cannot Pay for an Unexpected Car Repair Bill, [Online]. (2017). Available: https://newsroom.aaa.com/2017/04/one-three-u-s-drivers-cannot-pay-unexpected-car-repair-bill. [Accessed 28 04 2024]

  2. Karaman, E., Rende, H., Akşahin, M.F.: Recognition of vehicles from their engine sound. Mühendis Ve Makina. 60(695), 148–164 (2019)

    MATH  Google Scholar 

  3. Xu, L., Wang, T., Xie, J., Yang, J., Gao, G.: A Mechanism-Based Automatic Fault Diagnosis Method for Gearboxes, Sensors, vol. 22, no. 23, p. 9150, (2022)

  4. Lei, Y., Jia, F., Lin, J., Xing, S., Ding, S.X.: An Intelligent Fault diagnosis method using unsupervised feature learning towards mechanical Big Data. IEEE Trans. Industr. Electron. 63(5), 3137–3147 (2016)

    Article  MATH  Google Scholar 

  5. Liu, Y., Zhang, J., Ma, L.: A fault diagnosis approach for diesel engines based on self-adaptive WVD, improved FCBF and PECOC-RVM. Neurocomputing. 117, 600–611 (2016)

    Article  MATH  Google Scholar 

  6. Wang, Y., Ma, Q., Zhu, Q., Liu, X., Zhao, L.: An intelligent approach for engine fault diagnosis based on Hilbert–Huang transform and support vector machine. Appl. Acoust. 75, 1–9 (2014)

    Article  MATH  Google Scholar 

  7. Feng, Z., Zhang, D., Zuo, M.: Planetary Gearbox Fault diagnosis via joint amplitude and frequency demodulation analysis based on variational mode decomposition. Appl. Sci., 7(8), 775 (2017)

  8. López-Torres, C., Riba, J.-R., Garcia, A., Romeral, L.: Detection of eccentricity faults in five-phase Ferrite-PM assisted synchronous reluctance machines. Appl. Sci. 7(6), 565 (2017)

  9. Qu, Y., He, M., Deutsch, J., He, D.: Detection of Pitting in Gears using a deep sparse autoencoder. Appl. Sci. 7(5), 515 (2017)

  10. Gao, C., Xue, W., Ren, Y., Zhou, Y.: Numerical control machine tool fault diagnosis using hybrid stationary subspace analysis and least squares support vector machine with a single sensor. Appl. Sci. 7(4), 346 (2017)

  11. Lupea, I., Lupea, M., Coroian, A.: Helical Gearbox Defect Detection with Machine Learning Using Regular Mesh Components and Sidebands, Sensors, vol. 24, no. 11, p. 3337, (2024)

  12. Moshrefi, A., Tawfik, H.H., Elsayed, M.Y., Nabki, F.: Industrial Fault Detection Employing Meta Ensemble Model Based on Contact Sensor Ultrasonic Signal, Sensors, vol. 24, no. 7, p. 2297, (2024)

  13. Qu, N., Wei, W., Hu, C.: Series Arc Fault Detection Based on Multimodal Feature Fusion, Sensors, vol. 23, no. 17, p. 7646, (2023)

  14. de las Morenas, J., Moya-Fernández, F., López-Gómez, J.A.: The Edge Application of Machine Learning Techniques for Fault Diagnosis in Electrical Machines, Sensors, vol. 23, no. 5, p. 2649, (2023)

  15. Yang, X., Yang, J., Jin, Y., Liu, Z.: A New Method for Bearing Fault Diagnosis across Machines Based on Envelope Spectrum and Conditional Metric Learning, Sensors, vol. 24, no. 9, p. 2674, (2024)

  16. Madain, M., Al-Mosaiden, A., Al-khassaweneh, M.: Fault diagnosis in vehicle engines using sound recognition techniques, in IEEE International Conference on Electro/Information Technology, Normal, IL, USA, 2010. (2010)

  17. Mofleh, A., Shmroukh, A., Ghazaly, N.: Fault Detection and classification of Spark Ignition Engine based on acoustic signals and Artificial neural network. Int. J. Mech. Prod. Eng. Res. Dev. 10(3), 5571–5578 (2020)

    MATH  Google Scholar 

  18. Navea, R., Sybingco, E.: Design and Implementation of an Acoustic-Based Car Engine Fault Diagnostic System in the Android Platform, in International Research Conference in Higher Education, Manila, Philippines, (2013)

  19. Siegel, J., Kumar, S., Ehrenberg, I., Sarma, S.: Engine Misfire Detection with Pervasive Mobile Audio, in Joint European Conference on Machine Learning and Knowledge Discovery in Databases, Turin, Italy, (2016)

  20. Wu, J.-D., Liu, C.-H.: Investigation of engine fault diagnosis using discrete wavelet transform and neural network. Expert Syst. Appl. 35(3), 1200–1213 (2008)

    Article  MATH  Google Scholar 

  21. Figlus, T., Liščák, Š., Wilk, A., Łazarz, B.: Condition monitoring of engine timing system by using wavelet packet decomposition of a acoustic signal. J. Mech. Sci. Technol. 28(5), 1663–1671 (2014)

    Article  MATH  Google Scholar 

  22. Yılmaz, G., Mete, N.F., Umugabekazi, U., Aydemir, Ç.: Dalgacık Dönüşümü ve Özbağlanım Model Parametreleri Öznitelikleri Ile Otomobil Motor Seslerinden Arıza Tespiti. J. Investigations Eng. Technol. 3(2), 48–54 (2020)

    Google Scholar 

  23. Akbalık, F., Yıldız, A., Ertuğrul, Ö.F., Zan, H.: Engine fault detection by sound analysis and machine learning. Appl. Sci. 14(15), 6532 (2024)

  24. Shrestha, A., Mahmood, A.: Review of Deep Learning algorithms and architectures. IEEE Access. 7, 53040–53065 (2019)

    Article  MATH  Google Scholar 

  25. Wiatowski, T., Bolcskei, H.: A Mathematical Theory of Deep Convolutional neural networks for feature extraction. IEEE Trans. Inf. Theory. 64(3), 1845–1866 (2018)

    Article  MathSciNet  MATH  Google Scholar 

  26. Metin, S.Z., Uyulan, Ç., Farhad, S., Ergüzel, T.T., Türk, Ö., Metin, B., Çerezci, Ö., Tarhan, N.: Deep learning-based artificial Intelligence can differentiate treatment-resistant and responsive depression cases with high accuracy. Clin. EEG Neurosci. 0(0), (2024)

  27. Cao, C., Liu, F., Tan, H., Song, D., Shu, W., Li, W., Zhou, Y., Bo, X., Xie, Z.: Deep learning and its applications in Biomedicine. Genom. Proteom. Bioinform. 16(1), 17–32 (2018)

    Article  MATH  Google Scholar 

  28. Long, M., Cao, Y., Cao, Z., Wang, J. and M., Jordan: Transferable representation learning with Deep Adaptation Networks. IEEE Trans. Pattern Anal. Mach. Intell. 41(12), 3071–3085 (2019)

    Article  MATH  Google Scholar 

  29. Ding, Z., Nasrabadi, N.M., Fu, Y.: Semi-supervised deep domain adaptation via coupled neural networks. IEEE Trans. Image Process. 27(11), 5214–5224 (2018)

    Article  MathSciNet  MATH  Google Scholar 

  30. Mirzaei, A., Pourahmadi, V., Soltani, M., Sheikhzadeh, H.: Deep feature selection using a teacher-student network. Neurocomputing. 383, 396–408 (2020)

    Article  MATH  Google Scholar 

  31. Mian, Z., Deng, X., Dong, X., Tian, Y., Cao, T., Chen, K., Jaber, T.A.: A literature review of fault diagnosis based on ensemble learning. Eng. Appl. Artif. Intell. 127, 107357 (2024)

    Article  Google Scholar 

  32. Liu, D., Wang, M., Chen, M.: Feature ensemble net: A Deep Framework for detecting incipient faults in dynamical processes. IEEE Trans. Industr. Inf. 18(12), 8618–8628 (2022)

    Article  MATH  Google Scholar 

  33. Ruan, D., Chen, X., Gühmann, C., Yan, J.: Improvement of Generative Adversarial Network and Its Application in Bearing Fault Diagnosis: A Review, Lubricants, vol. 11, no. 2, p. 74, (2023)

  34. Chen, M., Shao, H., Dou, H., Li, W., Liu, B.: Data Augmentation and Intelligent Fault diagnosis of Planetary Gearbox using ILoFGAN under extremely limited samples. IEEE Trans. Reliab. 72(3), 1029–1037 (2023)

    Article  MATH  Google Scholar 

  35. Shen, S., Sadoughi, M., Chen, X., Hong, M., Hu, C.: A deep learning method for online capacity estimation of lithium-ion batteries. J. Energy Storage. 25, 100817 (2019)

    Article  Google Scholar 

  36. Wu, Y., Li, W.: Online Capacity Estimation for Lithium-Ion batteries based on semi-supervised convolutional neural network. World Electr. Veh. J. 12(4), 256 (2021)

    Article  MATH  Google Scholar 

  37. Li, X., Yu, S., Lei, Y., Li, N., Yang, B.: Dynamic vision-based Machinery Fault diagnosis with Cross-modality Feature Alignment. IEEE/CAA J. Automatica Sinica. 11(10), 2068–2081 (2024)

    Article  MATH  Google Scholar 

  38. Li, X., Zhang, W., Li, X., Hao, H.: Partial domain adaptation in remaining useful life Prediction with Incomplete Target Data. IEEE/ASME Trans. Mechatron. 29(3), 1903–1913 (2024)

    Article  MathSciNet  MATH  Google Scholar 

  39. Qin, C., Jin, Y., Tao, J., Xiao, D., Yu, H., Liu, C., Shi, G., Lei, J., Liu, C.: DTCNNMI: A deep twin convolutional neural networks with multi-domain inputs for strongly noisy diesel engine misfire detection. Measurement. 180, 109548 (2021)

    Article  MATH  Google Scholar 

  40. Terwilliger, A.M., Siegel, J.E.: Improving Misfire Fault Diagnosis with Cascading Architectures via Acoustic Vehicle Characterization, Sensors, vol. 22, no. 20, p. 7736, (2022)

  41. Firmino, J.L., Neto, J.M., Oliveira, A.G., Silva, J.C., Mishina, K.V., Rodrigues, M.C.: Misfire detection of an internal combustion engine based on vibration and acoustic analysis. J. Brazilian Soc. Mech. Sci. Eng. 43(7), 336 (2021)

    Article  MATH  Google Scholar 

  42. Müller, M.: Fourier Analysis of Signals. In: Fundamentals of Music Processing, pp. 39–114. Springer International Publishing, Cham (2015)

    Chapter  MATH  Google Scholar 

  43. Yildiz, A., Zan, H., Said, S.: Classification and analysis of epileptic EEG recordings using convolutional neural network and class activation mapping. Biomed. Signal Process. Control. 68, 102720 (2021)

    Article  MATH  Google Scholar 

  44. Daubechies, I.: The wavelet transform, time-frequency localization and signal analysis. IEEE Trans. Inf. Theory. 36(5), 961–1005 (1990)

    Article  MathSciNet  MATH  Google Scholar 

  45. Shao, F., Shen, Z.: How can artificial neural networks approximate the brain? Front. Psychol., 13, (2023)

  46. Chen, L., Li, S., Bai, Q., Yang, J., Jiang, S., Miao, Y.: Review of image classification algorithms based on convolutional neural networks. Remote Sens. 13(22), 4712 (2021)

    Article  MATH  Google Scholar 

  47. Yu, W., Yang, K., Yao, H., Sun, X., Xu, P.: Exploiting the complementary strengths of multi-layer CNN features for image retrieval. Neurocomputing. 237, 235–241 (2017)

    Article  MATH  Google Scholar 

  48. Yamashita, R., Nishio, M., Do, R.K.G., Togashi, K.: Convolutional neural networks: An overview and application in radiology. Insights into Imaging. 9(4), 611–629 (2018)

    Article  Google Scholar 

  49. Hung, J.C., Chang, J.-W.: Multi-level transfer learning for improving the performance of deep neural networks: Theory and practice from the tasks of facial emotion recognition and named entity recognition. Appl. Soft Comput. 109, 107491 (2021)

    Article  MATH  Google Scholar 

  50. Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., Berg, A.C., Fei-Fei, L.: ImageNet large scale visual recognition challenge. Int. J. Comput. Vision. 115(3), 211–252 (2015)

    Article  MathSciNet  Google Scholar 

  51. He, X., Chen, Y.: Transferring CNN ensemble for hyperspectral image classification. IEEE Geosci. Remote Sens. Lett. 18(5), 876–880 (2021)

    Article  MATH  Google Scholar 

  52. Ding, Y., Ding, P., Zhao, X., Cao, Y., Jia, M.: Transfer learning for remaining useful life prediction across operating conditions based on Multisource Domain Adaptation. IEEE/ASME Trans. Mechatron. 27(5), 4143–4152 (2022)

    Article  MATH  Google Scholar 

  53. Kimura, N., Yoshinaga, I., Sekijima, K., Azechi, I., Baba, D.: Convolutional Neural Network Coupled with a Transfer-Learning Approach for Time-Series Flood Predictions, Water, vol. 12, no. 1, p. 96, (2019)

  54. Pan, S.J., Yang, Q.: A survey on transfer learning. IEEE Trans. Knowl. Data Eng. 22(10), 1345–1359 (2010)

    Article  MATH  Google Scholar 

  55. Shao, S., McAleer, S., Yan, R., Baldi, P.: Highly Accurate Machine Fault diagnosis using deep transfer learning. IEEE Trans. Industr. Inf. 15(4), 2446–2455 (2019)

    Article  Google Scholar 

  56. Ancel, J., Mahmoud, W., Denis, M.: Gunshot detection from audio excerpts of urban sounds using transfers learning, The Journal of the Acoustical Society of America, vol. 153, no. 3_supplement, p. A45, (2023)

  57. Krizhevsky, A., Sutskever, I., Hinton, G.E.: Imagenet classification with deep convolutional neural networks, in Advances in Neural Information Processing Systems 25 (NIPS Lake Tahoe, NV, United States, 2012. (2012)

  58. Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., Rabinovich, A.: Going deeper with convolutions, in IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Boston, MA, USA, 2015. (2015)

  59. Zhang, X., Zhou, X., Lin, M., Sun, J.: ShuffleNet: An Extremely Efficient Convolutional Neural Network for Mobile Devices, in IEEE/CVF Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA, 2018. (2018)

  60. Iandola, F.N., Han, S., Moskewicz, M.W., Ashraf, K., Dally, W.J., Keutzer, K.: SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and < 0.5 MB model size, arXiv preprint arXiv:1602.07360, (2016)

  61. Dong, K., Zhou, C., Ruan, Y., Li, Y., MobileNetV2 Model for Image Classification, in: 2nd International Conference on Information Technology and Computer Application (ITCA), Guangzhou, China, 2020. (2020)

  62. Kingma, D.P., Ba, J.: Adam: A Method for Stochastic Optimization, arXiv preprint arXiv:1412.6980, (2014)

  63. Kemalkar, A.K., Bairagi, V.K.: Engine fault diagnosis using sound analysis, in International Conference on Automatic Control and Dynamic Optimization Techniques (ICACDOT), Pune, India, 2016. (2016)

Download references

Acknowledgements

The numerical calculations reported in this paper were fully performed at TUBITAK ULAKBIM, High Performance and Grid Computing Center (TRUBA resources).

Funding

This research received no external funding. The numerical calculations reported in this paper were fully performed at TUBITAK ULAKBIM, High Performance and Grid Computing Center (TRUBA resources).

Author information

Authors and Affiliations

Authors

Contributions

Methodology, F.A. and H.Z.; Software, H.Z.; Data curation, F.A.; Writing—review & editing, A.Y. and Ö.F.E.; Supervision, A.Y. and Ö.F.E. All authors have read and agreed to the published version of the manuscript.

Corresponding author

Correspondence to Ferit Akbalik.

Ethics declarations

Institutional review board

Not applicable.

Informed consent

Not applicable.

Competing interests

The authors declare no competing interests.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Akbalik, F., Yildiz, A., Ertuğrul, Ö.F. et al. Enhancing vehicle fault diagnosis through multi-view sound analysis: integrating scalograms and spectrograms in a deep learning framework. SIViP 19, 182 (2025). https://doi.org/10.1007/s11760-024-03746-5

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s11760-024-03746-5

Keywords

Navigation