Abstract
Mechanical system fault diagnosis is essential to save costs and ensure safety. Generally, rotating machinery operates in nonstationary cases, which makes fault features complex and difficult to identify. However, existing fault diagnosis methods have the following limitations. (1) Consider only time or frequency domain features fusion. (2) Extract the fusion features representation only in the Euclidean domain. Based on that, a novel method based on multisensory time-frequency features fusion and graph attention network is proposed for rotating machinery fault diagnosis under the nonstationary case. First, the multi-sensor time series are converted into multi-sensor time-frequency maps by image-to-matrix, matrix concatenation, and matrix-to-image operations. Then, simple linear iterative clustering is applied to make the superpixels in the multi-sensor time-frequency maps into nodes and form graphs based on color and texture features. Finally, the graph attention network with residual connection is applied to distinguish the rotating machinery’s health status. The proposed method is verified using gearbox test rig data and bearing public data, respectively. The experimental results indicate that the proposed method can provide more reliable and accurate fault diagnosis results for rotating machinery than other methods.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
Data availability
The data that support the findings of this study are available from the corresponding author upon request.
References
Achanta, R., Shaji, A., Smith, K., Lucchi, A., Fua, P., & Süsstrunk, S. (2012). SLIC superpixels compared to state-of-the-art superpixel methods. IEEE Transactions on Pattern Analysis and Machine Intelligence, 34(11), 2274–2282. https://doi.org/10.1109/TPAMI.2012.120.
Azamfar, M., Singh, J., Bravo-Imaz, I., & Lee, J. (2020). Multisensor data fusion for gearbox fault diagnosis using 2-D convolutional neural network and motor current signature analysis. Mechanical Systems and Signal Processing, 144, 106861. https://doi.org/10.1016/j.ymssp.2020.106861.
Brody, S., Alon, U., & Yahav, E. (2022). How attentive are graph attention networks. arXiv. https://doi.org/10.48550/arXiv.2105.14491
Cabrera, D., Sancho, F., Sánchez, R. V., Zurita, G., Cerrada, M., Li, C., & Vásquez, R. E. (2015). Fault diagnosis of spur gearbox based on random forest and wavelet packet decomposition. Frontiers of Mechanical Engineering, 10(3), 277–286. https://doi.org/10.1007/s11465-015-0348-8.
Chen, F., Yuan, Z., & Huang, Y. (2020). Multi-source data fusion for aspect-level sentiment classification. Knowledge-Based Systems, 187, 104831. https://doi.org/10.1016/j.knosys.2019.07.002
Chen, X., Zhang, B., & Gao, D. (2021). Bearing fault diagnosis base on multi-scale CNN and LSTM model. Journal of Intelligent Manufacturing, 32(4), 971–987. https://doi.org/10.1007/s10845-020-01600-2.
Cheng, Y., Zhu, H., Hu, K., Wu, J., Shao, X., & Wang, Y. (2019). Reliability prediction of machinery with multiple degradation characteristics using double-Wiener process and Monte Carlo algorithm. Mechanical Systems and Signal Processing, 134, 106333. https://doi.org/10.1016/j.ymssp.2019.106333.
Defferrard, M., Bresson, X., & Vandergheynst, P. (2016). Convolutional neural networks on graphs with fast localized spectral filtering. In advances in neural information processing systems (Vol. 29). Curran Associates, Inc. Retrieved December 20, 2022, From https://proceedings.neurips.cc/paper/2016/hash/04df4d434d481c5bb723be1b6df1ee65-Abstract.html
Diehl, F. (2019). Edge contraction pooling for graph neural networks. arXiv. https://doi.org/10.48550/arXiv.1905.10990
Ding, Y., Jia, M., Miao, Q., & Cao, Y. (2022). A novel time–frequency transformer based on self–attention mechanism and its application in fault diagnosis of rolling bearings. Mechanical Systems and Signal Processing, 168, 108616. https://doi.org/10.1016/j.ymssp.2021.108616.
Guan, Y., Meng, Z., Sun, D., Liu, J., & Fan, F. (2021). 2MNet: Multi-sensor and multi-scale model toward accurate fault diagnosis of rolling bearing. Reliability Engineering & System Safety, 216, 108017. https://doi.org/10.1016/j.ress.2021.108017.
He, K., Zhang, X., Ren, S., Sun, J. (2016). Deep residual learning for image recognition (pp. 770–778). Presented at the proceedings of the IEEE conference on computer vision and, & recognition, Retrieved October 12, 2022, From https://openaccess.thecvf.com/content_cvpr_2016/html/He_Deep_Residual_Learning_CVPR_2016_paper.html
Jiang, L., Li, X., Wu, L., & Li, Y. (2022). Bearing fault diagnosis method based on a multi-head graph attention network. Measurement Science and Technology, 33(7), 075012. https://doi.org/10.1088/1361-6501/ac56f1.
Khaleghi, B., Khamis, A., Karray, F. O., & Razavi, S. N. (2013). Multisensor data fusion: A review of the state-of-the-art. Information Fusion, 14(1), 28–44. https://doi.org/10.1016/j.inffus.2011.08.001.
Li, C., Sanchez, R. V., Zurita, G., Cerrada, M., Cabrera, D., & Vásquez, R. E. (2015). Multimodal deep support vector classification with homologous features and its application to gearbox fault diagnosis. Neurocomputing, 168, 119–127. https://doi.org/10.1016/j.neucom.2015.06.008.
Li, C., Sanchez, R. V., Zurita, G., Cerrada, M., Cabrera, D., & Vásquez, R. E. (2016). Gearbox fault diagnosis based on deep random forest fusion of acoustic and vibratory signals. Mechanical Systems and Signal Processing. https://doi.org/10.1016/j.ymssp.2016.02.007
Li, C., Mo, L., & Yan, R. (2020a). Rolling bearing fault diagnosis based on horizontal visibility graph and graph neural networks. In 2020a international conference on sensing, measurement & data analytics in the era of artificial intelligence (ICSMD) (pp. 275–279). https://doi.org/10.1109/ICSMD50554.2020a.9261687.
Li, S., Wang, H., Song, L., Wang, P., Cui, L., & Lin, T. (2020b). An adaptive data fusion strategy for fault diagnosis based on the convolutional neural network. Measurement, 165, 108122. https://doi.org/10.1016/j.measurement.2020b.108122.
Li, T., Zhao, Z., Sun, C., Yan, R., & Chen, X. (2021a). Multireceptive field graph convolutional networks for machine fault diagnosis. IEEE Transactions on Industrial Electronics, 68(12), 12739–12749. https://doi.org/10.1109/TIE.2020.3040669.
Li, T., Zhao, Z., Sun, C., Yan, R., & Chen, X. (2021b). Hierarchical attention graph convolutional network to fuse multi-sensor signals for remaining useful life prediction. Reliability Engineering & System Safety, 215, 107878. https://doi.org/10.1016/j.ress.2021.107878.
Li, C., Mo, L., & Yan, R. (2021c). Fault diagnosis of rolling bearing based on WHVG and GCN. IEEE transactions on instrumentation and measurement, 70, 1–11. https://doi.org/10.1109/TIM.2021c.3087834.
Li, N., Gebraeel, N., Lei, Y., Fang, X., Cai, X., & Yan, T. (2021d). Remaining useful life prediction based on a multi-sensor data fusion model. Reliability Engineering & System Safety, 208, 107249. https://doi.org/10.1016/j.ress.2020.107249
Li, X., Zhong, X., Shao, H., Han, T., & Shen, C. (2021e). Multi-sensor gearbox fault diagnosis by using feature-fusion covariance matrix and multi-riemannian kernel ridge regression. Reliability Engineering & System Safety, 216, 108018. https://doi.org/10.1016/j.ress.2021e.108018.
Li, T., Zhou, Z., Li, S., Sun, C., Yan, R., & Chen, X. (2022). The emerging graph neural networks for intelligent fault diagnostics and prognostics: A guideline and a benchmark study. Mechanical Systems and Signal Processing, 168, 108653. https://doi.org/10.1016/j.ymssp.2021.108653.
Liu, Y. J., Yu, M., Li, B. J., & He, Y. (2018). Intrinsic Manifold SLIC: A simple and efficient method for computing content-sensitive superpixels. IEEE Transactions on Pattern Analysis and Machine Intelligence, 40(3), 653–666. https://doi.org/10.1109/TPAMI.2017.2686857.
Liu, Y., Yan, X., Zhang, C., & Liu, W. (2019). An ensemble convolutional neural networks for bearing fault diagnosis using multi-sensor data. Sensors (Basel, Switzerland), 19(23), 5300. https://doi.org/10.3390/s19235300.
Liu, C., Tong, J., Zheng, J., Pan, H., & Bao, J. (2022). Rolling bearing fault diagnosis method based on multi-sensor two-stage fusion. Measurement Science and Technology, 33(12), 125105. https://doi.org/10.1088/1361-6501/ac8894.
Liu, S., Jiang, H., Wu, Z., Yi, Z., & Wang, R. (2023). Intelligent fault diagnosis of rotating machinery using a multi-source domain adaptation network with adversarial discrepancy matching. Reliability Engineering & System Safety, 231, 109036. https://doi.org/10.1016/j.ress.2022.109036.
Long, Z., Zhang, X., Zhang, L., Qin, G., Huang, S., Song, D., et al. (2021). Motor fault diagnosis using attention mechanism and improved adaboost driven by multi-sensor information. Measurement, 170, 108718. https://doi.org/10.1016/j.measurement.2020.108718.
Long, J., Zhang, R., Yang, Z., Huang, Y., Liu, Y., & Li, C. (2022). Self-adaptation graph attention network via meta-learning for machinery fault diagnosis with few labeled data. IEEE Transactions on Instrumentation and Measurement, 71, 1–11. https://doi.org/10.1109/TIM.2022.3181894.
Peng, D., Smith, W. A., Randall, R. B., & Peng, Z. (2019). Use of mesh phasing to locate faulty planet gears. Mechanical Systems and Signal Processing, 116, 12–24. https://doi.org/10.1016/j.ymssp.2018.06.035.
Saravanan, N., & Ramachandran, K. I. (2009). A case study on classification of features by fast single-shot multiclass PSVM using Morlet wavelet for fault diagnosis of spur bevel gear box. Expert Systems with Applications, 36(8), 10854–10862. https://doi.org/10.1016/j.eswa.2009.01.053.
Shi, J., Peng, D., Peng, Z., Zhang, Z., Goebel, K., & Wu, D. (2022). Planetary gearbox fault diagnosis using bidirectional-convolutional LSTM networks. Mechanical Systems and Signal Processing, 162, 107996. https://doi.org/10.1016/j.ymssp.2021.107996.
Smith, W. A., & Randall, R. B. (2015). Rolling element bearing diagnostics using the Case Western Reserve University data: A benchmark study. Mechanical Systems and Signal Processing, 64–65, 100–131. https://doi.org/10.1016/j.ymssp.2015.04.021.
Song, J., Lee, Y. C., & Lee, J. (2023). Deep generative model with time series-image encoding for manufacturing fault detection in die casting process. Journal of Intelligent Manufacturing, 34(7), 3001–3014. https://doi.org/10.1007/s10845-022-0198-6.
Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J., Wojna, Z. (2016). Rethinking the inception architecture for computer vision (pp. 2818–2826). Presented at the proceedings of the IEEE conference on computer vision and, & recognition, Retrieved October, 12, 2022, From https://www.cv-foundation.org/openaccess/content_cvpr_2016/html/Szegedy_Rethinking_the_Inception_CVPR_2016_paper.html
Tang, Y., Zhang, X., Qin, G., Long, Z., Huang, S., Song, D., & Shao, H. (2022). Graph cardinality preserved attention network for fault diagnosis of induction motor under varying speed and load condition. IEEE Transactions on Industrial Informatics, 18(6), 3702–3712. https://doi.org/10.1109/TII.2021.3112696.
Tian, H., & Xu, Q. (2022). A spatio-temporal fault diagnosis method based on STF-DBN for reciprocating compressor. Journal of Intelligent Manufacturing. https://doi.org/10.1007/s10845-022-02025-9.
Veličković, P., Cucurull, G., Casanova, A., Romero, A., Liò, P., & Bengio, Y. (2018). Graph attention networks. arXiv. https://doi.org/10.48550/arXiv.1710.10903
Wang, H., Xu, J., Yan, R., & Gao, R. X. (2020a). A new intelligent bearing fault diagnosis method using SDP representation and SE-CNN. IEEE Transactions on Instrumentation and Measurement, 69(5), 2377–2389. https://doi.org/10.1109/TIM.2019.2956332.
Wang, T., Lu, G., & Yan, P. (2020b). A novel statistical time-frequency analysis for rotating machine condition monitoring. IEEE Transactions on Industrial Electronics, 67(1), 531–541. https://doi.org/10.1109/TIE.2019.2896109.
Wang, Y., Gao, L., Gao, Y., & Li, X. (2021). A new graph-based semi-supervised method for surface defect classification. Robotics and Computer-Integrated Manufacturing, 68, 102083. https://doi.org/10.1016/j.rcim.2020.102083.
Wu, J., Guo, P., Cheng, Y., Zhu, H., Wang, X. B., & Shao, X. (2020a). Ensemble generalized multiclass support-vector-machine-based health evaluation of complex degradation systems. IEEE/ASME Transactions on Mechatronics, 25(5), 2230–2240. https://doi.org/10.1109/TMECH.2020.3009449
Wu, J., Hu, K., Cheng, Y., Zhu, H., Shao, X., & Wang, Y. (2020b). Data-driven remaining useful life prediction via multiple sensor signals and deep long short-term memory neural network. ISA Transactions, 97, 241–250. https://doi.org/10.1016/j.isatra.2019.07.004
Xie, T., Huang, X., & Choi, S. K. (2022). Intelligent mechanical fault diagnosis using multisensor fusion and convolution neural network. IEEE Transactions on Industrial Informatics, 18(5), 3213–3223. https://doi.org/10.1109/TII.2021.3102017.
Xu, Y., Yan, X., Sun, B., & Liu, Z. (2022). Dually attentive multiscale networks for health state recognition of rotating machinery. Reliability Engineering & System Safety, 225, 108626. https://doi.org/10.1016/j.ress.2022.108626.
Zhang, Y., Xing, K., Bai, R., Sun, D., & Meng, Z. (2020). An enhanced convolutional neural network for bearing fault diagnosis based on time–frequency image. Measurement, 157, 107667. https://doi.org/10.1016/j.measurement.2020.107667.
Zhang, P., Li, T., Wang, G., Luo, C., Chen, H., Zhang, J., et al. (2021a). Multi-source information fusion based on rough set theory: A review. Information Fusion, 68, 85–117. https://doi.org/10.1016/j.inffus.2020.11.004
Zhang, Y., Li, C., Wang, R., & Qian, J. (2021b). A novel fault diagnosis method based on multi-level information fusion and hierarchical adaptive convolutional neural networks for centrifugal blowers. Measurement, 185, 109970. https://doi.org/10.1016/j.measurement.2021.109970
Zhang, J., Wu, Y., Xu, Z., Din, Z., & Chen, H. (2022). Fault diagnosis of high voltage circuit breaker based on multi-sensor information fusion with training weights. Measurement, 192, 110894. https://doi.org/10.1016/j.measurement.2022.110894.
Zhao, R., Yan, R., Chen, Z., Mao, K., Wang, P., & Gao, R. X. (2019). Deep learning and its applications to machine health monitoring. Mechanical Systems and Signal Processing, 115, 213–237. https://doi.org/10.1016/j.ymssp.2018.05.050.
Zhou, J., Cui, G., Hu, S., Zhang, Z., Yang, C., Liu, Z., et al. (2020). Graph neural networks: A review of methods and applications. AI Open, 1, 57–81. https://doi.org/10.1016/j.aiopen.2021.01.001.
Zhou, K., Yang, C., Liu, J., & Xu, Q. (2022). Dynamic graph-based feature learning with few edges considering noisy samples for rotating machinery fault diagnosis. IEEE Transactions on Industrial Electronics, 69(10), 10595–10604. https://doi.org/10.1109/TIE.2021.3121748.
Zhou, K., Yang, C., Liu, J., & Xu, Q. (2023). Deep graph feature learning-based diagnosis approach for rotating machinery using multi-sensor data. Journal of Intelligent Manufacturing, 34(4), 1965–1974. https://doi.org/10.1007/s10845-021-01884-y.
Acknowledgements
The authors would like to thank the editor and referees for their valuable comments. This work was supported by the National Natural Science Foundation of China (Grant No. 52075392).
Author information
Authors and Affiliations
Contributions
JL: methodology, software, formal analysis, writing—original draft. FX: formal analysis, writing—original draft. QZ: writing—original draft. QL: formal analysis, writing—original draft. XW: conceptualization, writing—review and editing. SW: writing—review and editing.
Corresponding author
Ethics declarations
Conflict of intrest
The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.
Additional information
Publisher’s Note
Springer nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.
About this article
Cite this article
Liu, J., Xie, F., Zhang, Q. et al. A multisensory time-frequency features fusion method for rotating machinery fault diagnosis under nonstationary case. J Intell Manuf 35, 3197–3217 (2024). https://doi.org/10.1007/s10845-023-02198-x
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10845-023-02198-x