Skip to main content
SpringerLink
Log in

Generalized transformer in fault diagnosis of Tennessee Eastman process

  • Original Article
  • Published:
Neural Computing and Applications Aims and scope Submit manuscript

Abstract

Fault diagnosis is an important yet challenging task. Because of the powerful feature representation capabilities of deep model, intelligent fault diagnosis on deep learning becomes a research hotspot in the field. Although many deep models as sparse autoencoder, deep belief network is developed for fault diagnosis with encouraging performance, integrating the merits of deep learning into fault diagnosis still has a long way to go. In this paper, we propose a novel method, namely generalized transformer. Compared to previous deep models, generalized transformer excavates relations among inputs and nonlinearity between inputs and outputs by attention mechanism. To deal with structured data, generalized transformer further borrows the idea from graph attention network. By replacing dot product between query and key information in transformer, we introduce a forward network with learned weight vector to compute the similarity. Through limiting the similarity calculations in a neighbor region, prior knowledge can be injected into generalized transformer. On Tennessee Eastman process dataset, our new model can produce high performance, which is better or competitive to state-of-the-art models. Extensive ablation studies validate the effectiveness of the proposed model.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5

Similar content being viewed by others

References

  1. Korbicz J, Koscielny JM, Kowalczuk Z, Cholewa W (2004) Fault diagnosis. Models, artificial intelligence, applications. Springer, Berlin

    MATH  Google Scholar 

  2. Ku W, Storer RH, Georgakis C (1995) Disturbance detection and isolation by dynamic principal component analysis. Chemom Intell Lab Syst 30(1):179–196

    Article  Google Scholar 

  3. Mansouri M, Nounou M, Nounou H, Karim N (2016) Kernel pca-based glrt for nonlinear fault detection of chemical processes. J Loss Prev Process Indust 40((Supplement C)):334–347

    Article  Google Scholar 

  4. Yin S, Ding SX, Xie X, Luo H (2014) A review on basic data-driven approaches for industrial process monitoring. IEEE Trans Industr Electron 61(11):6418–6428

    Article  Google Scholar 

  5. Chen Z, Li H (2017) Multisensor feature fusion for bearing fault diagnosis using sparse autoencoder and deep belief network. IEEE Trans Instrum Meas 66(7):1693–1702

    Article  Google Scholar 

  6. Wu H, Zhao J (2018) Deep convolutional neural network model based chemical process fault diagnosis. Comput Chem Eng 115(12):185–197

    Article  Google Scholar 

  7. Wang Y, Pan Z, Yuan X, Yang C, Gui W (2020) A novel deep learning based fault diagnosis approach for chemical process with extended deep belief network. ISA Trans 96:457–467

    Article  Google Scholar 

  8. Li X, Zhang W, Ding Q (2019) Understanding and improving deep learning-based rolling bearing fault diagnosis with attention mechanism. Signal Process 161:136–154

    Article  Google Scholar 

  9. He K, Zhang X, Ren S, Sun J (2016) Deep residual learning for image recognition. In: CVPR, pp 770–778

  10. Vaswani A, Shazeer N, Parmar N, Uszkoreit J, Jones L, Gomez AN, Kaiser L, Polosukhin I (2017) Attention is all you need. In: NIPS, pp 5998–6008

  11. Veličković P, Cucurull G, Casanova A, Romero A, Lió P, Bengio Y (2018) Graph attention networks. In ICLR, pages 1–12

  12. Turner CR, Fuggetta A, Lavazza L, Wolf AL (1999) A conceptual basis for feature engineering. J Syst Softw 49(1):3–15

    Article  Google Scholar 

  13. Bengio Y, Courville A, Vincent P (2013) Representation learning: A review and new perspectives. IEEE Trans Pattern Anal Mach Intell 35(8):1798–1828

    Article  Google Scholar 

  14. Sun Q, Ge Z (2021) A survey on deep learning for data-driven soft sensors. IEEE Trans Indust Inform 17(9):5853–5866

    Article  Google Scholar 

  15. Chiang LH, Russell EL, Braatz RD (2000) Fault diagnosis in chemical processes using fisher discriminant analysis, discriminant partial least squares, and principal component analysis. Chemom Intell Lab Syst 50:243–252

    Article  Google Scholar 

  16. Yin S, Ding SX, Haghani A, Hao H, Zhang P (2012) A comparison study of basic data-driven fault diagnosis and process monitoring methods on the benchmark tennessee eastman process. J Process Control 22:1567–1581

    Article  Google Scholar 

  17. Tax DM, Duin RP (1999) Support vector domain description. Pattern Recognit Lett 20:1191–1199

    Article  Google Scholar 

  18. Ge Z, Zhang M, Song Z (2010) Nonlinear process monitoring based on linear subspace and bayesian inference. J Process Control 20(5):676–688

    Article  Google Scholar 

  19. Zhang J, Chen H, Chen S, Hong X (2017) An improved mixture of probabilistic pca for nonlinear data-driven process monitoring. IEEE Trans Cybern (Spec) 49:198–210

    Article  Google Scholar 

  20. Wang Y, Sun F, Li B (2018) Multiscale neighborhood normalization-based multiple dynamic pca monitoring method for batch processes with frequent operations. IEEE Trans Autom Sci Eng 17(9):1053–1064

    Article  Google Scholar 

  21. Gao X, Hou J (2016) An improved svm integrated gs-pca fault diagnosis approach of tennessee eastman process. Neurocomputing 174:906–911

    Article  Google Scholar 

  22. You D, Gao X, Katayama S (2015) Wpd-pca-based laser welding process monitoring and defects diagnosis by using fnn and svm. IEEE Trans Industr Electron 62(1):628–636

    Article  Google Scholar 

  23. Deng X, Zhang Z (2020) Nonlinear chemical process fault diagnosis using ensemble deep support vector data description. Sensors 20(4599):1–19

    Google Scholar 

  24. Guo C, Hu W, Yang F, Huang D (2020) Deep learning technique for process fault detection and diagnosis in the presence of incomplete data. Chin J Chem Eng 28(9):2358–2367

    Article  Google Scholar 

  25. Zhao H, Sun S, Jin B (2018) Sequential fault diagnosis based on lstm neural network. IEEE Access 6:12929–12939

    Article  Google Scholar 

  26. Zhang Z, Zhao J (2017) A deep belief network based fault diagnosis model for complex chemical processes. Comput Chem Eng 107:395–407

    Article  Google Scholar 

  27. Cheng F, He QP, Zhao J (2019) A novel process monitoring approach based on variational recurrent autoencoder. Comput Chem Eng 129:106515106515

    Article  Google Scholar 

  28. Wang Z, Su Y, Shen W, Jin S, Clark JH, Ren J, Zhang X (2019) Predictive deep learning models for environmental properties: the direct calculation of octanol water partition coefficients from molecular graphs. Green Chem 16:4555–4565

    Article  Google Scholar 

  29. Bahdanau D, Cho K, Bengio Y (2015) Neural machine translation by jointly learning to align and translate. In: ICLR

  30. Gehring J, Auli M, Grangier D, Dauphin Y. N (2017) A convolutional encoder model for neural machine translation. In: ACL

  31. Downs JJ, Vogel EF (1993) A plant-wide industrial process control problem. Comput Chem Eng 17(3):245–255

    Article  Google Scholar 

  32. Kingma D. P, Ba J (2014) Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980

  33. Zhang L, Du Y, Li X, Zhen X (2020) Calibrated multivariate regression networks. IEEE Trans Circuits Syst Video Technol 30(11):4222–4231

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Guangdong University of Petrochemical Technology, Maoming, Guangdong, China

    Lei Zhang, Qinghua Zhang & Zhiping Peng

  2. The State Key Laboratory of Industrial Control Technology, College of Control Science and Engineering, Zhejiang University, Hangzhou, China

    Zhihuan Song

Authors
  1. Lei Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Zhihuan Song
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Qinghua Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Zhiping Peng
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

ZP and ZS were involved in conceptualization. LZ contributed to writing and experiments. ZP and QZ contributed to methodology and supervision. Thanks for the support from National Natural Science Foundation of China (61933013), Project of Educational Commission of Guangdong province of China (2018KCXTD019), and Natural Science Foundation of Guangdong Province of China (2021A1515011846).

Corresponding author

Correspondence to Zhiping Peng.

Ethics declarations

Conflict of interest

The authors declare no conflict of interest.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Zhang, L., Song, Z., Zhang, Q. et al. Generalized transformer in fault diagnosis of Tennessee Eastman process. Neural Comput & Applic 34, 8575–8585 (2022). https://doi.org/10.1007/s00521-021-06711-2

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00521-021-06711-2

Keywords

Search

Navigation