Abstract
In this paper, a hybrid model of regularized Long Short-Term Memory (LSTM) and autoencoder for fault detection in automated production systems is proposed. The presented LSTM autoencoder is used as a stochastic process model, which captures the normal behavior of a production system and allows to predict the probability distribution of sensor data. Discrepancies between the observed sensor data and the predicted probability density distribution are detected as potential faults. The approach combines the advantages of LSTMs and autoencoders: The correlations between individual sensor signals are exploited by an autoencoder, while the temporal dependencies are captured by LSTM neurons. A key challenge in training such a process model from historical data is to control the information passed through the latent space of the autoencoder. Different regularization methods are investigated for this purpose. Fault detection with the proposed LSTM autoencoder has been evaluated on the use case of an industrial penicillin production, achieving significantly improved results in comparison to the baseline LSTM.
Zusammenfassung
In diesem Beitrag wird ein hybrides Modell aus einem Long Short-Term Memory (LSTM) Modell und einem Autoencoder zur Fehlererkennung in automatisierten Produktionssystemen untersucht. Der untersuchte LSTM-Autoencoder wird als stochastisches Prozessmodell verwendet, welches das Normalverhalten eines Produktionssystems nachbildet und die Vorhersage der Wahrscheinlichkeitsverteilung von Sensordaten ermöglicht. Diskrepanzen zwischen den beobachteten Sensordaten und der vorhergesagten Wahrscheinlichkeitsverteilung werden als potentielle Fehler erkannt. Der Ansatz kombiniert die Vorteile von LSTMs und Autoencodern: Die Korrelationen zwischen einzelnen Sensorsignalen werden von einem Autoencoder ausgenutzt, während zeitliche Abhängigkeiten in den versteckten Zuständen der LSTM-Neuronen gespeichert werden. Eine zentrale Herausforderung beim Training eines solchen Prozessmodells aus historischen Daten ist die Regularisierung des Latenzbereichs, wozu verschiedene Methoden in diesem Paper untersucht werden. Die Fehlererkennung mit dem vorgeschlagenen LSTM-Autoencoder wurde am Anwendungsfall einer industriellen Penicillin-Produktion evaluiert, wobei im Vergleich zu einem herkömmlichen LSTM deutlich bessere Ergebnisse erzielt wurden.
Funding source: Bundesministerium für Wirtschaft und Energie
Award Identifier / Grant number: IGF project nr. 20726N
About the authors
Stefan Windmann received the Dipl.-Ing. and Dipl.-Inf. degrees in electrical engineering and technical computer sciences fromUniversity of Paderborn, Germany, in 2004, where he received the Ph.D. degree in electrical engineering in 2008. He is currentlyemployed as senior scientist at Fraunhofer IOSB-INA in Lemgo, Germany. His current research interests include machine learningalgorithms and methods for diagnosis and optimization of automated production systems.
Tim Westerhold is an electrical engineer with interests in the fields of autonomous systems and machine learning. He received aBachelor's degree in electrical engineering from TH OWL (2017) and a Master's degree in Intelligent Systems from University ofBielefeld (2021). Currently, he is employed at dSPACE GmbH in Paderborn as a project engineer in the field of driver assistance and autonomous driving.
-
Research ethics: Not applicable
-
Author contributions: The authors have accepted responsibility for theentire content of this manuscript and approved its submission.
-
Competing interests: The authors state no conflict of interest.
-
Research funding: The study was financially supported by theBundesministerium für Wirtschaft und Energie (IGF project nr. 20726N).
-
Data availability: The raw data can be obtained on request from thecorresponding author.
References
[1] A. Raza, K. M. Deen, R. Jaafreh, K. Hamad, A. Haider, and W. Haider, “Incorporation of machine learning in additive manufacturing: a review,” Int. J. Adv. Des. Manuf. Technol., vol. 122, pp. 1143–1166, 2022. https://doi.org/10.1007/s00170-022-09916-4.Search in Google Scholar
[2] X. Song, P. Sun, S. Song, and V. Stojanovic, “Event-driven NN adaptive fixed-time control for nonlinear systems with guaranteed performance,” J. Franklin Inst., vol. 359, pp. 4138–4159, 2022. https://doi.org/10.1016/j.jfranklin.2022.04.003.Search in Google Scholar
[3] Z. Zhuang, Y. Chen, H. Tao, V. Stojanovic, and W. Paszke, “Iterative learning control for repetitive tasks with randomly varying trial lengths using successive projection,” Int. J. Adapt. Control Signal Process., vol. 36, pp. 1196–1215, 2022. https://doi.org/10.1002/acs.3396.Search in Google Scholar
[4] H. Tao, L. Cheng, J. Qiu, and V. Stojanovic, “Few shot cross equipment fault diagnosis method based on parameter optimization and feature metric,” Meas. Sci. Technol., vol. 33, p. 115005, 2022. https://doi.org/10.1088/1361-6501/ac8368.Search in Google Scholar
[5] X. Zhang, H. Wang, V. Stojanovic, et al.., “Asynchronous Fault detection for interval type-2 fuzzy nonhomogeneous higher level Markov jump systems with uncertain transition probabilities,” IEEE Trans. Fuzzy Syst., vol. 30, pp. 2487–2499, 2022. https://doi.org/10.1109/tfuzz.2021.3086224.Search in Google Scholar
[6] P. Malhotra, A. Ramakrishnan, and G. Anand, “LSTM-Based encoder-decoder for multi-sensor anomaly detection,” in ICML 2016 Anomaly Detection Workshop, 2016.Search in Google Scholar
[7] T. Kieu, B. Yang, and C. S. Jensen, “Outlier detection for multidimensional time series using deep neural networks,” in 19th IEEE International Conference on Mobile Data Management, 2018.10.1109/MDM.2018.00029Search in Google Scholar
[8] Q. Chen, Z. Liu, X. Ma, and Y. Wang, “Artificial neural correlation analysis for performance-indicator-related nonlinear process monitoring,” IEEE Trans. Ind. Inf., vol. 18, pp. 1039–1049, 2022. https://doi.org/10.1109/tii.2021.3080285.Search in Google Scholar
[9] M. Dix, S. Pradhan, A. Chouhan, et al.., “Anomaly detection in the time-series data of industrial plants using neural network architectures,” in IEEE Seventh International Conference on Big Data Computing Service and Applications (BigDataService), 2021.10.1109/BigDataService52369.2021.00035Search in Google Scholar
[10] B. Eiteneuer, N. Hranisavljevic, and O. Niggemann, “Dimensionality reduction and anomaly detection for CPPS data using autoencoder,” in 2019 IEEE International Conference on Industrial Technology (ICIT), 2019, pp. 1286–1292.10.1109/ICIT.2019.8755116Search in Google Scholar
[11] K. Jang, S. Hong, M. Kim, J. Na, and I. Moon, “Adversarial autoencoder based feature learning for fault detection in industrial processes,” IEEE Trans. Ind. Inf., vol. 18, pp. 827–834, 2022. https://doi.org/10.1109/tii.2021.3078414.Search in Google Scholar
[12] K. Jiang, Z. Jiang, Y. Xie, D. Pan, and W. Gui, “Abnormality monitoring in the blast furnace ironmaking process based on stacked dynamic target-driven denoising autoencoders,” IEEE Trans. Ind. Inf., vol. 18, pp. 1854–1863, 2022. https://doi.org/10.1109/tii.2021.3084911.Search in Google Scholar
[13] Q. Jiang, S. Yan, X. Yan, H. Yi, and F. Gao, “Data-driven two-dimensional deep correlated representation learning for nonlinear batch process monitoring,” IEEE Trans. Ind. Inf., vol. 16, no. 4, pp. 2839–2848, 2020. https://doi.org/10.1109/tii.2019.2952931.Search in Google Scholar
[14] Q. Jiang and X. Yan, “Learning deep correlated representations for nonlinear process monitoring,” IEEE Trans. Ind. Inf., vol. 15, no. 12, pp. 6200–6209, 2019. https://doi.org/10.1109/tii.2018.2886048.Search in Google Scholar
[15] D. Kwon, H. Kim, J. Kim, S. C. Suh, I. Kim, and K. J. Kim, “A survey of deep learning-based network anomaly detection,” Cluster Comput., vol. 22, pp. 949–961, 2019. https://doi.org/10.1007/s10586-017-1117-8.Search in Google Scholar
[16] Z. Ren, W. Zhang, and Z. Zhang, “A deep nonnegative matrix factorization approach via autoencoder for nonlinear fault detection,” IEEE Trans. Ind. Inf., vol. 16, no. 8, pp. 5042–5052, 2020. https://doi.org/10.1109/tii.2019.2951011.Search in Google Scholar
[17] Z. Gao, C. Cecati, and S. X. Ding, “A survey of fault diagnosis and fault tolerant techniques,” IEEE Trans. Ind. Electron., vol. 62, no. 6, pp. 3757–3774, 2015.10.1109/TIE.2015.2417501Search in Google Scholar
[18] X. Kong and Z. Ge, “Deep learning of latent variable models for industrial process monitoring,” IEEE Trans. Ind. Inf., vol. 18, pp. 6778–6788, 2022. https://doi.org/10.1109/tii.2021.3134251.Search in Google Scholar
[19] R. T. Samuel and Y. Cao, “Nonlinear process fault detection and identification using kernel PCA and kernel density estimation,” Syst. Sci. Control Eng., vol. 4, no. 1, pp. 165–174, 2016. https://doi.org/10.1080/21642583.2016.1198940.Search in Google Scholar
[20] M. Mansouri, M. Nounou, H. Nounou, and N. Karim, “Kernel PCAbased GLRT for nonlinear fault detection of chemical processes,” J. Loss Prev. Process Ind., vol. 40, pp. 334–347, 2016. https://doi.org/10.1016/j.jlp.2016.01.011.Search in Google Scholar
[21] C. Botre, M. Mansouri, M. Nounou, H. Nounou, and M. N. Karim, “Kernel PLS-based GLRT method for fault detection of chemical processes,” J. Loss Prev. Process Ind., vol. 43, pp. 212–224, 2016. https://doi.org/10.1016/j.jlp.2016.05.023.Search in Google Scholar
[22] J. Fan and Y. Wang, “Fault detection and diagnosis of non-linear non-Gaussian dynamic processes using kernel dynamic independent component analysis,” Inf. Sci., vol. 259, pp. 369–379, 2014. https://doi.org/10.1016/j.ins.2013.06.021.Search in Google Scholar
[23] X. Xiu, Z. Miao, Y. Yang, and W. Liu, “Deep canonical correlation analysis using sparsity-constrained optimization for nonlinear process monitoring,” IEEE Trans. Ind. Inf., vol. 18, pp. 6690–6699, 2022. https://doi.org/10.1109/tii.2021.3121770.Search in Google Scholar
[24] Y. Wang, F. Sun, and B. Li, “Multiscale neighborhood normalization-based multiple dynamic PCA monitoring method for batch processes with frequent operations,” IEEE Trans. Autom. Sci. Eng., vol. 15, no. 3, pp. 1053–1064, 2018. https://doi.org/10.1109/tase.2017.2713800.Search in Google Scholar
[25] Q. Jiang and X. Yan, “Multimode process monitoring using variational bayesian inference and canonical correlation analysis,” IEEE Trans. Autom. Sci. Eng., vol. 16, no. 4, pp. 1814–1824, 2019. https://doi.org/10.1109/tase.2019.2897477.Search in Google Scholar
[26] L. Bontemps, J. McDermott, N.-A. Le-Khac, et al.., “Collective anomaly detection based on long short-term memory recurrent neural networks,” in International Conference on Future Data and Security Engineering, 2016.10.1007/978-3-319-48057-2_9Search in Google Scholar
[27] B. Eiteneuer and O. Niggemann, “LSTM for model-based anomaly detection in cyber-physical systems,” in 29th International Workshop on Principles of Diagnosis, 2018.Search in Google Scholar
[28] F. Kong, J. Li, B. Jiang, H. Wang, and H. Song, “Integrated generative model for industrial anomaly detection via bidirectional LSTM and attention mechanism,” IEEE Trans. Ind. Inf., vol. 19, pp. 541–550, 2023. https://doi.org/10.1109/tii.2021.3078192.Search in Google Scholar
[29] S. Windmann, “Data-driven fault detection in industrial BatchProcesses based on a StochasticHybrid process model,” IEEE Trans. Autom. Sci. Eng., vol. 19, pp. 3888–3902, 2022. https://doi.org/10.1109/tase.2021.3138925.Search in Google Scholar
[30] P. Malhotra, L. Vig, G. Shroff, and P. Agarwal, “Long short term memory networks for anomaly detection in time series,” in 23rd European Symposium on Artificial Neural Networks, 2015.Search in Google Scholar
[31] Y. Wang, X. Du, Z. Lu, Q. Duan, and J. Wu, “Improved LSTM-based time-series anomaly detection in rail transit operation environments,” IEEE Trans. Ind. Inf., vol. 18, pp. 9027–9036, 2022. https://doi.org/10.1109/tii.2022.3164087.Search in Google Scholar
[32] D. Park, Y. Hoshi, and C. Kemp, “A multimodal anomaly detector for robot-assisted feeding using an lstm-based variational autoencoder,” IEEE Rob. Autom. Lett., vol. 3, pp. 1544–1551, 2017. https://doi.org/10.1109/lra.2018.2801475.Search in Google Scholar
[33] P. Han, A. L. Ellefsen, G. Li, F. T. Holmeset, and H. Zhang, “Fault detection with LSTM-based variational autoencoder for maritime components,” IEEE Sensor. J., vol. 21, no. 19, pp. 21903–21912, 2021. https://doi.org/10.1109/jsen.2021.3105226.Search in Google Scholar
[34] D. P. Kingma and J. Ba. Adam, “A method for stochastic optimization,” in 3rd International Conference for Learning Representations, 2015.Search in Google Scholar
[35] I. Goodfellow, Y. Bengio, and A. Courville, Deep Learning, Cambridge, MA, MIT Press, 2016.Search in Google Scholar
[36] C. Zhou and R. C. Paffenroth, “Anomaly detection with robust deep autoencoders,” in Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 2017, pp. 665–674.10.1145/3097983.3098052Search in Google Scholar
[37] S. Merity, B. McCann, and R. Socher, Revisiting Regularization for Language RNNs. Available at: https://arxiv.org/pdf/170801009 [accessed: Aug. 25, 2020].Search in Google Scholar
[38] D. Park, Y. Hoshi, and C. C. Kemp, “A multimodal anomaly detector for robot assisted feeding using an LSTM-based variational autoencoder,” IEEE Rob. Autom. Lett., vol. 3, pp. 1544–1551, 2017. https://doi.org/10.1109/lra.2018.2801475.Search in Google Scholar
[39] P. Vincent, H. Larochelle, I. Lajoie, and P. A. Manzagol, “Stacked denoising autoencoders: learning useful representations in a deep network with a local denoising criterion,” J. Mach. Learn. Res., vol. 11, pp. 3371–3408, 2010.Search in Google Scholar
[40] H. Steck, “Autoencoders that don’t overfit towards the Identity,” in 34th Conference on Neural Information Processing Systems (NeurIPS), 2020.Search in Google Scholar
[41] J. van Impe and G. Gins, “An extensive reference dataset for fault detection and Identification in batch processes,” Chemom. Intell. Lab. Syst., vol. 148, pp. 20–31, 2015. https://doi.org/10.1016/j.chemolab.2015.08.019.Search in Google Scholar
[42] G. Birol, C. Ündey, and A. Cinar, “A modular simulation package for fed-batch fermentation: penicillin production,” Chemom. Intell. Lab. Syst., vol. 26, pp. 1552–1565, 2002. https://doi.org/10.1016/s0098-1354(02)00127-8.Search in Google Scholar
[43] M. Abadi, P. Barham, J. Chen, et al.., “Tensorflow: a system for large-scale machine learning,” OSDI, vol. 16, pp. 265–283, 2016.Search in Google Scholar
© 2023 Walter de Gruyter GmbH, Berlin/Boston
