Skip to main content

Advertisement

Springer Nature Link
Log in

LogAnomEX: An Unsupervised Log Anomaly Detection Method Based on Electra-DP and Gated Bilinear Neural Networks

  • Published:
Journal of Network and Systems Management Aims and scope Submit manuscript

Abstract

The aim of log anomaly detection is to accurately identify anomalies in system logs, with the objective of ensuring system reliability and stability, thereby mitigating avoidable losses. While anomaly detection schemes in this field have achieved some success, prior research typically relies solely on specific anomaly data for model training. This practice often fails to encompass all potential anomaly types, thereby limiting the generalizability of models in practical applications. Furthermore, the high dimensionality and dynamic nature of log data often pose challenges for traditional anomaly detection methods in effectively addressing novel or unknown anomaly patterns. Consequently, this paper introduces LogAnomEX, a novel unsupervised log anomaly detection model. LogAnomEX integrates a difficulty prediction module and a gated linear neural network, built upon the Electra model, to enhance its capacity in identifying unknown anomalies. The model achieves this by learning from normal log data and iteratively generating pseudo-anomalies resembling genuine anomalous logs. We evaluate LogAnomEX’s performance on the BGL, HDFS, and Thunderbird datasets, validating its effectiveness and superiority through comprehensive experimentation.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13

Similar content being viewed by others

Data Availability

Data supporting the findings are available from the corresponding author upon reasonable request. No datasets were generated or analysed during the current study.

References

  1. Jia, T., Li, Y., Wu, Z.: Survey of state-of-the-art log-based failure diagnosis. J. Softw. 31(07), 1997–2018 (2020) (in Chinese). https://doi.org/10.13328/j.cnki.jos.006045

  2. Zhu, Y., Dong, J., Xie, L., Wang, Z., Qin, S., Xu, P., Yin, M.: Recurrent multi-view collaborative registration network for 3D reconstruction and optical measurement of blade profiles. Knowl.-Based Syst. 295, 111857. Elsevier(2024)

  3. Qiu, K., Zhang, Y., Zhao, J., Zhang, S., Wang, Q., Chen, F.: A multimodal sentiment analysis approach based on a joint chained interactive attention mechanism. Electronics 13(10), 1922 (2024). (MDPI)

    Article  MATH  Google Scholar 

  4. Qiu, K., Zhang, Y., Ren, Z., et al.: SpemNet: a cotton disease and pest identification method based on efficient multi-scale attention and stacking patch embedding. Insects 15(9), 667 (2024)

    Article  MATH  Google Scholar 

  5. Zhang, X., Xu, Y., Lin, Q., Qiao, B., Zhang, H., Dang, Y., Xie, C., Yang, X., Cheng, Q., Li, Z., et al.: Robust log-based anomaly detection on unstable log data. In: Proceedings of the 2019 27th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, pp. 807–817 (2019)

  6. Studiawan, H., Sohel, F., Payne, C.: Anomaly detection in operating system logs with deep learning-based sentiment analysis. IEEE Trans. Depend. Secur. Comput. 18(5), 2136–2148 (2020). (IEEE)

    Article  MATH  Google Scholar 

  7. Clark, K., Luong, M.-T., Le, Q.V., Manning, C.D.: Electra: pre-training text encoders as discriminators rather than generators. arXiv preprint arXiv:2003.10555 (2020)

  8. Wang, T., Chen, B., Huang, R., Reng, L., Chen, Y., Qin, Y.: Chinese grammatical error diagnosis model based on electra and gated-bilinear neural network. J. Chin. Inf. Process. 37(08), 169–178 (2023). ((in Chinese))

    MATH  Google Scholar 

  9. Chandola, V., Banerjee, A., Kumar, V.: Anomaly detection: a survey. ACM Comput. Surv. (CSUR), 41(3), 1–58 (2009). ACM, New York

  10. He, S., Zhu, J., He, P., Lyu, M.R.: Experience report: System log analysis for anomaly detection. In: 2016 IEEE 27th International Symposium on Software Reliability Engineering (ISSRE), pp. 207–218. IEEE (2016)

  11. Xu, W., Huang, L., Fox, A., Patterson, D., Jordan, M.I.: Detecting large-scale system problems by mining console logs. In: Proceedings of the ACM SIGOPS 22nd Symposium on Operating Systems Principles, pp. 117–132 (2009)

  12. Lin, Q., Zhang, H., Lou, J.-G., Zhang, Y., Chen, X.: Log clustering based problem identification for online service systems. In: Proceedings of the 38th International Conference on Software Engineering Companion, pp. 102–111 (2016)

  13. Liang, Y., Zhang, Y., Xiong, H., Sahoo, R.: Failure prediction in ibm bluegene/l event logs. In: 7th IEEE International Conference on Data Mining (ICDM 2007), pp. 583–588. IEEE (2007)

  14. Bodik, P., Goldszmidt, M., Fox, A., Woodard, D.B., Andersen, H.: Fingerprinting the datacenter: automated classification of performance crises. In: Proceedings of the 5th European Conference on Computer Systems, pp. 111–124 (2010)

  15. Chen, M., Zheng, A.X., Lloyd, J., Jordan, M.I., Brewer, E.: Failure diagnosis using decision trees. In: International Conference on Autonomic Computing, 2004. Proceedings, pp. 36–43. IEEE (2004)

  16. Qi, J., Luan, Z., Huang, S., Fung, C., Yang, H., Li, H., Zhu, D., Qian, D.: Logencoder: log-based contrastive representation learning for anomaly detection. In: IEEE Transactions on Network and Service Management. IEEE (2023)

  17. Mikolov, T., Chen, K., Corrado, G., Dean, J.: Efficient estimation of word representations in vector space. arXiv preprint arXiv:1301.3781 (2013)

  18. Salton, G., Buckley, C.: Term Weighting Approaches in Automatic Text Retrieval. Cornell University (1987)

  19. Liu, F., Wen, Y., Zhang, D., Jiang, X., Xing, X., Meng, D.: Log2vec: a heterogeneous graph embedding based approach for detecting cyber threats within enterprise. In: Proceedings of the 2019 ACM SIGSAC Conference on Computer and Communications Security, pp. 1777–1794 (2019)

  20. Du, M., Li, F., Zheng, G., Srikumar, V.: Deeplog: Anomaly detection and diagnosis from system logs through deep learning. In: Proceedings of the 2017 ACM SIGSAC Conference on Computer and Communications Security, pp. 1285–1298 (2017)

  21. Sasaki, S., Suzuki, J., Inui, K.: Subword-based compact reconstruction of word embeddings. In: Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pp. 3498–3508 (2019)

  22. Wang, J., Yu, L.-C., Lai, K.R., Zhang, X.: Dimensional sentiment analysis using a regional CNN-LSTM model. In: Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pp. 225–230 (2016)

  23. Luo, L.: Network text sentiment analysis method combining LDA text representation and GRU-CNN. Pers. Ubiquit. Comput. 23, 405–412 (2019). (Springer)

    Article  MATH  Google Scholar 

  24. Yu, H., Yang, J.: A direct LDA algorithm for high-dimensional data?with application to face recognition. Pattern Recogn. 34, 2067–2070 (2001). (Elsevier)

    Article  MATH  Google Scholar 

  25. Yan, S., Shi, F., Yu, K., et al.: TASA: template-driven log anomaly detection with variable integration and sparse attention. Artif. Intell. Secur. 3(03), 12–20 (2024)

    MATH  Google Scholar 

  26. Devlin, J., Chang, M.-W., Lee, K., Toutanova, K.: Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 (2018)

  27. Vaswani, A. Attention is all you need. Advances in Neural Information Processing Systems, (2017).

  28. Guo, H., Yuan, S., Wu, X.: Logbert: log anomaly detection via BERT. In: Proceedings of the 2021 International Joint Conference on Neural Networks (IJCNN), pp. 1–8. IEEE (2021)

  29. Yu, J., Hu, Z., Jiang, C.: Multi-feature-based log event anomaly detection. Comput. Eng. Sci. 46(09), 1587–1597 (2024)

    MATH  Google Scholar 

  30. Yamanaka, Y., Takahashi, T., Minami, T., Nakajima, Y.: LogELECTRA: self-supervised anomaly detection for unstructured logs. arXiv preprint arXiv:2402.10397 (2024)

  31. Lin, Y., Deng, H., Li, X.: FastLogAD: log anomaly detection with mask-guided pseudo anomaly generation and discrimination. arXiv preprint arXiv:2404.08750 (2024)

  32. Mardani, M., Mateos, G., Giannakis, G.B.: Recovery of low-rank plus compressed sparse matrices with application to unveiling traffic anomalies. IEEE Trans. Inf. Theory 59(8), 5186–5205 (2013)

    Article  MathSciNet  MATH  Google Scholar 

  33. Jin, Y., Qiu, C., Sun, L., et al.: Anomaly detection in time series via robust PCA. In: Proceedings of the 2017 2nd IEEE International Conference on Intelligent Transportation Engineering (ICITE), pp. 352–355. IEEE (2017)

  34. Kalaki, P.S., Shameli-Sendi, A., Abbasi, B.K.E.: Anomaly detection on openstack logs based on an improved robust principal component analysis model and its projection onto column space. Softw.: Pract. Exp. 53(3), 665–681 (2023)

    Google Scholar 

  35. Weber, I., Garimella, V.R.K., Borra, E.: Mining web query logs to analyze political issues. In: Proceedings of the 4th Annual ACM Web Science Conference, pp. 330–334 (2012)

  36. Guzman, E., Azócar, D., Li, Y.: Sentiment analysis of commit comments in GitHub: an empirical study. In: Proceedings of the 11th Working Conference on Mining Software Repositories, pp. 352–355 (2014)

  37. Dong, Y., Zhao, K.: Log anomaly detection method based on attention mechanism multi-feature fusion and text sentiment analysis . J. Sichuan Univer. (Nat. Sci. Ed.) 61, 76–86 (2024) (in Chinese). https://doi.org/10.19907/j.0490-6756.2024.023001

  38. Meng, W., Liu, Y., Zhu, Y., Zhang, S., Pei, D., Liu, Y., Chen, Y., Zhang, R., Tao, S., Sun, P.: Loganomaly: unsupervised detection of sequential and quantitative anomalies in unstructured logs. In: IJCAI, pp. 4739–4745 (2019)

  39. Du, M., Li, F.: Spell: online streaming parsing of large unstructured system logs. IEEE Trans. Knowl. Data Eng. 31, 2213–2227 (2018). (IEEE)

    Article  MATH  Google Scholar 

  40. Song, X., Salcianu, A., Song, Y., Dopson, D., Zhou, D.: Fast wordpiece tokenization. arXiv preprint arXiv:2012.15524 (2020)

  41. He, P., Zhu, J., Zheng, Z., et al.: Drain: an online log parsing approach with fixed depth tree. In: Proceedings of the 2017 IEEE International Conference on Web Services (ICWS), Honolulu. IEEE (2017)

  42. Hao, Y., Dong, L., Bao, H., Xu, K., Wei, F.: Learning to sample replacements for electra pre-training. arXiv preprint arXiv:2106.13715 (2021)

  43. Nedelkoski, S., Bogatinovski, J., Acker, A., Cardoso, J., Kao, O.: Self-attentive classification-based anomaly detection in unstructured logs. In: 2020 IEEE International Conference on Data Mining (ICDM), pp. 1196–1201. IEEE (2020)

  44. Oliner, A., Stearley, J.: What supercomputers say: A study of five system logs. In: 37th Annual IEEE/IFIP International Conference on Dependable Systems and Networks (DSN’07), pp. 575–584. IEEE (2007)

  45. Zhu, J., He, S., He, P., Liu, J., Lyu, M.: Loghub: A large collection of system log datasets for ai-driven log analytics. In: 2023 IEEE 34th International Symposium on Software Reliability Engineering (ISSRE), pp. 355–366. IEEE (2023)

  46. Xu, W., Huang, L., Fox, A., Patterson, D., Jordan, M.I.: Detecting large-scale system problems by mining console logs. In: Proceedings of the ACM SIGOPS 22nd Symposium on Operating Systems Principles, pp. 117–132 (2009)

  47. Vaarandi, R., Pihelgas, M.: Logcluster-a data clustering and pattern mining algorithm for event logs. In: 2015 11th International Conference on Network and Service Management (CNSM), pp. 1–7. IEEE (2015)

  48. Lou, J.G., Fu, Q., Yang, S., et al.: Mining invariants from console logs for system problem detection. In: Proceedings of the 2010 USENIX Annual Technical Conference (USENIX ATC 10) (2010)

  49. Yin, K., Yan, M., Xu, L., et al.: Improving log-based anomaly detection with component-aware analysis. In: Proceedings of the 2020 IEEE International Conference on Software Maintenance and Evolution (ICSME), pp. 667–671. IEEE (2020)

  50. Wang, Z., Chen, Z., Ni, J., et al.: Multi-scale one-class recurrent neural networks for discrete event sequence anomaly detection. In: Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining, pp. 3726–3734 (2021)

  51. Zhang, C., Wang, X., Zhang, H., et al.: LayerLog: log sequence anomaly detection based on hierarchical semantics. Appl. Soft Comput. 132, 109860 (2023)

    Article  MATH  Google Scholar 

  52. Chen, X., Zhang, S., Jing, Y., Wang, S.: TRGATLog: log anomaly detection method based on log time relation graph attention network. Appl. Res. Comput. 41, 1034–1040 (2024) (in Chinese). https://doi.org/10.19734/j.issn.1001-3695.2023.07.0365

  53. Chen, L., Song, C., Wang, X., Fu, D., Li, F.: CSCLog: a component subsequence correlation-aware log anomaly detection method. arXiv preprint arXiv:2307.03359 (2023)

  54. Yin, Z., Kong, X., Yin, C.: Semi-supervised log anomaly detection based on bidirectional temporal convolution network. Comput. Secur. 140, 103808 (2024)

    Article  MATH  Google Scholar 

Download references

Funding

The authors are thankful for the support of the China Scholarship Council (No. 202206490011).

Author information

Authors and Affiliations

  1. College of Information Science and Technology, Shihezi University, No. 221, North Fourth Road, Shihezi, 832003, The Xinjiang Uygur Autonomous Region, China

    Keyuan Qiu, Yingjie Zhang & Feng Chen

  2. School of Economics, Beijing Wuzi University, 321 Fuhe Street, Beijing, 101126, China

    Yiqiang Feng

Authors
  1. Keyuan Qiu
    View author publications

    You can also search for this author inPubMed Google Scholar

  2. Yingjie Zhang
    View author publications

    You can also search for this author inPubMed Google Scholar

  3. Yiqiang Feng
    View author publications

    You can also search for this author inPubMed Google Scholar

  4. Feng Chen
    View author publications

    You can also search for this author inPubMed Google Scholar

Contributions

Keyuan Qiu was responsible for the design of the experiments and the writing of the thesis; Yingjie Zhang was responsible for the editing of the thesis format and the processing of the pictures; Feng Chen was responsible for the supervision of the thesis and the suggestion of the revision. Yiqiang Feng was responsible for providing technical support and financial assistance. All authors have read and agreed to the published version of the manuscript.

Corresponding author

Correspondence to Feng Chen.

Ethics declarations

Conflict of interest

The authors declare no Conflict of interest.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Qiu, K., Zhang, Y., Feng, Y. et al. LogAnomEX: An Unsupervised Log Anomaly Detection Method Based on Electra-DP and Gated Bilinear Neural Networks. J Netw Syst Manage 33, 33 (2025). https://doi.org/10.1007/s10922-025-09912-5

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s10922-025-09912-5

Keywords