Skip to main content
SpringerLink
Log in

Semi-supervised and unsupervised anomaly detection by mining numerical workflow relations from system logs

  • Published:
Automated Software Engineering Aims and scope Submit manuscript

Abstract

Large-scale software-intensive systems often generate logs for troubleshooting purpose. The system logs are semi-structured text messages that record the internal status of a system at runtime. In this paper, we propose ADR (Anomaly Detection by workflow Relations), which can mine numerical relations from logs and then utilize the discovered relations to detect system anomalies. Firstly the raw log entries are parsed into sequences of log events and transformed to an extended event-count-matrix. The relations among the matrix columns represent the relations among the system events in workflows. Next, ADR evaluates the matrix’s nullspace that corresponds to the linearly dependent relations of the columns. Anomalies can be detected by evaluating whether or not the logs violate the mined relations. We design two types of ADR: sADR (for semi-supervised learning) and uADR (for unsupervised learning). We have evaluated them on four public log datasets. The experimental results show that ADR can extract the workflow relations from log data, and is effective for log-based anomaly detection in both semi-supervised and unsupervised manners.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8

Similar content being viewed by others

References

  • Astekin, M., Zengin, H., Sözer, H.: Evaluation of distributed machine learning algorithms for anomaly detection from large-scale system logs: a case study. In: 2018 IEEE International Conference on Big Data (Big Data), pp. 2071–2077 (2018)

  • Bertero, C., Roy, M., Sauvanaud, C., et al.: Experience report: log mining using natural language processing and application to anomaly detection. In: 2017 IEEE 28th International Symposium on Software Reliability Engineering (ISSRE), pp. 351–360 (2017)

  • Bodik, P., Goldszmidt, M., Fox, A., et al.: Fingerprinting the datacenter: automated classification of performance crises. In: Proceedings of the 5th European Conference on Computer Systems. ACM, Paris, France, EuroSys ’10, pp. 111–124 (2010)

  • Breier, J., Branišová, J.: A dynamic rule creation based anomaly detection method for identifying security breaches in log records. Wirel. Pers. Commun. 94(3), 497–511 (2017)

    Article  Google Scholar 

  • Chen, M., Zheng, A.X., Lloyd, J., et al.: Failure diagnosis using decision trees. In: International Conference on Autonomic Computing, 2004. Proceedings., pp. 36–43 (2004)

  • Du, M., Li, F.: Spell: streaming parsing of system event logs. In: 2016 IEEE 16th International Conference on Data Mining (ICDM), pp. 859–864 (2016)

  • Du, M., Li, F., Zheng, G., et al.: DeepLog: anomaly detection and diagnosis from system logs through deep learning. In: Proceedings of the 2017 ACM SIGSAC Conference on Computer and Communications Security. ACM, Dallas, Texas, USA, CCS ’17, pp. 1285–1298 (2017)

  • Farshchi, M., Schneider, J., Weber, I., et al.: Experience report: anomaly detection of cloud application operations using log and cloud metric correlation analysis. In: 2015 IEEE 26th International Symposium on Software Reliability Engineering (ISSRE), pp. 24–34 (2015)

  • Fu, Q., Lou, J.G., Wang, Y., et al.: Execution anomaly detection in distributed systems through unstructured log analysis. In: International Conference on Data Mining (Full Paper). IEEE, pp. 149–158 (2009)

  • Hamooni, H., Debnath, B., Xu, J., et al.: LogMine: fast pattern recognition for log analytics. In: Proceedings of the 25th ACM International on Conference on Information and Knowledge Management. ACM, Indianapolis, Indiana, USA, CIKM ’16, pp. 1573–1582 (2016)

  • He, P., Zhu, J., Zheng, Z., et al.: Drain: an online log parsing approach with fixed depth tree. In: 2017 IEEE International Conference on Web Services (ICWS), pp. 33–40 (2017)

  • He, S., Zhu, J., He, P., et al.: Experience report: system log analysis for anomaly detection. In: 2016 IEEE 27th International Symposium on Software Reliability Engineering (ISSRE), pp. 207–218 (2016)

  • He, S., Lin, Q., Lou, J.G., et al.: Identifying impactful service system problems via log analysis. In: Proceedings of the 2018 26th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering. ACM, Lake Buena Vista, FL, USA, ESEC/FSE 2018, pp. 60–70 (2018)

  • He, S., Zhu, J., He, P., et al.: Loghub: a large collection of system log datasets towards automated log analytics. arXiv:2008.06448 [cs] (2020)

  • Jiang, Z.M., Hassan, A.E., Hamann, G., et al.: An automated approach for abstracting execution logs to execution events. J. Softw. Maint. Evol. Res. Pract. 20(4), 249–267 (2008)

    Article  Google Scholar 

  • Kruegel, C., Vigna, G.: Anomaly detection of web-based attacks. In: Proceedings of the 10th ACM Conference on Computer and Communications Security. ACM, Washington D.C., USA, CCS ’03, pp. 251–261 (2003)

  • Le, V.H., Zhang, H.: Log-based anomaly detection without log parsing. In: 2021 36th IEEE/ACM International Conference on Automated Software Engineering (ASE), IEEE, pp. 492–504 (2021)

  • Le, V.H., Zhang, H.: Log-based anomaly detection with deep learning: How far are we? In: 2022 IEEE/ACM 44th International Conference on Software Engineering (ICSE), IEEE, pp. 1356–1367 (2022)

  • Li, T., Jiang, Y., Zeng, C., et al.: FLAP: An end-to-end event log analysis platform for system management. In: Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. ACM, Halifax, NS, Canada, KDD ’17, pp. 1547–1556 (2017)

  • Liang, Y., Zhang, Y., Xiong, H., et al.: Failure prediction in IBM BlueGene/L event logs. In: Seventh IEEE International Conference on Data Mining (ICDM 2007), pp. 583–588 (2007)

  • Lin, Q., Zhang, H., Lou, J., et al.: Log clustering based problem identification for online service systems. In: 2016 IEEE/ACM 38th International Conference on Software Engineering (ICSE), pp. 102–111 (2016)

  • Liu, F.T., Ting, K.M., Zhou, Z.: Isolation forest. In: 2008 Eighth IEEE International Conference on Data Mining, pp. 413–422 (2008)

  • Lou, J.G., Fu, Q., Yang, S., et al.: Mining program workflow from interleaved traces. In: Proceedings of the 16th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. ACM, Washington, DC, USA, KDD ’10, pp. 613–622 (2010)

  • Makanju, A., Zincir-Heywood, A.N., Milios, E.E.: A lightweight algorithm for message type extraction in system application logs. IEEE Trans. Knowl. Data Eng. 24(11), 1921–1936 (2012)

    Article  Google Scholar 

  • Makanju, A.A., Zincir-Heywood, A.N., Milios, E.E.: Clustering event logs using iterative partitioning. In: Proceedings of the 15th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD ’09, pp. 1255–1264 (2009)

  • Mariani, L., Pastore, F.: Automated identification of failure causes in system logs. In: 2008 19th International Symposium on Software Reliability Engineering (ISSRE), pp. 117–126 (2008)

  • Meng, W., Liu, Y., Zhu, Y., et al.: LogAnomaly: unsupervised detection of sequential and quantitative anomalies in unstructured logs. In: IJCAI, pp. 4739–4745 (2019)

  • Meng, W., Liu, Y., Zhang, S., et al.: LogClass: anomalous log identification and classification with partial labels. IEEE Transactions on Network and Service Management p. 1 (2021)

  • Mikolov, T., Sutskever, I., Chen, K., et al.: Distributed representations of words and phrases and their compositionality. arXiv preprint arXiv:1310.4546 (2013)

  • Nedelkoski, S., Bogatinovski, J., Acker, A., et al.: Self-attentive classification-based anomaly detection in unstructured logs. arXiv:2008.09340 [cs, stat] (2020)

  • Oliner, A., Stearley, J.: What supercomputers say: a study of five system logs. In: 37th Annual IEEE/IFIP International Conference on Dependable Systems and Networks (DSN’07), pp. 575–584 (2007)

  • Pedregosa, F., Varoquaux, G., Gramfort, A., et al.: Scikit-learn: machine learning in python. J. Mach. Learn. Res. 12, 2825–2830 (2011)

    MathSciNet  MATH  Google Scholar 

  • Ross, S.M.: Introduction to Probability and Statistics for Engineers and Scientists, 4th edn. Academic Press, Amsterdam, Boston (2009)

    MATH  Google Scholar 

  • Strang, G.: Linear Algebra and Its Applications, 4th edn. Cengage Learning, Belmont, CA (2006)

    MATH  Google Scholar 

  • Virtanen, P., Gommers, R., Oliphant, T.E., et al.: SciPy 1.0: fundamental algorithms for scientific computing in Python. Nat. Methods 17(3), 261–272 (2020)

    Article  Google Scholar 

  • van der Walt, S., Colbert, S.C., Varoquaux, G.: The NumPy array: a structure for efficient numerical computation. Comput. Sci. Eng. 13(2), 22–30 (2011)

    Article  Google Scholar 

  • Xu, W., Huang, L., Fox, A., et al.: Detecting large-scale system problems by mining console logs. In: Proceedings of the ACM SIGOPS 22Nd Symposium on Operating Systems Principles. ACM, Big Sky, Montana, USA, SOSP ’09, pp. 117–132 (2009)

  • Yin, K., Yan, M., Xu, L., et al.: Improving log-based anomaly detection with component-aware analysis. In: 2020 IEEE International Conference on Software Maintenance and Evolution (ICSME), IEEE, pp. 667–671 (2020)

  • Zhang, B., Zhang, H., Moscato, P., et al.: Anomaly detection via mining numerical workflow relations from logs. In: Proceedings of the 39th International Symposium on Reliable Distributed Systems (SRDS 2020), Shanghai, China (2020)

  • Zhang, X., Xu, Y., Lin, Q., et al.: Robust log-based anomaly detection on unstable log data. In: Proceedings of the 2019 27th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering. ACM, Tallinn, Estonia, ESEC/FSE 2019, pp. 807–817 (2019)

  • Zhu, J., He, P., Fu, Q., et al.: Learning to log: helping developers make informed logging decisions. In: Proceedings of the 37th International Conference on Software Engineering, Vol. 1. IEEE Press, Florence, Italy, ICSE ’15, pp. 415–425 (2015)

  • Zhu, J., He, S., Liu, J., et al.: Tools and benchmarks for automated log parsing. In: Proceedings of the 41st International Conference on Software Engineering: Software Engineering in Practice. IEEE Press, Montreal, Quebec, Canada, ICSE-SEIP ’10, pp. 121–130 (2019)

Download references

Acknowledgements

This research is supported by Australian Research Council’s Discovery Projects (DP200102940 and DP220103044).

Author information

Authors and Affiliations

  1. School of Information and Physical Sciences, The University of Newcastle, Newcastle, NSW, Australia

    Bo Zhang, Hongyu Zhang, Van-Hoang Le & Pablo Moscato

  2. School of Mathematics, Sun Yat-sen University, Zhuhai, China

    Aozhong Zhang

Authors
  1. Bo Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Hongyu Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Van-Hoang Le
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Pablo Moscato
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Aozhong Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Hongyu Zhang.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Zhang, B., Zhang, H., Le, VH. et al. Semi-supervised and unsupervised anomaly detection by mining numerical workflow relations from system logs. Autom Softw Eng 30, 4 (2023). https://doi.org/10.1007/s10515-022-00370-w

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s10515-022-00370-w

Keywords

Search

Navigation