Abstract
In recent years, the existence of open source software (OSS) is indispensable for software development. While developer can benefit from functions of OSS, there is a problem that it is very difficult to locate the cause when problems occur. In this study, we propose a method to calculate anomaly score for each line of log data. In our method, the temporal pattern is learned using Hierarchical Temporal Memory, which is an unsupervised real-time learning algorithm, and the anomaly score is obtained based on the internal state of the model. In the experiment, we compare the learning situation in the following three input formats, word ID, word embedding, and sentence embedding. In the experiments using actual log data, it was found that the method with word ID has the highest f1 score and runtime performance, but the precision needs to be improved in order to suppress useless information.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Ahmad, S., Lavin, A., Purdy, S., Agha, Z.: Unsupervised real-time anomaly detection for streaming data. Neurocomputing 262, 134–147 (2017). https://doi.org/10.1016/j.neucom.2017.04.070
Wu, Y., Schuster, M., Chen, Z., Le, Q.V., Norouzi, M., Macherey, W., Krikun, M., Cao, Y., Gao, Q., Macherey, K., Klingner, J., Shah, A., Johnson, M., Liu, X., Kaiser, U., Gouws, S., Kato, Y., Kudo, T., Kazawa, H., Dean, J.: Google’s neural machine translation system: bridging the gap between human and machine translation (2016)
Devlin, J., Chang, M.-W., Lee, K., Kristina, T.: BERT: pre-training of deep bidirectional transformers for language understanding. In: NAACL-HLT (2018)
Acknowledgments
We thank Keiichi Tokuyama and his section members for their helpful feedback on the paper. This work is supported by a grant from Panasonic System Design.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2020 The Editor(s) (if applicable) and The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Hirakawa, R., Tominaga, K., Nakatoh, Y. (2020). Study on Software Log Anomaly Detection System with Unsupervised Learning Algorithm. In: Ahram, T., Taiar, R., Gremeaux-Bader, V., Aminian, K. (eds) Human Interaction, Emerging Technologies and Future Applications II. IHIET 2020. Advances in Intelligent Systems and Computing, vol 1152. Springer, Cham. https://doi.org/10.1007/978-3-030-44267-5_18
Download citation
DOI: https://doi.org/10.1007/978-3-030-44267-5_18
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-44266-8
Online ISBN: 978-3-030-44267-5
eBook Packages: EngineeringEngineering (R0)